WIBE-X: A Multi-Domain Benchmark for Digital Watermarking

WIBE-X is a modular and extensible framework for automated testing of invisible digital watermarking methods across image and audio domains under various attack scenarios. Extending the original WIBE framework, it provides a unified processing pipeline for systematic evaluation of both classical and neural watermarking techniques

The system architecture consists of a sequence of processing configurable stages.

Key features

• Multi-domain support for invisible digital watermarking in images and audio
• Modularity and extensibility through a unified plugin-based architecture for watermarking methods, datasets, attacks, and metrics
• Comprehensive robustness testing with configurable individual and composed transformations, including conventional and neural codecs and generative resynthesis
• Reproducibility through declarative YAML-configured experiments and isolated execution environments
• Comprehensive evaluation with modality-specific quality metrics and unified watermark robustness metrics
• Flexible persistence through multiple storage backends, including files and ClickHouse database
• Interactive visual diagnostics based on sample-level evaluation results
• Scalability from local workstations to multi-GPU and distributed cluster environments

Implemented algorithms, attacks, datasets and metrics

Algorithms

Image domain

Algorithm	Type	Config name	Default capacity	Reference
ARWGAN	post-hoc	arwgan	30 bits	ARWGAN: Attention-Guided Robust Image Watermarking Model Based on GAN
CIN	post-hoc	cin	30 bits	Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms
DCT Marker	post-hoc	dct_marker	1 - 800 bits	Real data performance evaluation of CAISS watermarking scheme
DFT Circle	post-hoc	dft_circle	zero-bit	Discrete Fourier transform-based watermarking method with an optimal implementation radius
DWSF	post-hoc	dwsf	30 bits	Practical Deep Dispersed Watermarking with Synchronization and Fusion - Image Watermarking Algorithm
DWT SVM	post-hoc	dwt_svm	512 bits	SVM-based robust image watermarking technique in LWT domain using different sub-bands
DWT DCT	post-hoc	dwt_dct	100 bits	frequency-domain transforms: DWT + DCT
DWT DCT SVD	post-hoc	dwt_dct_svd	100 bits	frequency-domain with additional SVD processing
HiDDeN	post-hoc	hidden	30 bits	HiDDeN: Hiding Data in Deep Networks --- Image Watermarking Algorithm
InvisMark	post-hoc	invismark	100 bits	Invisible and Robust Watermarking for AI-generated Image Provenance
MBRS	post-hoc	mbrs	30/256 bits	Enhancing Robustness of DNN-based Watermarking by Mini-Batch of Real and Simulated JPEG Compression
SS HiDDeN	post-hoc	sshidden	48 bits	HiDDeN watermarking algorithm adapted from the Stable Signature
RivaGAN	post-hoc	riva_gan	30 bits	Image watermarking via RivaGAN: a deep-learning-based encoder/decoder with attention mechanism
SSL	post-hoc	ssl_watermarking	zero-bit / 32 bits	Watermarking Images in Self-Supervised Latent-Spaces (SSL)
Stable Signature	built-in	stable_signature	48 bits (fixed for single model)	The Stable Signature: Rooting Watermarks in Latent Diffusion Models
StegaStamp	post-hoc	stega_stamp	100 bits	StegaStamp: Invisible Hyperlinks in Physical Photographs
TreeRing	built-in	treering	zero-bit	Tree-Ring: Fingerprints for Diffusion Images that are Invisible and Robust
TrustMark	post-hoc	trustmark	100 bits	TrustMark: Universal Watermarking for Arbitrary Resolution Images
VideoSeal	post-hoc	videoseal	256 bits	Video Seal: Open and Efficient Video Watermarking
PixelSeal	post-hoc	pixelseal	256 bits	Pixel Seal: Adversarial-only training for invisible image and video watermarking
ChunkySeal	post-hoc	chunkyseal	1024 bits	We Can Hide More Bits: The Unused Watermarking Capacity in Theory and in Practice
Watermark Anything	post-hoc	watermark_anything	32 bits	Watermark Anything with Localized Messages - Image Watermarking Algorithm
MaskWM	post-hoc	maskwm	32/64/128 bits	Mask Image Watermarking
SyncSeal	post-hoc	syncseal	method-dependent	GEOMETRIC IMAGE SYNCHRONIZATION WITH DEEP WATERMARKING
Gaussian Shading	built-in	gaussian_shading	256 bits	Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models
Ring-ID	built-in	ringid	zero-bit	RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification - Image Watermarking Algorithm
MaXsive	built-in	maxsive	zero-bit	MaXsive: High-Capacity and Robust Training-Free Generative Image Watermarking in Diffusion Models
METR	built-in	metr	10 bits	METR: Image Watermarking with Large Number of Unique Messages
PIMoG	post-hoc	pimog	30 bits	PIMoG: An Effective Screen-shooting Noise-Layer Simulation for Deep-Learning-Based Watermarking Network
Robust-Wide	post-hoc	robust_wide	64 bits	Robust-Wide: Robust Watermarking Against Instruction-Driven Image Editing
FIN	post-hoc	fin	64 bits	FIN: Flow-Based Robust Watermarking with Invertible Noise Layer for Black-Box Distortions
VINE	post-hoc	vine	100 bits	Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances
SepMark	post-hoc	sepmark	30/128 bits	SepMark: Deep Separable Watermarking for Unified Source Tracing and Deepfake Detection
RoSteALS	post-hoc	rosteals	100 bits	RoSteALS: Robust Steganography using Autoencoder Latent Space

Audio domain

Algorithm	Type	Config name	Default capacity	Reference
AudioSeal	post-hoc	audioseal	16 bits	AudioSeal: Efficient Localized Audio Watermarking
SilentCipher	post-hoc	silent_cipher	40 bits	SilentCipher: Deep Audio Watermarking
WavMark	post-hoc	wavmark	16 bits	WavMark: Watermarking for Audio Generation
AWARE	post-hoc	aware	20 bits	AWARE: Audio Watermarking via Adversarial Resistance to Edits
RobustDNN	post-hoc	dnn_audio	512 bits (6 keys)	Robust speech watermarking by a jointly trained embedder and detector using a DNN
Perth	post-hoc	perth	zero-bit	Perth: Open Source AI Watermarking Model
FSVC	post-hoc	fsvc	40 bits (default params)	Desynchronization Attacks Resilient Watermarking Method Based on Frequency Singular Value Coefficient Modification
Patchwork	post-hoc	patchwork	40 bits (default params)	Patchwork-Based Multilayer Audio Watermarking
DCT-b1	post-hoc	dct_b1	24 bits (default params)	Robust, transparent and high-capacity audio watermarking in DCT domain
Norm space	post-hoc	norm_space	40 bits (default params)	Novel secured scheme for blind audio/speech norm-space watermarking by Arnold algorithm
Spread Spectrum	post-hoc	spread_spectrum	10 bits (default params)	Spread-Spectrum Watermarking of Audio Signals
Quantization Index Modulation	post-hoc	qim	40 bits (default params)	Quantization Index Modulation: A Class of Provably Good Methods for Digital Watermarking and Information Embedding
Echo Hiding Watermarking	post-hoc	echo-positive, echo-negative, echo-forward	16 bits (default params)	Echo hiding
Least Significant Bit	post-hoc	lsb	40 bits (default params)	Audio Steganography Method Using Least Significant Bit (LSB) Encoding Technique

Attacks

Image domain

Attack	Config name	Description
Distortions	jpeg, rotate90, rotate, gaussianblur, gaussiannoise, centercrop, resize, randomcropout, brightness, contrast, pixelshift, colorinversion	Common distortions like JPEG, blur, noise, rotation, etc
WPWMAttacker	wpwmattacker	Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal
DIP	dip	DIP-based watermark evasion attack adopted from the github repository
Adversarial	adversarialembedding	Adversarial embedding attack from WAVES
Adversarial PSNR	adversarialembeddingpsnr	Modification of adversarial embedding attack that uses PSNR
Averaging	averaging	Attack based on simple averaging from paper
Blur Deblur	blurdeblurfpninception	Blur image and restore it with DeblurGAN-v2
DoG Blur Deblur	dogblurdeblurfpninception	DoG blur image and restore it with DeblurGAN-v2
BM3D	bm3d	Block-matching and 3D filtering image denoising
VAE	vaeattack	Adversarial attack using a FLUX.1-schnell VAE to generate noisy image reconstructions
StegastampInversion	stegastampinversion	Adversarial attack that inverts watermarks encoded by Stegastamp
DiffusionRegeneration	diffusionregeneration	Invisible Image Watermarks Are Provably Removable Using Generative AI
FluxRegeneration	fluxregeneration	Image regeneration attack using FLUX image-to-image diffusion model
FluxRinsing	fluxrinsing	Multi-step image purification using repeated FLUX regeneration
VAERegeneration	vaeregeneration	Based on WatermarkAttacker
Frequency Masking	frequencymasking	Image-domain frequency masking attack that suppresses low-frequency components.
Latent Frequency Masking	latentfrequencymasking	Latent-space frequency masking attack for diffusion model representations.
Image Editing Flux Kontext	imageeditingfluxcontext	Adversarial attack that edits images using instruction-guided generation. (Flux Kontext model)
Image Editing InstructPix2Pix	imageeditinginstructpix2pix	Adversarial attack that edits images using instruction-guided generation. (InstructPix2Pix model)
LIIF	liifattack	LIIF super-resolution
SEMAttack	semattack	Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models
WMForger	wmforger	Transferable Black-Box One-Shot Forging of Watermarks via Image Preference Models
TrustMarkRM	trustmarkrm	TrustMark: Universal Watermarking for Arbitrary Resolution Images
NRP small	nrpsmall	A Self-supervised Approach for Adversarial Robustness. Smaller backbone variant.
NRP large	nrplarge	A Self-supervised Approach for Adversarial Robustness. Larger backbone variant.
MPRNet	mprnetattack	Multi-stage progressive image restoration
Uni inv attack FLUX	uniinvattackflux	Image Inversion and Reconstruction using Flux model
Uni edit attack FLUX	unieditattackflux	Image Editing using Flux model
DISCO	discoattack	DISCO: Adversarial Defense with Local Implicit Functions
DiffPure	diffpureattack	Diffusion Models for Adversarial Purification
RealESRGAN	realesrganattack	Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
UnMarkerAttack	unmarkerattack	UnMarker: A Universal Attack on Defensive Image Watermarking
Instagram/CSS filters	instagramcss_filters	Instagram/CSS filters for image editing

Audio domain

Attack	Config name	Description
Signal distortions	signinversion, resampling, requantization, gain, filter, whitenoise, pinknoise, clipping	Common signal-level transformations affecting amplitude, sampling, quantization, frequency content, or noise.
Compression	mpeg, aac, opus, encodec, dac	Lossy conventional and neural audio compression codecs.
Acoustic	echo, reverb	Acoustic effects simulating propagation and reflections of sound.
Effect	pitchshift, dynamicrangecompressor, limiter	Different audio effects.
Enhancement	wienerfilter	Signal enhancement and denoising methods that modify the audio to suppress noise or unwanted components.
MetricGAN+	metricganplus	Black-box speech-enhancement attack using MetricGAN+ (paper, implementation).
GTCRN	gtcrn	Black-box speech-enhancement attack using the official GTCRN DNS3 checkpoint (paper).
Desynchronization	cut, speed, timestretch, invertedtimestretch, flipsamples, zerocrossinserts, framedropout	Distortions that modify the temporal structure or local alignment of the signal.
Vocos	vocos	Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis

Common

Attack	Config name	Description
Distortions	identity	No attack applied

Datasets

Dataset	Config name	Description
Image folder	imagefolderdataset	Loading images from a directory
Prompt folder	promptfolderdataset	Loading text prompts from a directory with .txt files
Audio Folder	audiofolderdataset	Loading audio from a directory
DiffusionDB	diffusiondb	https://github.com/poloclub/diffusiondb, both images and prompts
MSCOCO	mscoco	https://cocodataset.org/, both images and prompts
LibriSpeech	librispeech	ASR corpus based on public domain audio books, audio and transcription
LibriTTS	libritts	Dataset based on LibriSpeech, sr 22050 kHz, audio and transcription
AudioSet	audioset	Dataset of 10-second clips from YouTube
FreeMusicArchive	freemusicarchive	FMA is a large-scale collection of music
VCTK	vctk	The CSTR VCTK Corpus (Voice Cloning Toolkit) is a speech dataset of read English speech with diverse regional accents
CommonVoice	commonvoice	Public-domain multilingual voice database
AISHELL-1	aishell	Aishell is an open-source Chinese Mandarin speech corpus
Golos	golos	Russian ASR dataset with trained acoustic and language models

Metrics

Image domain

Metric	Type	Stage	Config name	Description
PSNR	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	psnr	peak signal-to-noise ratio
SSIM	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	ssim	structural similarity index
LPIPS	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	lpips	The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
DreamSim	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	dreamsim	DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data.
Aesthetic	single image quality	post_embed_metrics, post_attack_metrics	aesthetic	Aesthetic score predictor
CLIP IQA	single image quality	post_embed_metrics, post_attack_metrics	clip_iqa	Exploring CLIP for Assessing the Look and Feel of Images
BLIP	image quality, compare image with text prompt	post_embed_metrics	blip	BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
CLIP Score	image quality, compare image with text prompt	post_embed_metrics	clipscore	CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Image Reward	image quality, compare image with text prompt	post_embed_metrics	imagereward	Learning and Evaluating Human Preferences for Text-to-Image Generation
FID	image quality, compare two sets of images	post_pipeline_embed_metrics, post_pipeline_attack_metrics	fid	FID metric from GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Audio domain

Metric	Type	Stage	Config name	Description
SI-SNR	audio quality, compare with not marked image	post_embed_metrics	si_snr	Scale-Invariant Signal-to-Noise Ratio
PESQ	audio quality, compare with not marked image	post_embed_metrics	pesq	Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs
STOI	audio quality, compare with not marked image	post_embed_metrics	stoi	A short-time objective intelligibility measure for time-frequency weighted noisy speech
NISQA	audio quality assesment delta, compare with not marked image	post_embed_metrics	nisqa	NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
DNSMOS	audio quality assesment delta, compare with not marked image	post_embed_metrics	dnsmos	DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors
SECS	audio quality, compare with not marked image	post_embed_metrics	secs	Speaker Encoder Cosine Similarity

Common

Metric	Type	Stage	Config name	Description
BER	extraction success	post_extract_metrics	ber	Bit Error Rate (multi-bit)
WER	extraction success	post_extract_metrics	wer	Word Error Rate (multi-bit)
TPR at x% FPR	extraction success	post_extract_metrics	tpr@xfpr	True Positive Rate at fixed False Positive Rate threshold (both zero-bit and multi-bit)
Empirical TPR at x% FPR	extraction success	post_extract_metrics	empiricaltpr@xfpr	Empirical True Positive Rate at fixed False Positive Rate threshold (both zero-bit and multi-bit)
P-value	extraction success	post_extract_metrics	p-value	P-value denotes probability to observe the same result as in case of extraction from not watermarked object
Result	auxiliary	post_extract_metrics	result	Records extraction result (zero-bit case)
Embedded watermark	auxiliary	post_embed_metrics	embwm	Records embedded watermark (multi-bit case)
Extracted watermark	auxiliary	post_extract_metrics	extwm	Records extracted watermark (multi-bit case)

Quick start

To assess implemented watermarking algorithms and attacks on watermarks, follow the step-by-step procedure below.

0. Domain

To build the benchmarking environment, user must specify the proper domain profile. The list of supported profiles:

• image - image-based watermarking algorithms (default);
• audio - audio-based watermarking algorithms.

1. Clone

git clone https://github.com/ispras/wibe.git
cd wibe

All subsequent commands are run from this directory.

2. System dependencies

Ubuntu:

sudo apt-get install -y ffmpeg libgl1

• ffmpeg — multimedia processing
• libgl1 — OpenGL, required by some CV/DL dependencies

3. Set up the environment

Option A — one command

WIBENCH_PROFILE=audio source prepare.sh   # or image profile

Option B — step by step

git submodule update --init --recursive
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
(.venv) pip install uv
(.venv) uv sync
(.venv) wibench-venv rebuild --profile=audio   # builds additional venvs

Conflicting dependency pins mean one shared venv is not enough: wibench-venv builds several compatible ones, and wibench picks among them automatically. If you plan to add your own algorithm, attack, dataset or metric, read venv manager docs first.

Optional (not recommended) — download all pre-trained weights

(.venv) python download_models.py   # no need to use: weights download automatically on demand

4. Authenticate with HuggingFace

Some models need HuggingFace access. Set HF_TOKEN to your token (setup details), then:

(.venv) python huggingface_login.py

5. Run an experiment

Specify the path to your configuration file as a required parameter:

(.venv) wibench --config configs/audio/audioseal.yml --profile audio

Upon completion of computations, you can explore the evaluation results for different combinations of watermarking algorithms, attacks, and computed performance metrics.

By default, WIBE-X stores sample-level evaluation results, including watermark robustness, media quality metrics, and execution times. To save the media samples produced at different stages of the processing pipeline, use the --dump-context option.

This saves the original, watermarked, and transformed media samples, including intermediate results produced after each applied attack, for both audio and image experiments.

6. Explore the results

The following figure summarizes the robustness and imperceptibility of the evaluated audio watermarking algorithms. The horizontal axis represents the average watermark robustness measured as TPR@0.1%FPR, while the vertical axes show audio quality measured using SI-SNR and PESQ. Each point corresponds to a watermarking algorithm.

Resources

• Full documentation
• Tutorial video

Publications

This project builds upon our previous work on benchmarking invisible image watermarking:

• WIBE: Watermarks for generated Images – Benchmarking & Evaluation, ASE 2025.

  BibTeX

  @inproceedings{yakushev2025wibe,
    title={WIBE: Watermarks for generated Images--Benchmarking \& Evaluation},
    author={Yakushev, Aleksey and Akimenkov, Aleksandr and Abud, Khaled and Obydenkov, Dmitry and Serzhenko, Irina and Aistov, Kirill and Kovalev, Egor and Fomin, Stanislav and Antsiferova, Anastasia and Lukianov, Kirill and Markin, Yury},
    booktitle={2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
    pages={4033--4036},
    year={2025},
    organization={IEEE}
  }

• WARP: A Unified Benchmark for Invisible Image Watermarking — Robustness and Protection Against Attacks, ACM MM 2026.