Computer Vision

AdaCLIP-D: Hybrid Denoising Anomaly Detection

AdaCLIP-D is a multi-stage hybrid denoising framework that restores zero-shot anomaly detection performance under heavy noise and corruption by combining spatial U-Net image restoration with 3 embedded in-ViT DnCNN modules and noise-aware adaptive patch gating.

AdaCLIPU-Net DenoiserDnCNN ModulesPolarized Self-AttentionPyTorchMVTec AD / VisA

View source

83.44 → 88.25

VisA Image-F1 Gain

96.88%

MVTec AD Pixel-AUROC

Robust Restoration

Gaussian Noise (σ=50)

Depths 4, 8, 12

DnCNN ViT Depths

The problem

State-of-the-art zero-shot anomaly detection models like AdaCLIP drop sharply in accuracy under image corruption (e.g. AUROC falling from 89.7% to 71.2% under Gaussian noise σ=50). Conventional single-stage denoisers create domain gaps and over-smooth visual features, erasing critical anomaly cues.

Key features

Multi-Stage Hybrid Architecture: Dual-level noise removal combining global spatial U-Net restoration with in-transformer ViT feature refinement
U-Net Spatial Denoiser (D1): 5-level encoder-decoder with PReLU, multi-scale residual connections, depthwise-separable convolutions, and Polarized Self-Attention (PSA) bottleneck
Deep ViT DnCNN Integration (D2, D3, D4): Three 17-layer residual DnCNN modules injected at Transformer depths 4, 8, and 12 for feature-level noise subtraction
Noise-Aware Adaptive Patch Gating: Dynamic gating weight α = sigmoid(MLP(Var(P_i))) blending denoised representations while retaining original semantics on clean patches
Curriculum Training Scheme: Pretrained with Charbonnier + Perceptual + GAN loss on DIV2K, followed by 0.7 MSE + 0.3 SSIM hybrid objective with domain adaptation on MVTec AD & LoDoPaB-CT
Gradient-Controlled Feature Preservation: Stop-gradient operations restricting gradient flow by ~30% and bounding residual noise (||ε||_2 ≤ 0.1·σ_input) to preserve subtle anomaly patterns
Substantial Robustness Gains: Achieved +4.81 F1 score boost (83.44 → 88.25) and +0.70 AP on noisy VisA benchmark datasets
Domain Evaluation: Comprehensive benchmarking across medical (ColonDB), industrial (MVTec AD), and object anomaly (VisA) datasets

Architecture

1Noisy Image → U-Net Denoiser (D1: Spatial Restoration with Polarized Self-Attention)
2Restored Image → CLIP ViT Patch Embedding & Linear Projection
3Transformer Blocks 1–4 → DnCNN (D2: Mid-Level Feature Denoising)
4Transformer Blocks 5–8 → DnCNN (D3: High-Level Feature Denoising)
5Transformer Blocks 9–12 → DnCNN (D4: Deep Semantic Denoising)
6Noise-Aware Patch Gating (α = sigmoid(MLP(Var(P_i)))) → Hybrid Semantic Fusion
7Zero-Shot Text Prompt Alignment → AdaCLIP Anomaly Prediction Map

Important functions

Noise-Aware Adaptive Patch Gating & In-ViT DnCNN Modulepython

import torch
import torch.nn as nn

class NoiseAwarePatchGating(nn.Module):
    def __init__(self, embed_dim: int):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(1, 16),
            nn.ReLU(),
            nn.Linear(16, 1),
            nn.Sigmoid()
        )

    def forward(self, patch_tokens: torch.Tensor, denoised_tokens: torch.Tensor) -> torch.Tensor:
        # Calculate patch variance as a proxy for noise content
        patch_variance = torch.var(patch_tokens, dim=-1, keepdim=True)  # [B, N, 1]
        alpha = self.mlp(patch_variance)  # Dynamic gating weight alpha

        # Adaptive blending: high noise -> stronger denoising, clean -> retain original semantics
        return alpha * denoised_tokens + (1.0 - alpha) * patch_tokens

class InViTDnCNNModule(nn.Module):
    def __init__(self, num_layers: int = 17, in_channels: int = 768):
        super().__init__()
        layers = [nn.Conv2d(in_channels, 64, kernel_size=3, padding=1), nn.ReLU(inplace=True)]
        for _ in range(num_layers - 2):
            layers.extend([
                nn.Conv2d(64, 64, kernel_size=3, padding=1, bias=False),
                nn.BatchNorm2d(64),
                nn.ReLU(inplace=True)
            ])
        layers.append(nn.Conv2d(64, in_channels, kernel_size=3, padding=1, bias=False))
        self.dncnn = nn.Sequential(*layers)

    def forward(self, feature_map: torch.Tensor) -> torch.Tensor:
        predicted_noise = self.dncnn(feature_map)
        return feature_map - predicted_noise  # Residual noise subtraction

U-Net Spatial Denoiser with Polarized Self-Attention (PSA)python

class PolarizedSelfAttention(nn.Module):
    def __init__(self, channel: int):
        super().__init__()
        self.ch_wv = nn.Conv2d(channel, channel // 2, kernel_size=1)
        self.ch_wq = nn.Conv2d(channel, 1, kernel_size=1)
        self.sp_wv = nn.Conv2d(channel, channel // 2, kernel_size=1)
        self.sp_wq = nn.Conv2d(channel, channel, kernel_size=1)
        self.softmax = nn.Softmax(dim=-1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        b, c, h, w = x.size()
        # Channel-only self-attention branch
        ch_q = self.ch_wq(x).view(b, 1, -1)
        ch_v = self.ch_wv(x).view(b, c // 2, -1)
        ch_attn = torch.matmul(ch_v, self.softmax(ch_q).transpose(-1, -2)).view(b, c // 2, 1, 1)
        ch_out = torch.sigmoid(ch_attn) * x

        # Spatial-only self-attention branch
        sp_q = self.sp_wq(x).view(b, c, -1)
        sp_v = self.sp_wv(x).view(b, c // 2, -1)
        sp_attn = torch.matmul(self.softmax(sp_q).transpose(-1, -2), sp_v).view(b, 1, h, w)
        sp_out = self.sigmoid(sp_attn) * x

        return ch_out + sp_out

# Loss function: Charbonnier + Perceptual + GAN composite objective
def composite_unet_loss(y_pred, y_true, perceptual_loss, gan_loss):
    charbonnier = torch.mean(torch.sqrt((y_pred - y_true) ** 2 + 1e-6))
    return 0.5 * charbonnier + 0.3 * perceptual_loss + 0.2 * gan_loss

AdaCLIP-D Forward Execution & Gradient-Controlled Preservationpython

def forward_adaclip_d(noisy_image: torch.Tensor, sigma_input: float) -> torch.Tensor:
    # Stage 1: Spatial Image Denoising via U-Net (D1)
    denoised_image = unet_denoiser(noisy_image)

    # Enforce bounded residual noise: ||epsilon||_2 <= 0.1 * sigma_input
    residual = denoised_image - noisy_image
    residual_norm = torch.norm(residual, p=2, dim=[1,2,3], keepdim=True)
    max_norm = 0.1 * sigma_input
    residual = torch.where(residual_norm > max_norm, residual * (max_norm / residual_norm), residual)
    denoised_image = noisy_image + residual

    # Stage 2: In-ViT Feature Denoising with Stop-Gradient
    tokens = clip_vit.patch_embed(denoised_image)
    for i, block in enumerate(clip_vit.blocks):
        tokens = block(tokens)
        if i + 1 in [4, 8, 12]:  # Insert DnCNN modules at depths 4, 8, 12
            feat_map = tokens_to_grid(tokens)
            denoised_feat = dncnn_modules[i](feat_map)
            denoised_tokens = grid_to_tokens(denoised_feat)

            # Apply Noise-Aware Adaptive Patch Gating
            tokens = patch_gating(tokens, denoised_tokens)

            # Reduce gradient flow by 30% to preserve anomaly features
            tokens = 0.7 * tokens + 0.3 * tokens.detach()

    # Stage 3: Hybrid Semantic Fusion & Anomaly Map Calculation
    anomaly_map = adaclip_head(tokens, prompt_embeddings)
    return anomaly_map

Simulation & screenshots

AdaCLIP-D: Hybrid Denoising Anomaly Detection cover — AdaCLIP-D: Hybrid Denoising Anomaly Detection — concept visual

AdaCLIP-D multi-stage hybrid denoising architecture and anomaly localization map

Back to all projects