Abstract

Most open-vocabulary semantic segmentation (OVSS) methods leverage CLIP’s vision-language alignment to classify each pixel against arbitrary text-defined categories without task-specific training.

However, CLIP is trained at the image level, and its local patch features are inevitably contaminated by global semantics. This contamination cascades through self-attention across layers, producing outlier tokens, progressively degrading local discriminability and leaving a pervasive global bias that blurs inter-class boundaries.

To suppress feature contamination, we propose CLIP-Sanitizer, a training-free pipeline with three concise operations:

  • Feature Rectification (FR) corrects intermediate outlier tokens with local neighbors.
  • Global Debiasing (GD) adaptively subtracts residual $[\text{CLS}]$ bias from patch embeddings.
  • Guided Feature Refinement (GFR) upsamples and filters features to sharpen boundaries.

The pipeline introduces no learnable parameters and works with any frozen CLIP Vision Transformer (ViT) backbone. Experiments on ten remote sensing datasets covering satellite, aerial, and UAV imagery show consistent mIoU improvements over existing training-free methods.

Background

Open-Vocabulary Semantic Segmentation

Given an image $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$ and a category set $\mathcal{C} = \{c_1, \dots, c_K\}$ specified as free-form text prompts at inference, predict a dense label map $Y \in \mathcal{C}^{H \times W}$ .

CLIP [1] makes this feasible: a vision encoder and a text encoder jointly trained with a contrastive objective on hundreds of millions of image-text pairs.

At inference, each patch token $\mathbf{z}_i$ from the final layer is compared against text embeddings $\{\mathbf{t}_k\}$ via cosine similarity:

$$p_{i,k} = \frac{\mathbf{z}_i^\top \mathbf{t}_k}{\|\mathbf{z}_i\| \, \|\mathbf{t}_k\|}.$$

Why remote sensing?

  • Pixel-level annotation is exceptionally tedious and demands domain expertise
  • Land-cover vocabularies are heterogeneous and continually evolving
  • Closed-set supervised models cannot generalize to unseen categories

Applications: urban planning, environmental monitoring, precision agriculture, disaster response.

Problem: CLIP Patch Features Are Contaminated

CLIP’s contrastive loss operates on the global $[\text{CLS}]$ token — trained to summarize the entire image into one compact vector. Patch tokens are only indirectly aligned with text and lack local discriminability [2], [3].

Two coupled symptoms

  1. Outlier tokens — a few patches receive abnormally high $[\text{CLS}]$ attention; their local content is overwritten by global signal and propagated to neighbors [4], [5].
  2. Global bias — pervasive $[\text{CLS}]$ contamination drifts every patch toward dominant image-level semantics, blurring inter-class boundaries [6].

Amplified in remote sensing

  • Large scale variations across crops
  • Complex backgrounds and mixed land-cover classes within one image
  • Small / rare classes get suppressed by mixed scene-level signal

Related Work

Training-based

Fine-tune CLIP or attach learnable adapters.

PACL, Mask-Adapter, decoupled fine-tuning, perceptual grouping, CLIP-DINOiser. Domain-specific: RemoteCLIP, GeoCLIP.

Need annotations + compute; risk losing zero-shot generality.

Training-free

Modify frozen CLIP at inference.

MaskCLIP, SCLIP, CLIP-Surgery, ClearCLIP, GEM, NACLIP.

Mostly redesign the final attention layer — do not address contamination at intermediate layers.

Hybrid

Augment CLIP with other VFMs.

ProxyCLIP, Trident (CLIP+DINO+SAM), SAMRefiner.

Better quality, higher complexity, do not fix the underlying contamination.

Existing remedies either act only at the final layer, statistically filter outliers ignoring spatial relations, or subtract $[\text{CLS}]$ uniformly — over-correcting clean patches while under-correcting contaminated ones. CLIP-Sanitizer is a unified contamination-aware pipeline.

Method Overview

CLIP-Sanitizer pipeline. FR corrects outlier patch tokens during the forward pass, GD applies similarity-weighted correction to final patch features, and GFR uses upsampled features as structural guidance for sharper boundaries. The pipeline is training-free and works with frozen CLIP backbones.
CLIP-Sanitizer pipeline. FR corrects outlier patch tokens during the forward pass, GD applies similarity-weighted correction to final patch features, and GFR uses upsampled features as structural guidance for sharper boundaries. The pipeline is training-free and works with frozen CLIP backbones.

Key insight. Treat dense prediction as contamination removal. FR fixes where corruption originates; GD removes the residual global drift; GFR restores the spatial structure the patch tokenizer lost.

Feature Rectification (FR)

CLIP’s $[\text{CLS}]$ token may over-attend to a few structurally salient patches and inject image-level semantics into their local features. These contaminated outliers can then be reused by later attention layers and spread the bias across the feature map.

A patch is an outlier when it receives strong $[\text{CLS}]$ attention but has weak self-attention:

$$S_i = \frac{A_{\text{CLS}, i}}{A_{i,i} + \epsilon}.$$

For the top- $k$ outliers, FR replaces each feature by an inverse-similarity-weighted average of its eight spatial neighbors $\mathcal{N}_i = \mathcal{N}_8(i)$ :

$$w_{ij} = 1 - \text{sim}(\mathbf{x}_i, \mathbf{x}_j), \quad \alpha_{ij} = \text{softmax}_{\mathcal{N}_i}(w_{ij}), \quad \mathbf{x}'_i = \sum_{j \in \mathcal{N}_i} \alpha_{ij} \mathbf{x}_j.$$

Less similar neighbors get larger weight — preserves local content diversity while removing injected global signal.

FR pipeline
FR pipeline

Rectified tokens continue through the remaining layers, preventing early local corruption from cascading forward.

FR: Outlier Visualization

$[\text{CLS}]$ attention to the marked patch — outlier receives strong attention.
$[\text{CLS}]$ attention to the marked patch — outlier receives strong attention.
Contaminated feature map — local discriminability is weakened.
Contaminated feature map — local discriminability is weakened.
Outlier score map — the outlier stands out.
Outlier score map — the outlier stands out.
After FR — outlier receives less $[\text{CLS}]$ attention and recovers local discriminability.
After FR — outlier receives less $[\text{CLS}]$ attention and recovers local discriminability.

Global Debiasing (GD)

FR removes localized outliers, but final patch features may still share a common component aligned with $[\text{CLS}]$ . In remote sensing crops containing multiple classes, this global component becomes a mixed scene descriptor that suppresses small or rare classes.

Let $s_i = \mathbf{x}_i \cdot \mathbf{c}$ measure patch– $[\text{CLS}]$ alignment. Subtract a stronger $[\text{CLS}]$ component from patches more similar to $[\text{CLS}]$ :

$$\hat{\mathbf{x}}_i = \mathbf{x}_i - \lambda (s_i + 1) \mathbf{c}, \quad \hat{\mathbf{x}}_i \leftarrow \frac{\hat{\mathbf{x}}_i}{\|\hat{\mathbf{x}}_i\|}.$$

Adaptive correction. The $(s_i + 1)$ term makes correction strength proportional to contamination level, avoiding over-correction of clean patches (uniform subtraction would over-correct clean and under-correct contaminated patches).

Patch–CLS similarity before GD.
Patch–CLS similarity before GD.
Patch–CLS similarity after GD — more balanced distribution.
Patch–CLS similarity after GD — more balanced distribution.

Debiasing reduces excessive similarity to $[\text{CLS}]$ and yields a more balanced distribution for local discrimination.

Guided Feature Refinement (GFR)

CLIP’s ViT-B/16 produces only a coarse $14 \times 14$ patch grid, which blurs object boundaries. GFR:

  1. Upsamples sanitized features to full resolution with SimFeatUp [6]: $\mathbf{F}_{\text{high}} \in \mathbb{R}^{H \times W \times D}$ .
  2. Computes logits $\mathbf{P} = \mathbf{F}_{\text{high}} \mathbf{F}_{\text{text}}^\top$ .
  3. Projects $\mathbf{F}_{\text{high}}$ to a scalar guide image $\mathbf{I}$ by PCA.
  4. Refines each class logit $\mathbf{P}_c$ with a guided filter [7]:
$$q_i = a_k I_i + b_k, \quad a_k = \frac{\text{cov}(\mathbf{I}, \mathbf{P}_c)_k}{\sigma^2_{\mathbf{I},k} + \epsilon}.$$
Input
Input
SegEarth-OV
SegEarth-OV
CLIP-Sanitizer w/o GFR
CLIP-Sanitizer w/o GFR
CLIP-Sanitizer (full)
CLIP-Sanitizer (full)

Averaging coefficients from overlapping windows yields refined logits $\mathbf{Q}$ ; the final segmentation is obtained by pixel-wise $\arg\max$ . From left to right: input, SegEarth-OV, ours without GFR, full model — the full model produces cleaner boundaries.

Experiments: Datasets

Ten public remote sensing segmentation benchmarks across three imaging platforms.

Satellite / Aerial

  • Potsdam — 6 classes, 5 cm/pixel airborne
  • Vaihingen — 6 classes, 9 cm/pixel airborne
  • LoveDA [8] — 7 land-use classes, urban + rural
  • WHU $^{\text{Aerial}}$ [9] — binary building, 0.3 m/pixel
  • Inria [10] — 5 cities, building/background, 0.3 m/pixel
  • xBD [11] — disaster, 45,362 km $^2$ , 0.8 m/pixel
  • iSAID [12] — 15 categories, instance-seg origin

UAV

  • UDD5 — 5 classes, high-resolution drone
  • VDD [13] — 9 classes, ~4000×3000
  • UAVid [14] — 8 classes, oblique-view UAV

Implementation

  • CLIP ViT-B/16 visual encoder
  • Built on SegEarth-OV (last block FFN+residual removed; multi-source self-attention; SimFeatUp)
  • MMSegmentation, NVIDIA A40
  • 224×224 sliding window, no overlap
  • Metrics: mIoU and mF1

From the sensitivity study: top- $k=35$ for aerial/satellite, top- $k=5$ for UAV, $\lambda=0.2$ for global debiasing.

Results & Ablation

Comparison with state-of-the-art (mIoU, %)

Method LoveDA Potsdam Vaihingen WHU Inria xBD UDD5 VDD UAVid iSAID
SCLIP [3] 31.12 35.67 27.97 33.62 41.34 28.74 29.59 40.45 37.55 17.31
ProxyCLIP $^*$ [15] 31.45 44.15 28.86 38.84 45.15 32.07 34.49 41.31 45.55 21.88
SegEarth-OV [6] 31.43 45.27 29.62 46.78 48.97 39.74 43.41 42.65 43.06 20.36
Trident $^*$ [16] 31.91 44.28 27.89 35.24 42.83 30.91 35.53 43.26 44.45 22.40
SC-CLIP [5] 31.74 42.25 29.64 29.73 38.68 29.36 37.25 42.43 44.82 18.97
CLIP-Sanitizer (ours) 33.49 46.25 29.66 49.64 50.54 40.56 43.70 44.59 43.02 21.50

$^*$ hybrid with SAM/DINO features. Best on 8 of 10 benchmarks.

Ablation (mIoU, %)

Config Potsdam LoveDA VDD
baseline (SegEarth-OV) 45.27 31.43 42.65
+ FR 45.54 31.80 42.79
+ GD 46.07 33.35 44.43
+ GFR (full) 46.25 33.49 44.59
  • All three modules help.
  • GD contributes the largest gain — global bias is the dominant contamination.
  • Gains on UAVid / iSAID remain limited; hybrid SAM/DINO features still help there.

Qualitative Comparison

Qualitative comparison. CLIP-Sanitizer produces less fragmented masks and cleaner boundaries than CLIP-only baselines.
Qualitative comparison. CLIP-Sanitizer produces less fragmented masks and cleaner boundaries than CLIP-only baselines.

Sensitivity Analysis

top- $k$ in FR

$k=35$ for aerial/satellite (semantically diverse crops), $k=5$ for UAV (spatially coherent).

top- $k$ Potsdam LoveDA UDD5 VDD
0 45.27 31.43 43.41 42.65
5 45.33 31.54 43.34 42.79
35 45.54 31.80 43.29 42.20
50 45.50 31.84 43.25 41.89

$\lambda$ in GD

$\lambda=0.2$ balanced default; larger $\lambda$ suits mixed low-res scenes.

$\lambda$ (top- $k$ =35) Potsdam LoveDA
0 45.54 31.80
0.15 45.99 32.98
0.20 46.07 33.35
0.30 46.08 33.98
0.50 45.31 34.57

Ablation Visualization

Qualitative visualization of the ablation study. Modules are applied sequentially.
Qualitative visualization of the ablation study. Modules are applied sequentially.

Limitations and Discussion

Higher compute than pure CLIP baselines

Overhead comes from SimFeatUp upsampling + guided filtering — not a heavier backbone.

Method Params (M) FLOPs (G) Mem (MB)
vanilla CLIP 149.62 17.01 515.96
SCLIP 149.62 17.04 520.32
SegEarth-OV 149.91 31.80 3147.95
ProxyCLIP $^*$ 235.43 95.77 564.15
Trident $^*$ 329.15 509.55 2478.56
CLIP-Sanitizer 149.91 45.04 3150.69

Hyperparameter sensitivity

top- $k$ and $\lambda$ depend on image resolution and scene heterogeneity, so a fixed config is suboptimal across heterogeneous datasets.

Future work. Design an adaptive strategy that selects top- $k$ and $\lambda$ from image content or resolution automatically.

Where we still lose

UAVid (oblique angles) and iSAID (instance-seg origin). Hybrid methods (ProxyCLIP, Trident) benefit more from SAM/DINO features on these.

Conclusion

Treat dense prediction as contamination removal.

FR — Feature Rectification

Detect and correct outlier tokens inside the forward pass — block contamination before it cascades.

GD — Global Debiasing

Adaptive per-patch subtraction of residual $[\text{CLS}]$ bias — no uniform over-correction.

GFR — Guided Feature Refinement

Use upsampled features as structural guidance to sharpen boundaries beyond the 14×14 ViT grid.

Experiments on 10 remote sensing datasets show CLIP-Sanitizer consistently improves mIoU — demonstrating that CLIP’s inherent representations can be effectively sanitized to yield features with better local discriminability, without any training.

Three concise operations. No learnable parameters. Any frozen CLIP ViT backbone.

[1]
A. Radford et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning, PmLR, 2021, pp. 8748–8763. doi: 10.48550/arXiv.2103.00020.
[2]
J. Mukhoti et al., “Open vocabulary semantic segmentation with patch aligned contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19413–19423.
[3]
F. Wang, J. Mei, and A. Yuille, “Sclip: Rethinking self-attention for dense vision-language inference,” in European conference on computer vision, Springer, 2024, pp. 315–332.
[4]
S. Jin, S. Yu, B. Zhang, M. Sun, Y. Dong, and J. Xiao, “Feature Purification Matters: Suppressing Outlier Propagation for Training-Free Open-Vocabulary Semantic Segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 20291–20300.
[5]
S. Bai et al., “Self-calibrated clip for training-free open-vocabulary segmentation,” IEEE Transactions on Image Processing, 2025.
[6]
K. Li et al., “Segearth-ov: Towards training-free open-vocabulary segmentation for remote sensing images,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 10545–10556.
[7]
K. He, J. Sun, and X. Tang, “Guided Image Filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013, doi: 10.1109/TPAMI.2012.213.
[8]
J. Wang, Z. Zheng, A. Ma, X. Lu, and Y. Zhong, “LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung, Eds., Curran Associates, Inc., 2021. [Online]. Available: https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/4e732ced3463d06de0ca9a15b6153677-Paper-round2.pdf
[9]
S. Ji, S. Wei, and M. Lu, “Fully Convolutional Networks for Multisource Building Extraction From an Open Aerial and Satellite Imagery Data Set,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 1, pp. 574–586, 2019, doi: 10.1109/TGRS.2018.2858817.
[10]
E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark,” in 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2017, pp. 3226–3229. doi: 10.1109/IGARSS.2017.8127684.
[11]
R. Gupta et al., “xbd: A dataset for assessing building damage from satellite imagery,” arXiv preprint arXiv:1911.09296, 2019.
[12]
S. Waqas Zamir et al., “iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 28–37.
[13]
W. Cai, K. Jin, J. Hou, C. Guo, L. Wu, and W. Yang, “Vdd: Varied drone dataset for semantic segmentation,” Journal of Visual Communication and Image Representation, vol. 109, p. 104429, 2025.
[14]
Y. Lyu, G. Vosselman, G.-S. Xia, A. Yilmaz, and M. Y. Yang, “UAVid: A semantic segmentation dataset for UAV imagery,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 165, pp. 108–119, 2020, doi: 10.1016/j.isprsjprs.2020.05.009.
[15]
M. Lan, C. Chen, Y. Ke, X. Wang, L. Feng, and W. Zhang, “Proxyclip: Proxy attention improves clip for open-vocabulary segmentation,” in European Conference on Computer Vision, Springer, 2024, pp. 70–88.
[16]
Y. Shi, M. Dong, and C. Xu, “Harnessing vision foundation models for high-performance, training-free open vocabulary segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 23487–23497.