Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems

Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning.

This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.

The solver unrolls T = 2 ADMM iterations for demosaicing and T = 3 for deblurring, with parameters shared across iterations, so a single decoder serves every layer. Freezing the encoder is what preserves CLIP's distortion-invariant, content-related features: the ablation shows that learning those same features from scratch collapses performance, confirming the gains come from the pretrained representation rather than from the unrolling architecture alone.

The method stays competitive across both photon regimes and under the BSDS500 → DIV2K dataset shift. The margins over the baselines widen where the photon count is lowest — at γ = 0.05 the self-supervised model leads every baseline on both tasks — and the self-supervised objective recovers most of the supervised performance, never falling more than 0.31 dB behind its own supervised counterpart across all eight settings.

Synthetic Poisson simulations only go so far. This experiment moves to real photon-limited data from the SID dataset (Chen et al., Learning to See in the Dark) under a zero-shot adaptation protocol: a single real low-light image, one 512×512 patch held out for testing, and the remaining patches used only to optimise the self-supervised objective. No clean target and no paired supervision are used at any point.

Because real sensors exhibit channel-dependent photon statistics, the Poisson scaling is estimated independently for each RGB channel from the raw measurements, giving γ_rgb = (0.018, 0.017, 0.026). Reconstruction is performed in the linear RGB domain, then a standard affine colour transform is applied for sRGB visualisation and metric evaluation. Adapting from that one image alone lifts the raw mosaic from 14.95 dB to 26.37 dB — +11.42 dB with nothing clean to learn from — which is the evidence that the GR2R + EI training signal survives contact with real acquisition statistics rather than only the simulated ones.

Keeping the physics in closed form is what makes the solver cheap: the same 10.99 M-parameter decoder as Transfer CLIP runs at a hundredth of the TFLOPs and roughly a hundred times faster per image, because the data-consistency step needs no inner optimisation.