YOLO26-RGB: repurposing YOLO26's depth-trained backbone for image deraining [P]

YOLO26 ships a depth-estimation model — dense, full-resolution, per-pixel regression, a task architecturally much closer to image restoration than to detection. I wanted to know whether the backbone+neck weights it learns through depth training transfer to a different dense-regression task (deraining), compared with training the same architecture from scratch. The deraining model that came out of it is a useful byproduct, but the transfer result is the part I think is worth discussing.

- The depth decoder's multi-scale fusion (project the P3/P4/P5 pyramid to a common width, progressively upsample-and-add P5→P4→P3). That part isn't depth-specific — it's just feature fusion — so RGBHead reuses it.

- Replaced the 1-channel Depth head with a new RGBHead. The config change is one line; RGBHead itself is a new restoration decoder, not a re-pointed depth head.

- Reconstruction tail that continues to full input resolution (deraining needs pixel-exact output; depth stops at 1/4 res).

- Skip connections from the stride-2 and stride-4 backbone layers into the tail, so fine detail has a path that doesn't route through an 8×-downsampled bottleneck.

- Residual output — the head predicts a correction added to the input (NAFNet/Restormer-style), not the image directly.

- LayerNorm in the head's own conv blocks; the backbone and neck stay on BatchNorm (folds into conv at TensorRT export, and keeps the model loadable from the whole YOLO26 pretrained zoo, not just the depth checkpoint).

ClearView as an external lib — its mixed synthetic+real rain recipe, Charbonnier loss, and 10-test-set protocol — so the numbers land on ClearView's own model-zoo scale, not a benchmark I made up. Released scales: nano (5.25M) and small (12.13M).

Loading the YOLO26-depth checkpoint into this architecture matches 468/468 backbone+neck tensors exactly — only the new RGBHead is randomly initialized. So the controlled comparison is: identical architecture and recipe, backbone+neck either from the depth checkpoint or from scratch.

A controlled initialization experiment at nano scale — same architecture, same recipe, fixed 100 epochs each — backbone+neck from the YOLO26-depth checkpoint vs. random init:

Init Avg PSNR (10 sets) Avg SSIM Test sets won Random 27.45 0.807 0 / 10 YOLO26-depth 27.94 0.813 10 / 10 Δ (depth − random) +0.48 +0.006 — Deltas are from the unrounded averages (27.935 vs 27.452 PSNR). Small, but the depth init wins on every one of the 10 test sets.

(These are 10-set averages, AllWeather included, from the 100-epoch controlled run — so they're lower and not directly comparable to the 9-rain-only released-model numbers in the ranking table below, which come from longer training.)

On "did the random model just need longer to converge?" — both conditions ran a fixed 100 epochs, and the gap isn't a convergence-speed artifact: a 1-epoch check was a statistical wash, by 20 epochs the gap was already ~+0.49 dB, and at 100 epochs it was +0.48. It appeared early and didn't close with more training.

This does not establish why — whether depth supervision teaches geometry/spatial structure that's useful for restoration, or whether YOLO26-depth just happens to be a strong pretrained checkpoint. Only that, in this setup, the depth-initialized representation is a better starting point than random. Per-dataset deltas are in the repo; happy to paste them in a comment.

Model Params Avg PSNR Restormer 15.3M 35.10 NAFNet-Large 116M 34.16 NAFNet-Mid 14.3M 33.97 Restormer-Small 2.3M 31.98 UNet 21.5M 31.74 NAFNet-Small 1.1M 31.15 yolo26_rgb_s 12.13M 30.95 yolo26_rgb_n 5.25M 30.83 ResNet50-UNet 73.3M 30.63 ResNet34-UNet 24.5M 30.45 ResNet18-UNet 14.4M 30.23 ClearView's own analysis points to the classification stem's early downsampling (a stride-4 entry before any residual block runs) as a likely reason the ResNet-UNet baselines underperform. This project doesn't test that directly — the ResNet-UNet comparison is a whole-architecture comparison, not a pretraining ablation — but it's the context the depth-vs-random experiment sits in.

Note NAFNet-Small (1.1M, 31.15 dB): smaller and higher PSNR than yolo26_rgb_n, but ~4× slower (26.9 qps). So this isn't Pareto-dominant on every axis — the story is specifically the real-time / YOLO-derived operating point, not "more efficient in every sense."

Deployment (TensorRT fp16, 1920×1080, batch 1, RTX 4070 SUPER 12GB; baseline figures are ClearView's own on the same GPU/TRT version)

- yolo26_rgb_s — 12.13M, 30.95 dB, 92.2 qps vs ResNet34-UNet — 24.5M, 30.45 dB, 94.9 qps → same speed, ~half the params, +0.5 dB

- yolo26_rgb_n — 5.25M, 30.83 dB, 108.6 qps vs ResNet18-UNet — 14.4M, 30.23 dB, 110.3 qps → same throughput, ~1/3 the params, +0.6 dB

- Both are ~3× faster than ResNet50-UNet (73.3M, 30.63 dB, 33.1 qps), while also scoring higher PSNR.

- Restormer (rank 1 on PSNR) doesn't build under TensorRT at 1080p on this 12GB card in my setup — TensorRT reports ~14.4GB of scratch needed to fuse its attention path.

- Demonstrated: YOLO26-depth initialization beats random init for deraining in this setup (10/10 test sets, +0.48 dB), same architecture and recipe.

- Supported: the trained models sit at an attractive real-time quality/latency point relative to the ResNet-UNet baselines.

- Not demonstrated: that depth pretraining beats classification pretraining for restoration, or why depth helps. Those need experiments I haven't run.