Drag the slider. The left side is the quality-1 JPEG that HyRES starts from, JPEG at equal file size, or the original. The right side is HyRES, decoded from a real bitstream. The q1 view shows what the residual adds, but it uses far fewer bits (see the labels), so the equal-size view is the fair comparison with plain JPEG; there the differences are subtle, so look at edges and fine texture.
TL;DR
JPEG first, neural second. We encode with JPEG at quality 1, then a small learned codec encodes only the residual (original minus decoded JPEG).
Backward compatible. The JPEG part is a standard file, so any viewer can open it without the neural decoder.
Small and quick. The codec has 10.14M parameters and handles a Kodak image in about 0.76 s on one GPU.
Measured from real files. All HyRES numbers come from real bitstreams, and measuring them exposed three bugs that we fixed.
Method
Compress what JPEG gets wrong
JPEG is fast and universal but loses detail at low bitrates, while learned codecs compress far better but are large and slow. We keep JPEG as the base layer and use the learned codec only for what JPEG gets wrong.
Figure 1. Pipeline. The decoded JPEG xJ is subtracted from the input, and only the residual r goes through the learned codec. The decoder adds the decoded residual back. Thumbnails and bitrates are real outputs on kodim23 (residual shown 2×). Averaged over Kodak, the total is 1.56 bpp, and the residual takes 1.36 of it.
What the residual looks like
At quality 1, JPEG keeps only coarse 8×8 block averages. The residual holds everything else (edges, texture and color), so that is all the codec has to reproduce.
Residuals. Residuals are amplified 2× around mid-gray. The decoded residual (right) costs 1.36 bpp on average.
Residual codec
Figure 2. Residual codec (10.14M parameters, layers as in models/checkerboard.py). A scale hyperprior (Ballé et al., 2018) predicts the mean and scale of every latent. Attention and residual bottleneck blocks follow Cheng et al. (2020). A checkerboard context model (He et al., 2021) decodes the latent in two parallel passes instead of one slow pass.
Training: start sharp, then trade bits away
We train with the rate-distortion loss bpp + λ·MSE, where λ sets the trade-off between file size and error. Early runs at three λ values confirmed that a higher λ gives more bits and lower error. As a result, we split training into six phases: Phase 1 uses a high λ so the model first learns to reconstruct, then λ is lowered step by step, and each phase starts from the previous best checkpoint. We used Mini-ImageNet (60k images) on a single A40.
Figure 3. One crop, six operating points. The crop (160×160) is placed where JPEG's error is largest. The PSNR under each crop is for the whole image. A lower λ means fewer bits, and fine texture is the first thing to go.
Results
Rate-distortion on Kodak
All 24 Kodak images at full resolution (768×512), PSNR averaged per image. Published codecs come from CompressAI's benchmark files, and the JPEG curve uses the same TurboJPEG settings as the HyRES base layer.
Figure 4. Rate-distortion on Kodak. The filled blue point is the only HyRES result measured from a real bitstream. It is 2.1 dB above JPEG at the same size. However, ELIC reaches the same 37.4 dB at 0.83 bpp, about half the bits. The hollow points are the training-time estimates of the six phases; their bpp came from a rate estimate that was too low (see below), so the true points lie further right.
Speed
Figure 5. Speed. Encode plus decode seconds per Kodak image, all models on the same A40 machine, measured during the project. HyRES includes the CPU-side JPEG step.
Phase 1 at a glance
Table 1. Phase 1 at a glance. Kodak averages from the real bitstream: the JPEG base, the residual latent y and the hyper-latent z.
Per-image results (24 Kodak images)
Reproducibility
What re-measuring found
During the project, every number came from the model's training-time estimate. When we later decoded real bitstreams with the surviving weights, we found three bugs. All three are fixed, and none needs retraining.
1. Clamped residual
The decoder clamped the residual to [0, 1], but residuals are signed, so every negative correction was lost.
24.46 → 37.41 dB
2. Coding empty positions
Each checkerboard pass coded the full tensor, including the other pass's empty positions. Now each pass codes only its own positions.
1.878 → 1.563 bpp
3. Optimistic rate estimate
Training scored each latent with both passes' parameters added together, which no decoder has. It now uses the coder's own parameters.
est. 1.279 vs real 1.563 bpp
Bug 3 also means that every model trained so far optimized the wrong rate term. Therefore, retraining with the fixed estimate is the most promising next step.
Limitations
Only the Phase 1 weights survive, so phases 2 to 6 cannot be re-measured.
Kodak also selected checkpoints during training, so it is not fully held out.
Training uses MSE only, so reconstructions are smooth rather than sharp.
Image sides must be multiples of 32, and inference runs on CUDA or CPU (not Apple MPS).
Next steps
Retrain all phases with the corrected rate estimate, then report a full rate-distortion curve and a BD-rate against VVC.
Add perceptual losses (VGG, GAN) for sharper reconstructions.
Train on a larger dataset and evaluate on a held-out set beyond Kodak.
Project history
How it got here
Feb 2025
Proposal. A hybrid codec with a JPEG base and a residual coded only in important regions, chosen by a learned importance map.
Mar 2025
Design. We surveyed learned compression (2018 to 2023) and replaced the importance map with a lightweight residual codec.
Apr 2025
First training runs. Three runs with different λ and JPEG quality confirmed the trade-off and led to the multi-phase schedule.
May 2025
Six phases and refinement. λ from 0.045 down to 0.002, then a refinement network on the frozen final model.
Oct 2026
Re-measurement. Real-bitstream evaluation of the surviving weights, three codec fixes, and this page.
Citation
BibTeX
@misc{hyres2025,
title = {HyRES: Residual-Enhanced Hybrid Image Compression},
author = {Tran, Minh Khang and Dinh, Hoang Dai},
year = {2025},
howpublished = {\url{https://github.com/tmkhang1999/HyRES-Residual-Enhanced-Hybrid-Image-Compression}}
}