Add an AEIC quality and rate evaluation harness
tools/aeic/eval runs a folder of photos through the same ONNX graphs the app downloads, counts exact bytes with the app's own entropy layer and rANS coder (aeic_bytes.dart, checked byte-for-byte against the golden recordings with --selftest), and scores what the receiver sees with LPIPS, DISTS, CLIP, MS-SSIM and PSNR. Variants cover preprocessing (stretch, crops, letterbox), subject selection, the send sheet's re-roll tweaks, quantisation-step rate control (fixed steps and per-image byte budgets) and two-part encodes. compare_codecs.py compares AEIC with JPEG XL and AVIF at the same byte budgets.
This commit is contained in:
1 parent
c0702806de
commit
aafd541b0f
7 files changed
+1476
No files matched your search
@@ -9,6 +9,10 @@ Dart side is checked against.
|
||||
They are committed here so `test/services/golden/` is reproducible. Nothing in
|
||||
the app runs them.
|
||||
|
||||
To measure reconstruction quality and byte cost instead, use the
|
||||
[evaluation harness](eval/README.md), which runs on the downloaded model bundle
|
||||
and needs no research checkout.
|
||||
|
||||
| script | produces | used by |
|
||||
|---|---|---|
|
||||
| `export_golden.py` | `golden/aeic_cdf_ft32.bin`, `golden/vectors/*.gv` | `rans_coder_test.dart`, `entropy_tables_test.dart` |
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# AEIC quality evaluation
|
||||
|
||||
`eval.py` measures how good received images are and how many bytes they cost,
|
||||
so codec changes can be compared with numbers rather than by eye. It runs the
|
||||
same ONNX graphs the app downloads. Unlike the fixture scripts in the parent
|
||||
directory, it needs no research checkout.
|
||||
|
||||
For each photo and each preprocessing variant it:
|
||||
|
||||
1. prepares the 512×512 codec input (the `stretch` variant mirrors the app's `_toSquareRgb`),
|
||||
2. runs `aeic_entropy_side_fp32_op17.onnx` on CPU,
|
||||
3. counts the exact bytes with `aeic_bytes.dart`, which calls the app's own entropy layer and rANS coder,
|
||||
4. runs the decoder on `y_hat` (GPU when available),
|
||||
5. restores the aspect ratio the way the receiver does and scores the result against the region that was sent.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
python3 -m venv ~/.venvs/aeic-eval # or: uv venv ~/.venvs/aeic-eval
|
||||
~/.venvs/aeic-eval/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
|
||||
~/.venvs/aeic-eval/bin/pip install -r tools/aeic/eval/requirements.txt
|
||||
```
|
||||
|
||||
You also need `dart` on PATH (`flutter pub get` already run), and the model
|
||||
bundle. By default the bundle is read from `~/Documents/image_codec_models`;
|
||||
use `--models` to point elsewhere. The first run downloads the LPIPS, DISTS and
|
||||
CLIP weights into the torch and Hugging Face caches.
|
||||
|
||||
Check that the byte counter matches the reference coder:
|
||||
|
||||
```bash
|
||||
dart run tools/aeic/eval/aeic_bytes.dart --selftest test/services/golden/e2e/*.aeicrec
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
# Kodak (24 images, downloaded once)
|
||||
~/.venvs/aeic-eval/bin/python -I tools/aeic/eval/eval.py \
|
||||
--download-kodak ~/aeic-data/kodak --out ~/aeic-runs/kodak-baseline \
|
||||
--variants stretch,crop,crop43,pad
|
||||
|
||||
# Your own photos; Kodak alone is not representative of what people send
|
||||
~/.venvs/aeic-eval/bin/python -I tools/aeic/eval/eval.py \
|
||||
--images ~/aeic-data/mesh-photos --out ~/aeic-runs/mesh-baseline
|
||||
```
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
| `--variants` | comma list of variants to compare (below) |
|
||||
| `--rois FILE` | JSON `{image stem: [l, t, r, b]}` as fractions; adds subject-region (`roi_*`) scores and limits the run to listed images. `rois/kodak.json` has 17 Kodak subjects |
|
||||
| `--display N` | long side of the receiver view that is scored (default 512; use 1024 for two-part variants) |
|
||||
| `--decoder` | score a different decoder export, e.g. fp16 instead of int8 |
|
||||
| `--limit N` | first N images only |
|
||||
| `--cpu` | run the decoder on CPU (much slower) |
|
||||
|
||||
| variant | what is sent |
|
||||
|---|---|
|
||||
| `stretch` | the whole photo stretched to the square, as the app does |
|
||||
| `crop`, `crop43`, `pad` | centre square crop; crop to at most 4:3 then stretch; letterbox on grey |
|
||||
| `reroll1`..`reroll5` | the whole photo with the send sheet's "Try another version" tweaks (`kImageSendRerollTweaks`; keep the two tables in step) |
|
||||
| `select`, `select_sq` | only the `--rois` box; the box grown to a square |
|
||||
| `q<step>` | quantisation step `<step>` instead of 1 (e.g. `q0.7` spends more bytes) |
|
||||
| `fit<bytes>` | per image, the finest step from the 16-step ladder that fits `<bytes>` (e.g. `fit157` = one packet) |
|
||||
| `hd_tiles`, `hd_detail` | two encodes per photo: two overlapping halves; or the whole photo plus the `--rois` box, blended |
|
||||
|
||||
`q`/`fit` re-run the four context stages with the decode-side entropy graph;
|
||||
at step 1 they reproduce the app's bytes exactly. Two-part byte counts include
|
||||
the extra stream header such a format would need.
|
||||
|
||||
To add an experiment, write a function returning `Prepared` and register it in
|
||||
`VARIANTS`.
|
||||
|
||||
## Comparing with JPEG XL and AVIF
|
||||
|
||||
`compare_codecs.py` sweeps AEIC over every quantisation step and fits JPEG XL
|
||||
and AVIF to the same budgets (16–192 px thumbnails, best size per image, scored
|
||||
after upscaling). AVIF is counted both as a real `.avif` file and as its AV1
|
||||
data alone. Needs `imagecodecs` (in `requirements.txt`).
|
||||
|
||||
```bash
|
||||
~/.venvs/aeic-eval/bin/python -I tools/aeic/eval/compare_codecs.py \
|
||||
--images ~/aeic-data/kodak --out ~/aeic-runs/rd
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
| file | contents |
|
||||
|---|---|
|
||||
| `report.html` | source next to each variant's reconstruction, with bytes, packets, LPIPS and CLIP; a red border marks CLIP < 0.80 |
|
||||
| `results.csv` | one row per image × variant |
|
||||
| `summary.json` | per-variant mean/median/min/max, packet histogram, run metadata |
|
||||
| `recon/`, `ref/` | the scored images, at the receiver's aspect ratio |
|
||||
| `tensors/*.aeic` | the bitstreams, exactly as the app would send them |
|
||||
|
||||
## Reading the metrics
|
||||
|
||||
| metric | better | what it catches |
|
||||
|---|---|---|
|
||||
| `bytes`, `data_chunks` | lower | airtime; one chunk carries up to 157 image bytes |
|
||||
| `lpips`, `dists` | lower | perceptual difference; the main quality score for a generative decoder |
|
||||
| `clip` | higher | whether the reconstruction shows the same *things* as the sent region; low values are the "fox came back as a sheep" failures |
|
||||
| `clip_full` | higher | the same, against the whole original photo, so a crop pays for what it dropped |
|
||||
| `psnr`, `ms_ssim` | higher | pixel fidelity; included for reference, but weak signals for this codec |
|
||||
|
||||
`1-pkt` in the console summary is the share of images that fit in a single data
|
||||
chunk. Each image also gets one parity packet when recovery is on.
|
||||
|
||||
## Caveats
|
||||
|
||||
- Desktop ORT on CPU is not bit-identical to ORT on a phone, so an occasional rounding difference can change a byte count by one. The byte counter itself matches the reference coder exactly (see `--selftest`).
|
||||
- The `stretch` resize approximates `dart:ui`'s `FilterQuality.medium`; it is not pixel-identical.
|
||||
- CLIP and LPIPS are proxies. Look at the images flagged in `report.html` before drawing conclusions.
|
||||
@@ -0,0 +1,240 @@
|
||||
// Byte counter for the AEIC evaluation harness (`eval.py`).
|
||||
//
|
||||
// Python runs the encode-side ONNX graph; this turns its tensors into the exact
|
||||
// bitstream the app would send, using the app's own entropy layer and rANS
|
||||
// coder rather than a port. A port could drift; this cannot, because it is the
|
||||
// shipping code.
|
||||
//
|
||||
// Usage (from the repository root):
|
||||
//
|
||||
// dart run tools/aeic/eval/aeic_bytes.dart <cdf_table.bin> <tensors.f32>...
|
||||
// dart run tools/aeic/eval/aeic_bytes.dart --selftest <recording.aeicrec>...
|
||||
//
|
||||
// Each `.f32` file holds little-endian float32 tensors back to back, in this
|
||||
// order: z_q [1,128,4,4], yq0..yq3 [1,256,16,16], sc0..sc3 [1,256,16,16]. The
|
||||
// bitstream is written next to it as `.aeic`, and one JSON object per input is
|
||||
// printed to stdout.
|
||||
//
|
||||
// `--selftest` replays the golden `.aeicrec` recordings in
|
||||
// `test/services/golden/e2e/` and checks that the output is byte-identical to
|
||||
// the reference C++ bitstream, which shows the harness counts what the app
|
||||
// sends.
|
||||
|
||||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
import 'dart:typed_data';
|
||||
|
||||
import 'package:meshcore_open/services/entropy_tables.dart';
|
||||
import 'package:meshcore_open/services/image_chunk_transport.dart';
|
||||
import 'package:meshcore_open/services/image_codec_entropy.dart';
|
||||
import 'package:meshcore_open/services/rans_coder.dart';
|
||||
|
||||
const int _resolution = 512;
|
||||
|
||||
Future<void> main(List<String> args) async {
|
||||
if (args.isEmpty) {
|
||||
stderr.writeln(
|
||||
'usage: aeic_bytes.dart <cdf_table.bin> <tensors.f32>...\n'
|
||||
' aeic_bytes.dart --selftest <recording.aeicrec>...',
|
||||
);
|
||||
exitCode = 64;
|
||||
return;
|
||||
}
|
||||
if (args.first == '--selftest') {
|
||||
exitCode = await _selfTest(args.skip(1).toList());
|
||||
return;
|
||||
}
|
||||
|
||||
final tables = EntropyTables.parse(File(args.first).readAsBytesSync());
|
||||
final geometry = AeicEntropyGeometry.forResolution(_resolution);
|
||||
for (final path in args.skip(1)) {
|
||||
final tensors = _readTensorFile(path, geometry);
|
||||
final stream = await _encode(tables, geometry, tensors);
|
||||
final outPath = path.replaceFirst(RegExp(r'\.f32$'), '.aeic');
|
||||
File(outPath).writeAsBytesSync(stream);
|
||||
stdout.writeln(
|
||||
jsonEncode(<String, Object>{
|
||||
'file': path,
|
||||
'bitstream': outPath,
|
||||
'bytes': stream.length,
|
||||
'data_chunks': imageDataChunkCount(stream.length),
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Future<Uint8List> _encode(
|
||||
EntropyTables tables,
|
||||
AeicEntropyGeometry geometry,
|
||||
AeicEncodeSideTensors tensors,
|
||||
) {
|
||||
final codec = AeicEntropyCodec(
|
||||
geometry: geometry,
|
||||
network: _FixedNetwork(tensors),
|
||||
coders: _Coders(tables),
|
||||
);
|
||||
// The network ignores its input; encode() only needs a correctly sized RGB
|
||||
// buffer to get past its own conversion step.
|
||||
return codec.encode(Uint8List(_resolution * _resolution * 3));
|
||||
}
|
||||
|
||||
AeicEncodeSideTensors _readTensorFile(
|
||||
String path,
|
||||
AeicEntropyGeometry geometry,
|
||||
) {
|
||||
final bytes = File(path).readAsBytesSync();
|
||||
final z = geometry.zElements;
|
||||
final y = geometry.yElements;
|
||||
final expected = (z + 8 * y) * 4;
|
||||
if (bytes.length != expected) {
|
||||
throw FormatException(
|
||||
'$path is ${bytes.length} bytes, expected $expected '
|
||||
'(z_q + 4 yq + 4 sc as float32)',
|
||||
);
|
||||
}
|
||||
final all = Float32List.sublistView(bytes);
|
||||
Float32List at(int offset, int length) =>
|
||||
Float32List.sublistView(all, offset, offset + length);
|
||||
return AeicEncodeSideTensors(
|
||||
zQ: at(0, z),
|
||||
yQ: <Float32List>[for (var i = 0; i < 4; i++) at(z + i * y, y)],
|
||||
scales: <Float32List>[for (var i = 0; i < 4; i++) at(z + (4 + i) * y, y)],
|
||||
);
|
||||
}
|
||||
|
||||
Future<int> _selfTest(List<String> recordings) async {
|
||||
if (recordings.isEmpty) {
|
||||
stderr.writeln('--selftest needs at least one .aeicrec file');
|
||||
return 64;
|
||||
}
|
||||
final geometry = AeicEntropyGeometry.forResolution(_resolution);
|
||||
var failures = 0;
|
||||
for (final path in recordings) {
|
||||
final r = _Recording.load(path);
|
||||
final tables = EntropyTables.parse(
|
||||
File(
|
||||
'${File(path).parent.parent.path}/${r.meta['cdf_table']}',
|
||||
).readAsBytesSync(),
|
||||
);
|
||||
final tensors = AeicEncodeSideTensors(
|
||||
zQ: r.f32('enc/z_q'),
|
||||
yQ: <Float32List>[for (var i = 0; i < 4; i++) r.f32('enc/yq$i')],
|
||||
scales: <Float32List>[for (var i = 0; i < 4; i++) r.f32('enc/sc$i')],
|
||||
);
|
||||
final got = await _encode(tables, geometry, tensors);
|
||||
final want = r.u8('enc/bitstream');
|
||||
final ok =
|
||||
got.length == want.length &&
|
||||
Iterable<int>.generate(got.length).every((i) => got[i] == want[i]);
|
||||
if (!ok) failures++;
|
||||
stdout.writeln(
|
||||
'${ok ? 'ok ' : 'FAIL'} ${r.meta['stem']}: ${got.length} B '
|
||||
'(reference ${want.length} B)',
|
||||
);
|
||||
}
|
||||
return failures == 0 ? 0 : 1;
|
||||
}
|
||||
|
||||
class _FixedNetwork implements AeicEntropyNetwork {
|
||||
_FixedNetwork(this.tensors);
|
||||
|
||||
final AeicEncodeSideTensors tensors;
|
||||
|
||||
@override
|
||||
bool get supportsDecodeSide => false;
|
||||
|
||||
@override
|
||||
Future<AeicEncodeSideTensors> runEncodeSide(Float32List imageChw) async =>
|
||||
tensors;
|
||||
|
||||
@override
|
||||
Future<Float32List> runHyperSynthesis(Float32List zQ) =>
|
||||
throw UnsupportedError('encode only');
|
||||
|
||||
@override
|
||||
Future<AeicStageParams> runStage(int stage, Float32List base) =>
|
||||
throw UnsupportedError('encode only');
|
||||
}
|
||||
|
||||
class _Coders implements AeicRansCoderFactory {
|
||||
_Coders(this.tables);
|
||||
|
||||
final EntropyTables tables;
|
||||
|
||||
@override
|
||||
AeicRansEncoder createEncoder() => _Encoder(RansEncoder(tables));
|
||||
|
||||
@override
|
||||
AeicRansDecoder createDecoder(Uint8List bitstream) =>
|
||||
throw UnsupportedError('encode only');
|
||||
}
|
||||
|
||||
class _Encoder implements AeicRansEncoder {
|
||||
_Encoder(this._encoder);
|
||||
|
||||
final RansEncoder _encoder;
|
||||
|
||||
@override
|
||||
void pushSymbols(Int16List symbols, Int16List indexes, int cdfGroup) =>
|
||||
_encoder.encodeWithIndexes(symbols, indexes, cdfGroup);
|
||||
|
||||
@override
|
||||
Uint8List finish() => _encoder.finish();
|
||||
}
|
||||
|
||||
/// Minimal reader for the `.aeicrec` container (see
|
||||
/// `test/services/image_codec_e2e_test.dart` for the full one).
|
||||
class _Recording {
|
||||
_Recording(this.index, this.bytes)
|
||||
: _entries = <String, Map<String, dynamic>>{
|
||||
for (final e
|
||||
in (index['entries'] as List<dynamic>).cast<Map<String, dynamic>>())
|
||||
e['name'] as String: e,
|
||||
};
|
||||
|
||||
final Map<String, dynamic> index;
|
||||
final Uint8List bytes;
|
||||
final Map<String, Map<String, dynamic>> _entries;
|
||||
|
||||
static _Recording load(String path) {
|
||||
final bytes = File(path).readAsBytesSync();
|
||||
final bd = ByteData.sublistView(bytes);
|
||||
if (ascii.decode(bytes.sublist(0, 8)) != 'AEICREC1') {
|
||||
throw FormatException('bad .aeicrec magic in $path');
|
||||
}
|
||||
final indexOffset = bd.getUint64(16, Endian.little);
|
||||
final indexLength = bd.getUint32(24, Endian.little);
|
||||
final index =
|
||||
jsonDecode(
|
||||
utf8.decode(
|
||||
bytes.sublist(indexOffset, indexOffset + indexLength),
|
||||
),
|
||||
)
|
||||
as Map<String, dynamic>;
|
||||
return _Recording(index, bytes);
|
||||
}
|
||||
|
||||
Map<String, dynamic> get meta => index['meta'] as Map<String, dynamic>;
|
||||
|
||||
Map<String, dynamic> _entry(String name) {
|
||||
final e = _entries[name];
|
||||
if (e == null) throw StateError('no entry "$name" in recording');
|
||||
return e;
|
||||
}
|
||||
|
||||
Float32List f32(String name) {
|
||||
final e = _entry(name);
|
||||
final offset = e['offset'] as int;
|
||||
return Float32List.sublistView(
|
||||
bytes,
|
||||
offset,
|
||||
offset + (e['length'] as int),
|
||||
);
|
||||
}
|
||||
|
||||
Uint8List u8(String name) {
|
||||
final e = _entry(name);
|
||||
final offset = e['offset'] as int;
|
||||
return Uint8List.sublistView(bytes, offset, offset + (e['length'] as int));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rate-distortion comparison: AEIC (as shipped and with step rate control)
|
||||
against JPEG XL and AVIF at the same byte budgets.
|
||||
|
||||
python -I tools/aeic/eval/compare_codecs.py --images ~/aeic-data/kodak --out ~/aeic-runs/rd
|
||||
|
||||
AEIC is swept over every quantisation step in STEP_LADDER (step 1.0 is the
|
||||
stream the app sends today) and also fitted per image to 1 and 2 packets.
|
||||
JPEG XL and AVIF are fitted per image to each byte budget: for every candidate
|
||||
size the quality setting is searched to fit, the result is decoded and
|
||||
upscaled to the display size, and the size with the lowest LPIPS is kept. That
|
||||
choice uses the original, which the sender has, so it is a fair best case for
|
||||
the conventional codecs rather than an oracle.
|
||||
|
||||
AVIF is counted twice. `avif` is the real .avif file, whose ftyp+meta boxes
|
||||
cost about 275 bytes before any image data. `avif_av1` counts only the AV1
|
||||
payload (the mdat contents), as if the receiver rebuilt the fixed container
|
||||
from a template; that is AVIF's best case for a custom wire format.
|
||||
|
||||
Everything is scored against the source at the same display size (long side
|
||||
512, the AEIC output size). Needs `imagecodecs` with JPEG XL support on top of
|
||||
the eval.py requirements.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
_spec = importlib.util.spec_from_file_location("aeic_eval", HERE / "eval.py")
|
||||
ev = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["aeic_eval"] = ev # dataclasses look their module up here
|
||||
_spec.loader.exec_module(ev) # type: ignore[union-attr]
|
||||
|
||||
import imagecodecs # noqa: E402 (after eval.py, which loads torch first)
|
||||
from concurrent.futures import ThreadPoolExecutor # noqa: E402
|
||||
|
||||
POOL = ThreadPoolExecutor(max_workers=12)
|
||||
|
||||
PACKET_BUDGETS = {"1 packet": ev.CHUNK_FIRST_CAPACITY,
|
||||
"2 packets": ev.CHUNK_FIRST_CAPACITY + ev.CHUNK_CAPACITY}
|
||||
BUDGETS = [100, 157, 200, 250, 315, 400, 500, 700]
|
||||
SIZES = [16, 24, 32, 40, 48, 56, 64, 80, 96, 128, 160, 192]
|
||||
JXL_EFFORT = 9
|
||||
AVIF_SPEED = 2
|
||||
DISPLAY = ev.SIZE
|
||||
|
||||
|
||||
def display_size(img: Image.Image, long_side: int) -> tuple[int, int]:
|
||||
s = long_side / max(img.size)
|
||||
return (max(1, round(img.width * s)), max(1, round(img.height * s)))
|
||||
|
||||
|
||||
def mdat_payload(avif: bytes) -> int:
|
||||
"""Size of the AV1 data in an .avif file (its mdat box contents)."""
|
||||
i = 0
|
||||
while i + 8 <= len(avif):
|
||||
size, kind = struct.unpack(">I4s", avif[i:i + 8])
|
||||
header = 8
|
||||
if size == 1:
|
||||
size = struct.unpack(">Q", avif[i + 8:i + 16])[0]
|
||||
header = 16
|
||||
elif size == 0:
|
||||
size = len(avif) - i
|
||||
if kind == b"mdat":
|
||||
return size - header
|
||||
i += size
|
||||
raise ValueError("no mdat box")
|
||||
|
||||
|
||||
def _jxl(arr: np.ndarray, q: float) -> bytes:
|
||||
return imagecodecs.jpegxl_encode(arr, distance=q, effort=JXL_EFFORT)
|
||||
|
||||
|
||||
def _avif(arr: np.ndarray, q: float) -> bytes:
|
||||
return imagecodecs.avif_encode(arr, level=int(q), speed=AVIF_SPEED)
|
||||
|
||||
|
||||
# name -> (encode(arr, q), decode, bytes counted, q for the smallest file,
|
||||
# q for the largest, integer q?)
|
||||
CODECS = {
|
||||
"jxl": (_jxl, imagecodecs.jpegxl_decode, len, 25.0, 0.05, False),
|
||||
"avif": (_avif, imagecodecs.avif_decode, len, 0, 100, True),
|
||||
"avif_av1": (_avif, imagecodecs.avif_decode, mdat_payload, 0, 100, True),
|
||||
}
|
||||
|
||||
|
||||
def codec_fit(name: str, src: Image.Image, long_side: int, budget: int) -> tuple[int, Image.Image] | None:
|
||||
"""Best stream of codec `name` for `src` at `long_side` px within `budget`.
|
||||
|
||||
Returns (counted bytes, decoded image), or None if even the lowest quality
|
||||
does not fit.
|
||||
"""
|
||||
enc, dec, count, q_small, q_large, integer = CODECS[name]
|
||||
small = np.asarray(src.resize(display_size(src, long_side), Image.LANCZOS))
|
||||
data = enc(small, q_small)
|
||||
if count(data) > budget:
|
||||
return None
|
||||
best = data
|
||||
lo, hi = q_small, q_large # lo always fits
|
||||
for _ in range(12):
|
||||
if integer:
|
||||
if abs(hi - lo) <= 1:
|
||||
break
|
||||
mid = (lo + hi) // 2
|
||||
else:
|
||||
mid = (lo * hi) ** 0.5 # JPEG XL distance acts roughly logarithmically
|
||||
data = enc(small, mid)
|
||||
if count(data) <= budget:
|
||||
best, lo = data, mid
|
||||
else:
|
||||
hi = mid
|
||||
decoded = Image.fromarray(np.asarray(dec(best))[..., :3], "RGB")
|
||||
return count(best), decoded
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--images", type=Path, required=True)
|
||||
ap.add_argument("--out", type=Path, required=True)
|
||||
ap.add_argument("--models", type=Path, default=ev.DEFAULT_MODELS)
|
||||
ap.add_argument("--limit", type=int)
|
||||
args = ap.parse_args()
|
||||
|
||||
images = ev.find_images(args.images, args.limit)
|
||||
out: Path = args.out
|
||||
work = out / "tensors"
|
||||
(out / "img").mkdir(parents=True, exist_ok=True)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
import torch
|
||||
gpu = torch.cuda.is_available()
|
||||
codec = ev.Codec(args.models, args.models / ev.DECODER_GRAPH, gpu)
|
||||
metrics = ev.Metrics("cuda" if gpu else "cpu")
|
||||
|
||||
# ---- AEIC: every step on the ladder, one Dart run for all byte counts.
|
||||
srcs, aeic_y = {}, {}
|
||||
files = []
|
||||
for path in images:
|
||||
stem = path.stem
|
||||
src = ev.load_source(path)
|
||||
srcs[stem] = src
|
||||
side = codec.encode_side(ev.variant_stretch(src).square)
|
||||
for step in ev.STEP_LADDER:
|
||||
t = codec.encode_with_step(side, step)
|
||||
tf = work / f"{stem}.s{step:.2f}.f32"
|
||||
ev.write_tensor_file(tf, t)
|
||||
aeic_y[(stem, step)] = t["y_hat"]
|
||||
files.append(tf)
|
||||
print(f"encoded {stem}", file=sys.stderr)
|
||||
nbytes = {k: v["bytes"] for k, v in ev.count_bytes(args.models / ev.CDF_TABLE, files).items()}
|
||||
for tf in files:
|
||||
tf.unlink()
|
||||
|
||||
rows = []
|
||||
emb = {}
|
||||
|
||||
def score(stem: str, shown: Image.Image) -> dict:
|
||||
src = srcs[stem]
|
||||
ref = src.resize(display_size(src, DISPLAY), Image.LANCZOS)
|
||||
if stem not in emb:
|
||||
emb[stem] = metrics.clip_embed(src)
|
||||
s = metrics.score(ref, shown.resize(ref.size, Image.BICUBIC), emb[stem])
|
||||
s.pop("clip_full")
|
||||
return s
|
||||
|
||||
aeic_shown = {}
|
||||
for path in images:
|
||||
stem = path.stem
|
||||
src = srcs[stem]
|
||||
for step in ev.STEP_LADDER:
|
||||
recon = codec.decode(aeic_y[(stem, step)])
|
||||
shown = recon.resize(display_size(src, DISPLAY), Image.LANCZOS)
|
||||
aeic_shown[(stem, step)] = shown
|
||||
b = nbytes[str(work / f"{stem}.s{step:.2f}.f32")]
|
||||
rows.append({"image": stem, "codec": "aeic", "step": step, "bytes": b, **score(stem, shown)})
|
||||
print(f"scored AEIC {stem}", file=sys.stderr)
|
||||
|
||||
# ---- Conventional codecs: per image and budget, the best size that fits.
|
||||
for name in CODECS:
|
||||
for path in images:
|
||||
stem = path.stem
|
||||
src = srcs[stem]
|
||||
for budget in BUDGETS:
|
||||
best = None
|
||||
# imagecodecs releases the GIL, so the size search runs in threads.
|
||||
fits = list(POOL.map(lambda ls: codec_fit(name, src, ls, budget), SIZES))
|
||||
for long_side, fit in zip(SIZES, fits):
|
||||
if fit is None:
|
||||
continue
|
||||
nb, small = fit
|
||||
shown = small.resize(display_size(src, DISPLAY), Image.BICUBIC)
|
||||
ref = src.resize(shown.size, Image.LANCZOS)
|
||||
lp = float(metrics.lpips(metrics._t(shown), metrics._t(ref)))
|
||||
if best is None or lp < best[0]:
|
||||
best = (lp, long_side, nb, shown)
|
||||
if best is None:
|
||||
continue
|
||||
_, long_side, nb, shown = best
|
||||
rows.append({"image": stem, "codec": name, "budget": budget, "px": long_side,
|
||||
"bytes": nb, **score(stem, shown)})
|
||||
if budget in PACKET_BUDGETS.values():
|
||||
shown.save(out / "img" / f"{stem}.{name}{budget}.png")
|
||||
print(f"scored {name} {stem}", file=sys.stderr)
|
||||
|
||||
# ---- AEIC fitted to the packet budgets, picked from the sweep.
|
||||
for stem in srcs:
|
||||
for label, budget in PACKET_BUDGETS.items():
|
||||
cands = [r for r in rows if r["codec"] == "aeic" and r["image"] == stem and r["bytes"] <= budget]
|
||||
if not cands:
|
||||
cands = [max((r for r in rows if r["codec"] == "aeic" and r["image"] == stem),
|
||||
key=lambda r: r["step"])]
|
||||
pick = min(cands, key=lambda r: r["step"])
|
||||
rows.append({**pick, "codec": f"aeic_fit{budget}", "budget": budget})
|
||||
aeic_shown[(stem, pick["step"])].save(out / "img" / f"{stem}.aeic{budget}.png")
|
||||
aeic_shown[(stem, 1.0)].save(out / "img" / f"{stem}.aeic_today.png")
|
||||
srcs[stem].resize(display_size(srcs[stem], DISPLAY), Image.LANCZOS).save(out / "img" / f"{stem}.src.png")
|
||||
|
||||
(out / "rows.json").write_text(json.dumps(rows, indent=0))
|
||||
|
||||
def mean(rs, k):
|
||||
return float(np.mean([r[k] for r in rs]))
|
||||
|
||||
def line(name, rs):
|
||||
return (f"{name:<22}{mean(rs, 'bytes'):>7.0f}{mean(rs, 'lpips'):>8.3f}{mean(rs, 'dists'):>8.3f}"
|
||||
f"{mean(rs, 'clip'):>7.3f}{mean(rs, 'ms_ssim'):>8.3f}{mean(rs, 'psnr'):>7.2f}")
|
||||
|
||||
print(f"\n{'':<22}{'bytes':>7}{'lpips':>8}{'dists':>8}{'clip':>7}{'msssim':>8}{'psnr':>7}")
|
||||
print(line("AEIC today (step 1)", [r for r in rows if r["codec"] == "aeic" and r["step"] == 1.0]))
|
||||
for label, budget in PACKET_BUDGETS.items():
|
||||
print(f"-- {label} ({budget} B)")
|
||||
print(line(" AEIC step-fit", [r for r in rows if r["codec"] == f"aeic_fit{budget}"]))
|
||||
for name in CODECS:
|
||||
rs = [r for r in rows if r["codec"] == name and r["budget"] == budget]
|
||||
print(line(f" {name} ({len(rs)} img)", rs) if rs else f" {name}: does not fit")
|
||||
print("\nAEIC curve")
|
||||
for step in ev.STEP_LADDER:
|
||||
print(line(f" step {step:.2f}", [r for r in rows if r["codec"] == "aeic" and r["step"] == step]))
|
||||
for name in CODECS:
|
||||
print(f"{name} curve")
|
||||
for budget in BUDGETS:
|
||||
rs = [r for r in rows if r["codec"] == name and r["budget"] == budget]
|
||||
if rs:
|
||||
print(line(f" <= {budget} B ({len(rs)} img)", rs))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,831 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Quality/rate evaluation harness for the AEIC image codec.
|
||||
|
||||
Runs a folder of photos through the same three ONNX graphs the app ships,
|
||||
counts the exact bytes with the app's own Dart rANS coder (aeic_bytes.dart),
|
||||
and scores what the receiver would see.
|
||||
|
||||
python -I tools/aeic/eval/eval.py --images ~/photos --out /tmp/aeic-eval
|
||||
|
||||
See README.md in this directory for setup and how to read the output.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
# torch must be imported before onnxruntime so the CUDA libraries that ship in
|
||||
# the torch wheels are already loaded when ORT's CUDA provider looks for them.
|
||||
import torch # noqa: E402 (import order is deliberate)
|
||||
import onnxruntime as ort # noqa: E402
|
||||
|
||||
SIZE = 512
|
||||
REPO = Path(__file__).resolve().parents[3]
|
||||
DART_TOOL = Path(__file__).resolve().parent / "aeic_bytes.dart"
|
||||
DEFAULT_MODELS = Path.home() / "Documents" / "image_codec_models"
|
||||
SIDE_GRAPH = "aeic_entropy_side_fp32_op17.onnx"
|
||||
DECODER_GRAPH = "aeic_decoder_qdq_conv_pct.onnx"
|
||||
CDF_TABLE = "aeic_cdf_ft32.bin"
|
||||
CONTEXT_GRAPH = "aeic_entropy_decode_fp32_op17.onnx"
|
||||
|
||||
|
||||
def _stage_masks() -> list[np.ndarray]:
|
||||
"""The four checkerboard masks, as AeicMaskSet.isLive defines them."""
|
||||
perm = [0, 3, 2, 1]
|
||||
c = np.arange(256)[:, None, None] // 64
|
||||
yy = np.arange(16)[None, :, None]
|
||||
xx = np.arange(16)[None, None, :]
|
||||
micro = ((yy & 1) << 1) | (xx & 1)
|
||||
return [((c ^ perm[s]) == micro)[None] for s in range(4)]
|
||||
|
||||
|
||||
STAGE_MASKS = _stage_masks()
|
||||
|
||||
# Quantisation steps a stream could signal in 4 bits (the metadata byte's rate
|
||||
# and resolution fields). 1.0 is today's ft32 stream.
|
||||
STEP_LADDER = [0.40, 0.45, 0.50, 0.55, 0.60, 0.66, 0.72, 0.79,
|
||||
0.86, 0.93, 1.00, 1.10, 1.20, 1.32, 1.45, 1.60]
|
||||
IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".webp", ".bmp", ".tif", ".tiff"}
|
||||
|
||||
# Mirrors lib/services/image_chunk_transport.dart. aeic_bytes.dart reports the
|
||||
# authoritative count; these are only used for the summary's packet totals.
|
||||
PARITY_CHUNKS = 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Preprocessing variants
|
||||
# --------------------------------------------------------------------------
|
||||
#
|
||||
# A variant turns the (EXIF-rotated) source photo into the 512x512 codec input
|
||||
# and says which part of the source it represents, so the reconstruction can
|
||||
# be compared with that region at its true aspect ratio, which is what the
|
||||
# receiver displays.
|
||||
|
||||
|
||||
@dataclass
|
||||
class Prepared:
|
||||
square: Image.Image # 512x512 RGB, what the encoder sees
|
||||
region: tuple[int, int, int, int] # source box (l, t, r, b) it represents
|
||||
content_box: tuple[int, int, int, int] # box inside `square` holding it
|
||||
# For multi-part variants: width in source pixels of the cross-fade along
|
||||
# this part's edges that lie inside the picture (not on its border).
|
||||
feather: int = 0
|
||||
|
||||
|
||||
def _resize(img: Image.Image, size: tuple[int, int]) -> Image.Image:
|
||||
# The app draws with dart:ui FilterQuality.medium: bilinear sampling over
|
||||
# mipmaps. A box pre-reduce followed by bilinear is the closest PIL match.
|
||||
return img.resize(size, Image.BILINEAR, reducing_gap=2.0)
|
||||
|
||||
|
||||
def _full(img: Image.Image) -> tuple[int, int, int, int]:
|
||||
return (0, 0, img.width, img.height)
|
||||
|
||||
|
||||
def variant_stretch(img: Image.Image) -> Prepared:
|
||||
"""What the app does today: the whole frame squeezed into the square."""
|
||||
return Prepared(_resize(img, (SIZE, SIZE)), _full(img), (0, 0, SIZE, SIZE))
|
||||
|
||||
|
||||
def _centre_crop_box(img: Image.Image, max_ratio: float) -> tuple[int, int, int, int]:
|
||||
w, h = img.size
|
||||
ratio = w / h
|
||||
if ratio > max_ratio:
|
||||
cw = round(h * max_ratio)
|
||||
left = (w - cw) // 2
|
||||
return (left, 0, left + cw, h)
|
||||
if ratio < 1 / max_ratio:
|
||||
ch = round(w * max_ratio)
|
||||
top = (h - ch) // 2
|
||||
return (0, top, w, top + ch)
|
||||
return _full(img)
|
||||
|
||||
|
||||
def variant_crop(img: Image.Image) -> Prepared:
|
||||
"""Centre square crop: no distortion, loses the edges of the frame."""
|
||||
box = _centre_crop_box(img, 1.0)
|
||||
return Prepared(_resize(img.crop(box), (SIZE, SIZE)), box, (0, 0, SIZE, SIZE))
|
||||
|
||||
|
||||
def variant_crop43(img: Image.Image) -> Prepared:
|
||||
"""Crop to at most 4:3 (or 3:4), then stretch what is left."""
|
||||
box = _centre_crop_box(img, 4 / 3)
|
||||
return Prepared(_resize(img.crop(box), (SIZE, SIZE)), box, (0, 0, SIZE, SIZE))
|
||||
|
||||
|
||||
def variant_pad(img: Image.Image) -> Prepared:
|
||||
"""Letterbox on mid-grey: no distortion, spends latent area on the bars."""
|
||||
w, h = img.size
|
||||
scale = SIZE / max(w, h)
|
||||
nw, nh = max(1, round(w * scale)), max(1, round(h * scale))
|
||||
canvas = Image.new("RGB", (SIZE, SIZE), (128, 128, 128))
|
||||
left, top = (SIZE - nw) // 2, (SIZE - nh) // 2
|
||||
canvas.paste(_resize(img, (nw, nh)), (left, top))
|
||||
return Prepared(canvas, _full(img), (left, top, left + nw, top + nh))
|
||||
|
||||
|
||||
# Re-roll tweaks: (zoom, dx, dy) as fractions of the source. Mirrors
|
||||
# kImageSendRerollTweaks / rerollSourceRect in
|
||||
# lib/widgets/image_send_codec_binding.dart; keep the two in step.
|
||||
REROLL_TWEAKS = [
|
||||
(1.0, 0.0, 0.0),
|
||||
(1.02, 0.005, 0.0),
|
||||
(1.02, -0.005, 0.005),
|
||||
(1.04, 0.0, -0.005),
|
||||
(1.03, 0.01, 0.01),
|
||||
(1.03, -0.01, -0.01),
|
||||
]
|
||||
|
||||
|
||||
def reroll_source_rect(w: int, h: int, variant: int) -> tuple[float, float, float, float]:
|
||||
"""Source rect for re-roll `variant`: a near-invisible zoom and shift."""
|
||||
zoom, dx, dy = REROLL_TWEAKS[variant]
|
||||
cw, ch = w / zoom, h / zoom
|
||||
left = min(max((w - cw) / 2 + dx * w, 0.0), w - cw)
|
||||
top = min(max((h - ch) / 2 + dy * h, 0.0), h - ch)
|
||||
return (left, top, left + cw, top + ch)
|
||||
|
||||
|
||||
def variant_reroll(img: Image.Image, variant: int) -> Prepared:
|
||||
"""Whole photo with re-roll tweak `variant`, stretched like `stretch`.
|
||||
|
||||
Scored against the full frame: the tweak only drops a percent or two at
|
||||
the edges, which is what the sender sees as "the whole photo".
|
||||
"""
|
||||
l, t, r, b = reroll_source_rect(img.width, img.height, variant)
|
||||
# PIL's resize box takes float source coordinates, like drawImageRect.
|
||||
square = img.resize((SIZE, SIZE), Image.BILINEAR, box=(l, t, r, b), reducing_gap=2.0)
|
||||
return Prepared(square, _full(img), (0, 0, SIZE, SIZE))
|
||||
|
||||
|
||||
VARIANTS: dict[str, Callable[[Image.Image], Prepared]] = {
|
||||
"stretch": variant_stretch,
|
||||
"crop": variant_crop,
|
||||
"crop43": variant_crop43,
|
||||
"pad": variant_pad,
|
||||
**{f"reroll{k}": (lambda img, k=k: variant_reroll(img, k)) for k in range(1, len(REROLL_TWEAKS))},
|
||||
}
|
||||
|
||||
# Need a box per image (--rois), so they are not in VARIANTS.
|
||||
SELECT = "select"
|
||||
SELECT_SQUARE = "select_sq"
|
||||
SELECT_VARIANTS = (SELECT, SELECT_SQUARE)
|
||||
|
||||
Box = tuple[int, int, int, int]
|
||||
|
||||
|
||||
def roi_box(img: Image.Image, frac: list[float]) -> Box:
|
||||
"""Fractional [l, t, r, b] -> pixel box in the (EXIF-rotated) source."""
|
||||
l, t, r, b = frac
|
||||
w, h = img.size
|
||||
return (round(l * w), round(t * h), round(r * w), round(b * h))
|
||||
|
||||
|
||||
def variant_select(img: Image.Image, box: Box) -> Prepared:
|
||||
"""The user's selection, stretched into the square like `stretch`.
|
||||
|
||||
The aspect nibble restores the box's shape on the receiver, so this is the
|
||||
app's existing pipeline applied to a crop the user chose.
|
||||
"""
|
||||
return Prepared(_resize(img.crop(box), (SIZE, SIZE)), box, (0, 0, SIZE, SIZE))
|
||||
|
||||
|
||||
def square_box(img: Image.Image, box: Box) -> Box:
|
||||
"""Grow `box` to a square around its centre, shifted to stay in the image.
|
||||
|
||||
Models a selection tool locked to 1:1, so the codec never sees a stretch.
|
||||
Capped at the image's short side.
|
||||
"""
|
||||
l, t, r, b = box
|
||||
side = min(max(r - l, b - t), img.width, img.height)
|
||||
cx, cy = (l + r) / 2, (t + b) / 2
|
||||
nl = int(round(min(max(cx - side / 2, 0), img.width - side)))
|
||||
nt = int(round(min(max(cy - side / 2, 0), img.height - side)))
|
||||
return (nl, nt, nl + side, nt + side)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# HD: two encoded parts per photo, aimed at a two-packet budget
|
||||
# --------------------------------------------------------------------------
|
||||
#
|
||||
# The model is fixed at 512x512 / ft32, so "HD" spends the second packet on a
|
||||
# second 512x512 encode. The parts travel as one stream; the receiver decodes
|
||||
# both and blends them. Overheads below are what the stream would need on top
|
||||
# of the bitstreams (a hypothetical layout, not implemented in the app).
|
||||
|
||||
HD_TILES = "hd_tiles"
|
||||
HD_DETAIL = "hd_detail"
|
||||
HD_VARIANTS = (HD_TILES, HD_DETAIL)
|
||||
HD_OVERHEAD = {
|
||||
HD_TILES: 1, # length of part 0, so the receiver can split the stream
|
||||
HD_DETAIL: 1 + 4, # that, plus the detail box quantised to 4 bytes
|
||||
}
|
||||
TILE_OVERLAP = 0.08 # of the long side, cross-faded at the seam
|
||||
|
||||
|
||||
def variant_hd_tiles(img: Image.Image) -> list[Prepared]:
|
||||
"""Two overlapping halves along the long side, each encoded at 512x512.
|
||||
|
||||
Doubles the pixel count. A 3:2 photo splits into two near-square halves,
|
||||
so each half is also stretched far less than the whole frame would be.
|
||||
"""
|
||||
w, h = img.size
|
||||
long_side = max(w, h)
|
||||
ov = round(long_side * TILE_OVERLAP)
|
||||
half = (long_side + ov) // 2
|
||||
if w >= h:
|
||||
boxes = [(0, 0, half, h), (w - half, 0, w, h)]
|
||||
else:
|
||||
boxes = [(0, 0, w, half), (0, h - half, w, h)]
|
||||
return [Prepared(_resize(img.crop(b), (SIZE, SIZE)), b, (0, 0, SIZE, SIZE), feather=ov)
|
||||
for b in boxes]
|
||||
|
||||
|
||||
def variant_hd_detail(img: Image.Image, box: Box) -> list[Prepared]:
|
||||
"""The whole photo, plus the selected subject blended in at higher detail."""
|
||||
detail = variant_select(img, box)
|
||||
detail.feather = max(4, round(0.08 * min(box[2] - box[0], box[3] - box[1])))
|
||||
return [variant_stretch(img), detail]
|
||||
|
||||
|
||||
def composite(src: Image.Image, parts: list[Prepared], recons: list[Image.Image],
|
||||
long_side: int) -> tuple[Image.Image, Image.Image, Box]:
|
||||
"""Reference and receiver view for a variant made of one or more parts.
|
||||
|
||||
The view covers the union of the parts' regions at `long_side` pixels on
|
||||
its long edge. Parts are laid down in order; each later part fades in over
|
||||
`feather` pixels along its edges that are inside the union, so seams
|
||||
cross-fade instead of cutting hard.
|
||||
"""
|
||||
region = (min(p.region[0] for p in parts), min(p.region[1] for p in parts),
|
||||
max(p.region[2] for p in parts), max(p.region[3] for p in parts))
|
||||
rw, rh = region[2] - region[0], region[3] - region[1]
|
||||
s = long_side / max(rw, rh)
|
||||
size = (max(1, round(rw * s)), max(1, round(rh * s)))
|
||||
ref = src.crop(region).resize(size, Image.LANCZOS)
|
||||
canvas = np.zeros((size[1], size[0], 3), dtype=np.float64)
|
||||
for i, (p, recon) in enumerate(zip(parts, recons)):
|
||||
l, t, r, b = p.region
|
||||
x0, y0 = round((l - region[0]) * s), round((t - region[1]) * s)
|
||||
x1, y1 = round((r - region[0]) * s), round((b - region[1]) * s)
|
||||
tile = np.asarray(recon.crop(p.content_box).resize((x1 - x0, y1 - y0), Image.LANCZOS),
|
||||
dtype=np.float64)
|
||||
alpha = np.ones((y1 - y0, x1 - x0), dtype=np.float64)
|
||||
f = min(round(p.feather * s), alpha.shape[0], alpha.shape[1])
|
||||
if i > 0 and f > 0:
|
||||
ramp = np.arange(1, f + 1, dtype=np.float64) / f
|
||||
if l > region[0]:
|
||||
alpha[:, :f] *= ramp[None, :]
|
||||
if r < region[2]:
|
||||
alpha[:, -f:] *= ramp[::-1][None, :]
|
||||
if t > region[1]:
|
||||
alpha[:f, :] *= ramp[:, None]
|
||||
if b < region[3]:
|
||||
alpha[-f:, :] *= ramp[::-1][:, None]
|
||||
a = alpha[..., None]
|
||||
canvas[y0:y1, x0:x1] = canvas[y0:y1, x0:x1] * (1 - a) + tile * a
|
||||
shown = Image.fromarray(np.clip(np.rint(canvas), 0, 255).astype(np.uint8), "RGB")
|
||||
return ref, shown, region
|
||||
|
||||
|
||||
# Mirrors imageDataChunkCount() in image_chunk_transport.dart.
|
||||
CHUNK_FIRST_CAPACITY = 157
|
||||
CHUNK_CAPACITY = 158
|
||||
|
||||
|
||||
def parse_rate_variant(v: str) -> dict | None:
|
||||
"""`q0.7` -> fixed step 0.7; `fit315` -> per-image step to fit 315 bytes."""
|
||||
try:
|
||||
if v.startswith("q"):
|
||||
return {"steps": [float(v[1:])], "budget": None}
|
||||
if v.startswith("fit"):
|
||||
return {"steps": STEP_LADDER, "budget": int(v[3:])}
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def data_chunks(payload_bytes: int) -> int:
|
||||
n = max(payload_bytes, 0)
|
||||
if n <= CHUNK_FIRST_CAPACITY:
|
||||
return 1
|
||||
return 1 + math.ceil((n - CHUNK_FIRST_CAPACITY) / CHUNK_CAPACITY)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Codec
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Codec:
|
||||
def __init__(self, models: Path, decoder: Path, gpu: bool) -> None:
|
||||
cpu = ["CPUExecutionProvider"]
|
||||
cuda = ["CUDAExecutionProvider", "CPUExecutionProvider"] if gpu else cpu
|
||||
# The entropy graph stays on CPU: its rounded outputs decide the bytes,
|
||||
# and CPU fp32 is the closest stand-in for the phone. It costs ~0.1 s.
|
||||
self.side = ort.InferenceSession(str(models / SIDE_GRAPH), providers=cpu)
|
||||
self.context = ort.InferenceSession(str(models / CONTEXT_GRAPH), providers=cpu)
|
||||
self.decoder = ort.InferenceSession(str(decoder), providers=cuda)
|
||||
self.decoder_provider = self.decoder.get_providers()[0]
|
||||
|
||||
def encode_side(self, square: Image.Image) -> dict[str, np.ndarray]:
|
||||
rgb = np.asarray(square, dtype=np.uint8)
|
||||
# Same float32 steps as aeicRgbToChw in image_codec_entropy.dart.
|
||||
chw = rgb.transpose(2, 0, 1).astype(np.float32) / np.float32(255.0)
|
||||
chw = (chw - np.float32(0.5)) * np.float32(2.0)
|
||||
names = (["z_q", "y_hat", "y", "base0"]
|
||||
+ [f"yq{i}" for i in range(4)] + [f"sc{i}" for i in range(4)])
|
||||
out = self.side.run(names, {"image": chw[None]})
|
||||
return dict(zip(names, out))
|
||||
|
||||
def encode_with_step(self, side: dict[str, np.ndarray], step: float) -> dict[str, np.ndarray]:
|
||||
"""Re-quantise y with step `step` instead of 1 (rate control).
|
||||
|
||||
Replays the four-stage context loop the receiver runs
|
||||
(AeicEntropyCodec.decodeToLatent) with the decode-side graph, coding
|
||||
round((y - mean) / step) against scale / step. The receiver's
|
||||
reconstruction becomes symbol * step + mean. step < 1 spends more
|
||||
bytes, step > 1 fewer. At step == 1 this reproduces the side graph's
|
||||
symbols, which the harness's q1 check relies on.
|
||||
"""
|
||||
st = np.float32(step)
|
||||
y, z_q = side["y"], side["z_q"]
|
||||
base = side["base0"].copy()
|
||||
out: dict[str, np.ndarray] = {"z_q": z_q}
|
||||
for stage in range(4):
|
||||
means, scales = self.context.run(
|
||||
["means", "scales"],
|
||||
{"z_q": z_q, "base": base, "stage": np.array([stage], dtype=np.int32)})
|
||||
m = STAGE_MASKS[stage]
|
||||
sym = np.where(m, np.rint((y - means) / st), np.float32(0)).astype(np.float32)
|
||||
base = np.where(m, sym * st + means, base).astype(np.float32)
|
||||
out[f"yq{stage}"] = sym
|
||||
out[f"sc{stage}"] = np.where(m, scales / st, np.float32(0)).astype(np.float32)
|
||||
out["y_hat"] = base
|
||||
return out
|
||||
|
||||
def decode(self, y_hat: np.ndarray) -> Image.Image:
|
||||
(out,) = self.decoder.run(["image"], {"y_hat": y_hat.astype(np.float32)})
|
||||
# Same mapping as _chwFloatsToRgb in image_codec_backend.dart: clamp,
|
||||
# then round half away from zero (Dart's double.round()).
|
||||
scaled = (out[0].astype(np.float64) + 1.0) * 127.5
|
||||
rgb = np.floor(np.clip(scaled, 0, 255) + 0.5).astype(np.uint8)
|
||||
return Image.fromarray(rgb.transpose(1, 2, 0), "RGB")
|
||||
|
||||
|
||||
def write_tensor_file(path: Path, t: dict[str, np.ndarray]) -> None:
|
||||
parts = [t["z_q"]] + [t[f"yq{i}"] for i in range(4)] + [t[f"sc{i}"] for i in range(4)]
|
||||
with open(path, "wb") as f:
|
||||
for p in parts:
|
||||
f.write(np.ascontiguousarray(p, dtype="<f4").tobytes())
|
||||
|
||||
|
||||
def count_bytes(cdf: Path, tensor_files: list[Path]) -> dict[str, dict]:
|
||||
if not tensor_files:
|
||||
return {}
|
||||
dart = shutil.which("dart")
|
||||
if dart is None:
|
||||
sys.exit("dart is not on PATH; it is needed to run aeic_bytes.dart")
|
||||
cmd = [dart, "run", str(DART_TOOL), str(cdf), *map(str, tensor_files)]
|
||||
proc = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
sys.exit(f"aeic_bytes.dart failed:\n{proc.stderr}")
|
||||
results = {}
|
||||
for line in proc.stdout.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("{"):
|
||||
row = json.loads(line)
|
||||
results[row["file"]] = row
|
||||
return results
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Metrics
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Metrics:
|
||||
"""Full-reference scores. Higher is better except LPIPS and DISTS."""
|
||||
|
||||
def __init__(self, device: str) -> None:
|
||||
import open_clip
|
||||
import piq
|
||||
|
||||
self.piq = piq
|
||||
self.device = device
|
||||
self.lpips = piq.LPIPS(reduction="none").to(device).eval()
|
||||
self.dists = piq.DISTS(reduction="none").to(device).eval()
|
||||
model, _, preprocess = open_clip.create_model_and_transforms(
|
||||
"ViT-B-32", pretrained="laion2b_s34b_b79k"
|
||||
)
|
||||
self.clip = model.to(device).eval()
|
||||
self.clip_pre = preprocess
|
||||
|
||||
def _t(self, img: Image.Image) -> torch.Tensor:
|
||||
a = np.asarray(img, dtype=np.float32) / 255.0
|
||||
return torch.from_numpy(a).permute(2, 0, 1)[None].to(self.device)
|
||||
|
||||
@torch.no_grad()
|
||||
def clip_embed(self, img: Image.Image) -> torch.Tensor:
|
||||
# CLIP's own preprocessing centre-crops to a square, which would hide
|
||||
# the frame edges -- exactly what crop variants throw away. Letterbox
|
||||
# first so the whole image is scored.
|
||||
side = max(img.size)
|
||||
boxed = Image.new("RGB", (side, side), (0, 0, 0))
|
||||
boxed.paste(img, ((side - img.width) // 2, (side - img.height) // 2))
|
||||
x = self.clip_pre(boxed)[None].to(self.device)
|
||||
e = self.clip.encode_image(x)
|
||||
return e / e.norm(dim=-1, keepdim=True)
|
||||
|
||||
@torch.no_grad()
|
||||
def score(self, ref: Image.Image, out: Image.Image, full_src_emb: torch.Tensor) -> dict:
|
||||
r, o = self._t(ref), self._t(out)
|
||||
out_emb = self.clip_embed(out)
|
||||
return {
|
||||
"psnr": float(self.piq.psnr(o, r, data_range=1.0)),
|
||||
"ms_ssim": float(self.piq.multi_scale_ssim(o, r, data_range=1.0)),
|
||||
"lpips": float(self.lpips(o, r)),
|
||||
"dists": float(self.dists(o, r)),
|
||||
# Semantic agreement with exactly what was sent ...
|
||||
"clip": float((self.clip_embed(ref) * out_emb).sum()),
|
||||
# ... and with the whole original photo, so cropping is charged
|
||||
# for what it threw away.
|
||||
"clip_full": float((full_src_emb * out_emb).sum()),
|
||||
}
|
||||
|
||||
|
||||
def roi_pair(src: Image.Image, region: Box, shown: Image.Image, box: Box) -> tuple[Image.Image, Image.Image] | None:
|
||||
"""The ROI as the receiver would see it after zooming in, plus its reference.
|
||||
|
||||
`shown` covers `region` of the source; the ROI is mapped into it and
|
||||
both crops are scaled to long side 512. For a full-frame variant that is an
|
||||
upscale of a few dozen pixels, which is exactly what zooming in shows.
|
||||
Returns None when the variant did not send the whole ROI.
|
||||
"""
|
||||
rl, rt, rr, rb = region
|
||||
l, t, r, b = box
|
||||
if l < rl or t < rt or r > rr or b > rb:
|
||||
return None
|
||||
sx = shown.width / (rr - rl)
|
||||
sy = shown.height / (rb - rt)
|
||||
out = shown.crop((round((l - rl) * sx), round((t - rt) * sy),
|
||||
round((r - rl) * sx), round((b - rt) * sy)))
|
||||
ref = src.crop(box)
|
||||
scale = SIZE / max(ref.size)
|
||||
size = (max(1, round(ref.width * scale)), max(1, round(ref.height * scale)))
|
||||
return ref.resize(size, Image.LANCZOS), out.resize(size, Image.BICUBIC)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Corpus
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
KODAK_URL = "https://r0k.us/graphics/kodak/kodak/kodim{:02d}.png"
|
||||
|
||||
|
||||
def download_kodak(dest: Path) -> None:
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
for i in range(1, 25):
|
||||
target = dest / f"kodim{i:02d}.png"
|
||||
if target.exists():
|
||||
continue
|
||||
print(f"downloading {target.name}", file=sys.stderr)
|
||||
tmp = target.with_suffix(".part")
|
||||
with urllib.request.urlopen(KODAK_URL.format(i), timeout=60) as r, open(tmp, "wb") as f:
|
||||
shutil.copyfileobj(r, f)
|
||||
tmp.rename(target)
|
||||
|
||||
|
||||
def load_source(path: Path) -> Image.Image:
|
||||
img = Image.open(path)
|
||||
# Picked photos are shown upright in the app, so honour EXIF orientation.
|
||||
img = ImageOps.exif_transpose(img)
|
||||
return img.convert("RGB")
|
||||
|
||||
|
||||
def find_images(folder: Path, limit: int | None) -> list[Path]:
|
||||
files = sorted(p for p in folder.rglob("*") if p.suffix.lower() in IMAGE_SUFFIXES)
|
||||
return files[:limit] if limit else files
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Report
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
METRIC_COLS = ["bytes", "data_chunks", "psnr", "ms_ssim", "lpips", "dists", "clip", "clip_full"]
|
||||
ROI_COLS = ["roi_psnr", "roi_ms_ssim", "roi_lpips", "roi_dists", "roi_clip"]
|
||||
LOWER_IS_BETTER = {"bytes", "data_chunks", "lpips", "dists", "roi_lpips", "roi_dists"}
|
||||
|
||||
|
||||
def summarise(rows: list[dict], variants: list[str]) -> dict:
|
||||
summary = {}
|
||||
for v in variants:
|
||||
vr = [r for r in rows if r["variant"] == v]
|
||||
if not vr:
|
||||
continue
|
||||
s: dict = {"images": len(vr)}
|
||||
for m in METRIC_COLS + ROI_COLS:
|
||||
vals = np.array([r[m] for r in vr if r.get(m) not in (None, "")], dtype=np.float64)
|
||||
if vals.size == 0:
|
||||
continue
|
||||
s[m] = {"mean": float(vals.mean()), "median": float(np.median(vals)),
|
||||
"min": float(vals.min()), "max": float(vals.max())}
|
||||
chunks = [r["data_chunks"] for r in vr]
|
||||
s["chunk_histogram"] = {str(k): chunks.count(k) for k in sorted(set(chunks))}
|
||||
s["packets_mean_with_parity"] = float(np.mean(chunks)) + PARITY_CHUNKS
|
||||
# The share of images whose CLIP agreement with the source is low.
|
||||
# This is the "fox came back as a sheep" bucket; worth eyeballing.
|
||||
s["clip_below_0.80"] = sum(1 for r in vr if r["clip"] < 0.80)
|
||||
summary[v] = s
|
||||
return summary
|
||||
|
||||
|
||||
def print_summary(summary: dict) -> None:
|
||||
head = (f"{'variant':<10}" + "".join(f"{m:>12}" for m in METRIC_COLS)
|
||||
+ f"{'1-pkt':>8}{'<=2-pkt':>9}{'clip<.8':>9}")
|
||||
print(head)
|
||||
print("-" * len(head))
|
||||
for v, s in summary.items():
|
||||
line = f"{v:<10}" + "".join(f"{s[m]['mean']:>12.4g}" for m in METRIC_COLS)
|
||||
one = s["chunk_histogram"].get("1", 0)
|
||||
two = one + s["chunk_histogram"].get("2", 0)
|
||||
line += f"{one / s['images']:>7.0%} {two / s['images']:>8.0%} {s['clip_below_0.80']:>8}"
|
||||
print(line)
|
||||
print("means; lower is better for bytes, data_chunks, lpips, dists")
|
||||
if any("roi_lpips" in s for s in summary.values()):
|
||||
print()
|
||||
print(f"{'variant':<10}" + "".join(f"{m:>13}" for m in ROI_COLS))
|
||||
for v, s in summary.items():
|
||||
if "roi_lpips" in s:
|
||||
print(f"{v:<10}" + "".join(f"{s[m]['mean']:>13.4g}" for m in ROI_COLS))
|
||||
print("scored on the selected region only, as the receiver sees it after zooming in")
|
||||
|
||||
|
||||
def write_html(out: Path, rows: list[dict], summary: dict, meta: dict) -> None:
|
||||
def fmt(v: float, m: str) -> str:
|
||||
return f"{v:.1f}" if m in ("bytes", "data_chunks") else f"{v:.3f}"
|
||||
|
||||
variants = list(summary)
|
||||
stems = list(dict.fromkeys(r["image"] for r in rows))
|
||||
by = {(r["image"], r["variant"]): r for r in rows}
|
||||
|
||||
sum_rows = "".join(
|
||||
f"<tr><th>{html.escape(v)}</th>"
|
||||
+ "".join(f"<td>{fmt(summary[v][m]['mean'], m)}</td>" for m in METRIC_COLS)
|
||||
+ "</tr>"
|
||||
for v in variants
|
||||
)
|
||||
cards = []
|
||||
for stem in stems:
|
||||
cells = []
|
||||
first = by[(stem, variants[0])]
|
||||
cells.append(
|
||||
f'<figure><img loading="lazy" src="{html.escape(first["ref_png"])}">'
|
||||
f"<figcaption>source</figcaption></figure>"
|
||||
)
|
||||
for v in variants:
|
||||
r = by.get((stem, v))
|
||||
if not r:
|
||||
continue
|
||||
warn = " warn" if r["clip"] < 0.80 else ""
|
||||
cells.append(
|
||||
f'<figure class="{warn.strip()}"><img loading="lazy" src="{html.escape(r["out_png"])}">'
|
||||
f"<figcaption><b>{html.escape(v)}</b> {r['bytes']} B / {r['data_chunks']} pkt<br>"
|
||||
f"LPIPS {r['lpips']:.3f} · CLIP {r['clip']:.3f}</figcaption></figure>"
|
||||
)
|
||||
cards.append(f"<section><h3>{html.escape(stem)}</h3><div class=row>{''.join(cells)}</div></section>")
|
||||
|
||||
doc = f"""<!doctype html><html lang=en><meta charset=utf-8>
|
||||
<meta name=viewport content="width=device-width,initial-scale=1">
|
||||
<title>AEIC eval</title>
|
||||
<style>
|
||||
:root{{color-scheme:light dark;--bg:#fff;--fg:#111;--muted:#666;--line:#ddd;--warn:#c62828}}
|
||||
@media (prefers-color-scheme:dark){{:root{{--bg:#121212;--fg:#eee;--muted:#aaa;--line:#333;--warn:#ef5350}}}}
|
||||
body{{background:var(--bg);color:var(--fg);font:15px/1.4 system-ui,sans-serif;margin:16px}}
|
||||
table{{border-collapse:collapse;margin:8px 0 24px}}td,th{{border:1px solid var(--line);padding:4px 8px;text-align:right}}
|
||||
.row{{display:flex;gap:8px;flex-wrap:wrap}}figure{{margin:0;width:256px}}
|
||||
figure img{{width:256px;height:auto;display:block;border:2px solid transparent}}
|
||||
figure.warn img{{border-color:var(--warn)}}figcaption{{font-size:13px;color:var(--muted)}}
|
||||
code{{font-size:13px}}
|
||||
</style>
|
||||
<h1>AEIC eval</h1>
|
||||
<p><code>{html.escape(json.dumps(meta))}</code></p>
|
||||
<h2>Means</h2>
|
||||
<table><tr><th></th>{''.join(f'<th>{m}</th>' for m in METRIC_COLS)}</tr>{sum_rows}</table>
|
||||
<p>Images are shown at the aspect ratio the receiver restores. A red border means CLIP similarity to the sent region is below 0.80, so check it by eye.</p>
|
||||
{''.join(cards)}
|
||||
</html>"""
|
||||
(out / "report.html").write_text(doc, encoding="utf-8")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Main
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
for block in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--images", type=Path, help="folder of photos (searched recursively)")
|
||||
ap.add_argument("--download-kodak", type=Path, metavar="DIR",
|
||||
help="download the 24 Kodak images to DIR and use them")
|
||||
ap.add_argument("--out", type=Path, required=True, help="output folder")
|
||||
ap.add_argument("--models", type=Path, default=DEFAULT_MODELS,
|
||||
help=f"model bundle folder (default {DEFAULT_MODELS})")
|
||||
ap.add_argument("--decoder", type=Path, help="alternative decoder .onnx (e.g. an fp16 export)")
|
||||
ap.add_argument("--variants", default="stretch",
|
||||
help=f"comma list from: {', '.join(VARIANTS)} (default stretch)")
|
||||
ap.add_argument("--limit", type=int, help="only the first N images")
|
||||
ap.add_argument("--cpu", action="store_true", help="run the decoder on CPU too")
|
||||
ap.add_argument("--rois", type=Path,
|
||||
help="JSON {image stem: [l, t, r, b] fractions}; enables the select and "
|
||||
"hd_detail variants and ROI scores, and limits the run to listed images")
|
||||
ap.add_argument("--display", type=int, default=SIZE,
|
||||
help="long side in pixels of the receiver view that is scored (default 512; "
|
||||
"use 1024 when comparing HD variants)")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.download_kodak:
|
||||
download_kodak(args.download_kodak)
|
||||
args.images = args.images or args.download_kodak
|
||||
if not args.images:
|
||||
ap.error("give --images or --download-kodak")
|
||||
variants = [v.strip() for v in args.variants.split(",") if v.strip()]
|
||||
unknown = [v for v in variants
|
||||
if v not in VARIANTS and v not in SELECT_VARIANTS and v not in HD_VARIANTS
|
||||
and parse_rate_variant(v) is None]
|
||||
if unknown:
|
||||
ap.error(f"unknown variants: {unknown}")
|
||||
rois: dict[str, list[float]] = {}
|
||||
if args.rois:
|
||||
rois = {k: v for k, v in json.loads(args.rois.read_text()).items() if not k.startswith("_")}
|
||||
if any(v in SELECT_VARIANTS + (HD_DETAIL,) for v in variants) and not rois:
|
||||
ap.error("the select and hd_detail variants need --rois")
|
||||
|
||||
images = find_images(args.images, None)
|
||||
if rois:
|
||||
images = [p for p in images if p.relative_to(args.images).with_suffix("").as_posix() in rois]
|
||||
images = images[: args.limit] if args.limit else images
|
||||
if not images:
|
||||
sys.exit(f"no images under {args.images}")
|
||||
|
||||
out: Path = args.out
|
||||
work = out / "tensors"
|
||||
dirs = [work] + [out / k / v for k in ("recon", "ref", "roi") for v in variants]
|
||||
for d in dirs:
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
decoder_path = args.decoder or (args.models / DECODER_GRAPH)
|
||||
gpu = not args.cpu and torch.cuda.is_available()
|
||||
codec = Codec(args.models, decoder_path, gpu)
|
||||
metrics = Metrics("cuda" if gpu else "cpu")
|
||||
print(f"{len(images)} images x {len(variants)} variants; decoder on {codec.decoder_provider}",
|
||||
file=sys.stderr)
|
||||
|
||||
# Pass 1: preprocess + encode side, so every byte count is one Dart run.
|
||||
jobs = []
|
||||
for path in images:
|
||||
src = load_source(path)
|
||||
key = path.relative_to(args.images).with_suffix("").as_posix()
|
||||
stem = key.replace("/", "__")
|
||||
box = roi_box(src, rois[key]) if key in rois else None
|
||||
for v in variants:
|
||||
if (v in SELECT_VARIANTS or v == HD_DETAIL) and box is None:
|
||||
continue
|
||||
if v == SELECT:
|
||||
parts = [variant_select(src, box)]
|
||||
elif v == SELECT_SQUARE:
|
||||
parts = [variant_select(src, square_box(src, box))]
|
||||
elif v == HD_TILES:
|
||||
parts = variant_hd_tiles(src)
|
||||
elif v == HD_DETAIL:
|
||||
parts = variant_hd_detail(src, box)
|
||||
elif parse_rate_variant(v) is not None:
|
||||
parts = [variant_stretch(src)]
|
||||
else:
|
||||
parts = [VARIANTS[v](src)]
|
||||
rate = parse_rate_variant(v)
|
||||
steps = rate["steps"] if rate else [None]
|
||||
t0 = time.perf_counter()
|
||||
encoded = [] # per part: [(step, y_hat, tensor file), ...]
|
||||
for i, p in enumerate(parts):
|
||||
side = codec.encode_side(p.square)
|
||||
cands = []
|
||||
for step in steps:
|
||||
tensors = side if step is None else codec.encode_with_step(side, step)
|
||||
tag = "" if step is None else f".s{step:.2f}"
|
||||
tf = work / f"{stem}.{v}.p{i}{tag}.f32"
|
||||
write_tensor_file(tf, tensors)
|
||||
cands.append((step, tensors["y_hat"], tf))
|
||||
encoded.append(cands)
|
||||
enc_s = time.perf_counter() - t0
|
||||
jobs.append((path, stem, v, src, parts, box, encoded, enc_s))
|
||||
byte_rows = count_bytes(args.models / CDF_TABLE,
|
||||
[tf for j in jobs for cands in j[6] for _, _, tf in cands])
|
||||
|
||||
# Pick one candidate per part. A fit<N> variant takes the finest step whose
|
||||
# stream fits N bytes, or the coarsest step if none does.
|
||||
for j in jobs:
|
||||
rate = parse_rate_variant(j[2])
|
||||
chosen = []
|
||||
for cands in j[6]:
|
||||
if rate and rate["budget"] is not None:
|
||||
fitting = [c for c in cands if byte_rows[str(c[2])]["bytes"] <= rate["budget"]]
|
||||
chosen.append(min(fitting, key=lambda c: c[0]) if fitting
|
||||
else max(cands, key=lambda c: c[0]))
|
||||
else:
|
||||
chosen.append(cands[0])
|
||||
j[6][:] = chosen
|
||||
|
||||
# Pass 2: decode + score.
|
||||
rows = []
|
||||
emb_cache: dict[str, torch.Tensor] = {}
|
||||
for n, (path, stem, v, src, parts, box, encoded, enc_s) in enumerate(jobs, 1):
|
||||
t0 = time.perf_counter()
|
||||
recons = [codec.decode(y_hat) for _, y_hat, _ in encoded]
|
||||
dec_s = time.perf_counter() - t0
|
||||
ref, shown, region = composite(src, parts, recons, args.display)
|
||||
part_bytes = [byte_rows[str(tf)]["bytes"] for _, _, tf in encoded]
|
||||
total_bytes = sum(part_bytes) + HD_OVERHEAD.get(v, 0)
|
||||
chunks = data_chunks(total_bytes)
|
||||
if len(parts) == 1:
|
||||
assert chunks == byte_rows[str(encoded[0][2])]["data_chunks"], "chunk mirror drifted"
|
||||
if stem not in emb_cache:
|
||||
emb_cache[stem] = metrics.clip_embed(src)
|
||||
scores = metrics.score(ref, shown, emb_cache[stem])
|
||||
roi = roi_pair(src, region, shown, box) if box else None
|
||||
if roi:
|
||||
roi_scores = metrics.score(roi[0], roi[1], emb_cache[stem])
|
||||
scores.update({f"roi_{k}": roi_scores[k] for k in ("psnr", "ms_ssim", "lpips", "dists", "clip")})
|
||||
roi[1].save(out / "roi" / v / f"{stem}.png")
|
||||
roi[0].save(out / "roi" / v / f"{stem}.ref.png")
|
||||
else:
|
||||
scores.update({c: "" for c in ROI_COLS})
|
||||
out_png = Path("recon") / v / f"{stem}.png"
|
||||
ref_png = Path("ref") / v / f"{stem}.png"
|
||||
shown.save(out / out_png)
|
||||
ref.save(out / ref_png)
|
||||
row = {
|
||||
"image": stem, "variant": v, "source": str(path),
|
||||
"width": src.width, "height": src.height,
|
||||
"bytes": total_bytes, "data_chunks": chunks,
|
||||
"part_bytes": "+".join(map(str, part_bytes)),
|
||||
"step": "+".join("1.00" if st is None else f"{st:.2f}" for st, _, _ in encoded),
|
||||
**scores,
|
||||
"encode_s": round(enc_s, 3), "decode_s": round(dec_s, 3),
|
||||
"out_png": out_png.as_posix(), "ref_png": ref_png.as_posix(),
|
||||
"bitstream": ";".join(os.path.relpath(byte_rows[str(tf)]["bitstream"], out)
|
||||
for _, _, tf in encoded),
|
||||
}
|
||||
rows.append(row)
|
||||
print(f"[{n}/{len(jobs)}] {stem} {v}: {row['bytes']} B, LPIPS {row['lpips']:.3f}, "
|
||||
f"CLIP {row['clip']:.3f}, decode {dec_s:.2f}s", file=sys.stderr)
|
||||
|
||||
summary = summarise(rows, variants)
|
||||
meta = {
|
||||
"date": dt.datetime.now().isoformat(timespec="seconds"),
|
||||
"images": str(args.images), "count": len(images), "variants": variants,
|
||||
"decoder": decoder_path.name, "decoder_sha256": sha256(decoder_path)[:16],
|
||||
"decoder_provider": codec.decoder_provider,
|
||||
"onnxruntime": ort.__version__,
|
||||
}
|
||||
with open(out / "results.csv", "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=list(rows[0]))
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
(out / "summary.json").write_text(json.dumps({"meta": meta, "variants": summary}, indent=1))
|
||||
write_html(out, rows, summary, meta)
|
||||
for tf in work.glob("*.f32"): # 2 MB each; the .aeic bitstreams are kept
|
||||
tf.unlink()
|
||||
print()
|
||||
print_summary(summary)
|
||||
print(f"\nreport: {out / 'report.html'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,10 @@
|
||||
# For the CUDA build of torch, install it first from the PyTorch index, e.g.
|
||||
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
|
||||
torch
|
||||
torchvision
|
||||
onnxruntime-gpu # or onnxruntime for CPU only
|
||||
numpy
|
||||
pillow
|
||||
piq
|
||||
open_clip_torch
|
||||
imagecodecs # compare_codecs.py: JPEG XL and AVIF
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "Subject boxes for the Kodak set, as fractions of width/height: [left, top, right, bottom]. Stand-ins for a user's 'select' crop.",
|
||||
"kodim02": [0.15, 0.15, 0.70, 0.75],
|
||||
"kodim03": [0.15, 0.05, 0.65, 0.65],
|
||||
"kodim04": [0.20, 0.15, 0.80, 0.65],
|
||||
"kodim05": [0.00, 0.00, 0.50, 1.00],
|
||||
"kodim06": [0.15, 0.15, 0.80, 0.85],
|
||||
"kodim09": [0.35, 0.10, 0.95, 0.75],
|
||||
"kodim11": [0.35, 0.30, 0.85, 0.85],
|
||||
"kodim12": [0.25, 0.20, 0.65, 0.85],
|
||||
"kodim14": [0.00, 0.15, 0.90, 0.85],
|
||||
"kodim15": [0.35, 0.00, 0.90, 0.75],
|
||||
"kodim16": [0.40, 0.35, 0.90, 0.60],
|
||||
"kodim17": [0.25, 0.00, 0.75, 0.45],
|
||||
"kodim18": [0.25, 0.15, 0.85, 1.00],
|
||||
"kodim19": [0.35, 0.00, 0.85, 0.75],
|
||||
"kodim21": [0.30, 0.05, 0.70, 0.65],
|
||||
"kodim22": [0.05, 0.10, 0.60, 0.70],
|
||||
"kodim23": [0.50, 0.00, 1.00, 0.75]
|
||||
}
|
||||
Reference in new issue
Block a user