Research
DON IT exceeds the published result in composed image retrieval
We trained a retrieval system that takes a photograph of a garment and a short written modification, and ranks real product photographs so the right garment comes first. On the public FashionIQ benchmark, our primary system scores 62.66% macro R@10 on VAL-Split, the DQU-CIR authors' own protocol, exceeding the published result of 61.97%. On the complete-gallery Original-Split, our public hybrid v0.1 scores 59.57%, above FactEmbed's published 57.71%. Every score on this page carries a dated, hash-identified execution receipt, and the per-query rank lists and evaluation code are published for independent verification.
- Version
- 2.0, 21 November 2026
- Evaluated
- 15-16 November 2026
- Artifacts verified
- 21 November 2026
- Dataset
- FashionIQ validation, 6,016 queries
- Protocols
- Original-Split and VAL-Split
1. What this looks like
Three real validation queries, one per category. The reference photograph and the human-written modification are the input. The photograph on the right is the annotated target garment, the one our primary system ranked first of the complete category gallery.

"is white with black polka dots"
"is not see through"

dress. Ranked first of 3,817 candidate dresses, Original-Split.

"is green with a four leaf clover"
"is green and has no text"

shirt. Ranked first of 6,346 candidate shirts, Original-Split.

"is a short sleeve v neck"
"is short sleeved and pink"

toptee. Ranked first of 5,373 candidate tops and tees, Original-Split.
Images and modification captions: FashionIQ validation set (Wu et al., CVPR 2021), Section 3. Queries selected from the published per-query rank lists as cases where the annotated target ranked first under both protocols.
2. Results at a glance
Macro R@10; definitions, per-category values and caveats below.
| Protocol | DON IT | Published reference |
|---|---|---|
| VAL-Split | 62.66%, 95% CI [61.41, 63.87] | DQU-CIR 61.97% |
| Original-Split | 55.98%, 95% CI [54.70, 57.26] | DQU-CIR 54.65%; FactEmbed 57.71% |
| Original-Split, hybrid v0.1 | 59.57% | FactEmbed 57.71%, cross-publication |
3. The task and the dataset
Composed image retrieval is the part of fashion search that understands intent. A shopper has seen something close to what they want and describes the change ("same dress, but in a lighter color"). The system must read the photograph, understand the language, and rank thousands of real garments so the right one appears first. A query is scored correct at rank K if the annotated target image appears in the top K.
FashionIQ (Wu et al., "Fashion IQ: A New Dataset towards Retrieving Images by Natural Language Feedback", CVPR 2021) is the standard public benchmark for this task. We evaluate on the original public validation split, using the original author metadata: repository XiaoxiaoGuo/fashion-iq, commit f71b16c3b09ef03b02c0fed160fa072fb3b8aa16. Categories: dress, shirt, toptee (tops and tees), all women's fashion. Queries: 6,016 human validation queries in total (dress 2,017; shirt 2,038; toptee 1,961), each a reference image plus one or two human-written modification captions.
4. How we evaluated
Two protocols, the same 6,016 queries, different candidate galleries. They measure different things and we report them separately throughout.
4.1 Original-Split (complete gallery)
Each category ranks its entire original same-category validation gallery, with the query's reference image excluded:
| Category | Queries | Candidate gallery |
|---|---|---|
| dress | 2,017 | 3,817 |
| shirt | 2,038 | 6,346 |
| toptee | 1,961 | 5,373 |
| total | 6,016 | 15,536 |
This is the harder protocol and the one most comparable to production retrieval.
4.2 VAL-Split (DQU-CIR author protocol)
The candidate gallery per category is the union of all reference and target image IDs appearing in the original author's validation triplets for that category, exactly as defined in the DQU-CIR authors' code (iLearn-Lab/SIGIR24-DQU-CIR, commit 49f53992447a9a84b897d9b2e2f2c91dd2cc8788, src/datasets.py::FashionIQ.get_test_data and src/test.py).
| Category | Queries | Candidate gallery |
|---|---|---|
| dress | 2,017 | 2,628 |
| shirt | 2,038 | 3,089 |
| toptee | 1,961 | 2,902 |
| total | 6,016 | 8,619 |
The galleries are smaller than in the Original-Split (8,619 vs 15,536 candidates), so scores on the two protocols are not interchangeable. We label every number with its protocol.
4.3 Metric definitions
- Recall@K (R@K) per query: 1 if the annotated target image is within the top K retrieved candidates, else 0.
- Category R@K: the mean over that category's queries.
- Macro R@K (the headline metric): the unweighted mean of the three category R@K values. No query-count weighting is applied.
- Reference exclusion: the query's reference image is removed from its candidate set before ranking.
- Tie ordering: deterministic and stable, as implemented in the evaluation runner; the complete per-query rank lists are published (Section 9) so every metric can be recomputed from the ranks alone.
- Score combination: for fused systems, each component's scores are z-score normalized within the candidate set of the split being scored. The target identity is never used in scoring, normalization, or ranking.
5. Results
5.1 VAL-Split (DQU-CIR author protocol)
| System | Macro R@10 | Macro R@50 |
|---|---|---|
| DON IT primary (Section 8) | 62.66% | 80.57% |
| DQU-CIR, published (Wen et al., SIGIR 2024, Table 1) | 61.97% (+/- 0.28 seed SD) | 81.56% |
Per-category R@10: dress 56.07%, shirt 64.18%, toptee 67.72%. Per-category R@50: dress 76.75%, shirt 80.52%, toptee 84.45%. Exact macro value: 0.6265816540877746. The DON IT point score exceeds DQU-CIR's published mean by +0.69 percentage points on the authors' own protocol. Section 7 states what this comparison establishes.
5.2 Original-Split (complete gallery)
Same receipt, same frozen system, same day.
| System | Macro R@10 | Macro R@50 |
|---|---|---|
| DON IT primary (Section 8) | 55.98% | 75.24% |
| DQU-CIR, published (Wen et al., SIGIR 2024, Table 1) | 54.65% (+/- 0.38 seed SD) | 75.60% |
| FactEmbed, published (project page) | 57.71% | 76.87% |
Per-category R@10: dress 51.17%, shirt 57.02%, toptee 59.77%. Per-category R@50: dress 72.58%, shirt 74.29%, toptee 78.84%. Exact macro value: 0.5598240183465593. The run reproduces the previously recorded value exactly. The DON IT point score exceeds DQU-CIR's published mean by +1.33 percentage points.
5.3 Hybrid system, Original-Split (public release v0.1)
Our strongest published system, released on Hugging Face as DON-IT/donit-fashion-cir-v0.1 on 16 November 2026, combines the DON IT-trained components with a DON IT-trained residual student and the externally released CSMCIR checkpoint, using fixed rank fusion (50/50 reciprocal-rank fusion, rank constant 60). It was evaluated on the Original-Split protocol.
| System | Macro R@10 | Macro R@50 |
|---|---|---|
| DON IT hybrid (trained student + Qwen + CSMCIR) | 59.57% | 79.41% |
| Previous DON IT hybrid | 59.40% | - |
| Official CSMCIR checkpoint, measured by us | 57.46% | - |
| DON IT trained student + Qwen, without CSMCIR | 56.35% | - |
| DON IT primary (Section 8) | 55.98% | 75.24% |
| FactEmbed, published | 57.71% | 76.87% |
Per-category R@10 (hybrid): dress 55.13%, shirt 59.13%, toptee 64.46%. The gap over FactEmbed's published number is a cross-publication comparison, not a matched re-execution of FactEmbed. The 59.57% hybrid requires the external CSMCIR checkpoint at inference; the fully DON IT-trained configuration without CSMCIR measures 56.35%.
6. What DON IT training adds
Same receipt family, full 15,536-image gallery, 6,016 queries. Each row adds one DON IT-trained component to the frozen foundation:
| Configuration | R@1 | R@5 | R@10 | R@50 |
|---|---|---|---|---|
| Qwen600, frozen (no DON IT training) | 23.73% | 42.93% | 51.52% | 71.46% |
| Frozen Qwen600 + trained FashionSigLIP | 25.75% | 45.30% | 53.91% | 73.52% |
| Trained Qwen600 alone | 25.18% | 44.80% | 54.01% | 73.47% |
| Trained Qwen600 + trained FashionSigLIP (primary) | 26.81% | 46.92% | 55.98% | 75.24% |
DON IT training on the query encoder adds +2.49 points of macro R@10 over the frozen foundation; adding the trained composition head adds +4.46 points over frozen Qwen600 alone.
7. What the comparisons establish
Confidence intervals use a reference-cluster bootstrap (queries sharing a reference image are resampled together), 4,315 clusters, 2,000 replicates, seed 20261115.
VAL-Split: the 95% CI for the DON IT absolute macro R@10 is [61.41%, 63.87%]. Our point score (62.66%) exceeds DQU-CIR's published mean (61.97%) by +0.69 points; the interval's lower bound sits below their published mean. DQU-CIR's authors publish training code but not their trained checkpoints, so we did not re-run their model and no paired per-query comparison exists. The claim we make: we exceed the published result on the authors' own protocol.
Original-Split: the 95% CI for the DON IT absolute macro R@10 is [54.70%, 57.26%]. The lower bound is above DQU-CIR's published mean (54.65%). The comparison is against their published mean, not a re-executed DQU-CIR.
Hybrid vs FactEmbed: 59.57% vs 57.71% published is +1.86 points, cross-publication only.
Two further full-gallery scores (56.42% and 56.52% macro R@10, Original-Split, receipts of 15 November 2026) are locked-weight re-evaluations of primary-system variants on previously seen public validation data. We record them as research-only, not benchmark results.
8. The systems under test
8.1 Primary system: Qwen600 + trained FashionSigLIP, predeclared 70/30 fusion
The system evaluated on both protocols on 15 November 2026:
| Component | Description | Artifact identity (SHA-256) |
|---|---|---|
| Query encoder | Qwen3-VL-Embedding-8B with a DON IT-trained 600-step LoRA adapter (Qwen600) | a0d0312c7406a4c1337165e9d80526d436dfe91c3f725bfb13ea0541adf89528 |
| Second scorer | Marqo FashionSigLIP with a DON IT-trained composition head | head checkpoint e4e6089df6aae44b28cb73b815f14cec80c2d247a457c4c7912fbf55b1b6a825; frozen embedding archive 6ca4c228574a577b2e2f5d510cc8b9fb5f76b00b96ed4cf1becc3f4cfa766955 |
| Fusion | per-query z-scored score fusion, 70% Qwen600 / 30% FashionSigLIP | weights fixed and declared before evaluation |
Both checkpoints were frozen before the evaluation. No gradient updates, no training, and no checkpoint selection were performed as part of the reported runs.
9. Verification and reproduction
9.1 Verify our published numbers without a GPU
The public release at huggingface.co/DON-IT/donit-fashion-cir-v0.1 contains the execution receipt, per-query rank lists and evaluation runner behind the 15 November runs (research/20261115/), machine-readable recorded results (research/20261116/recorded_results.json), an independent metric verifier (verify_rank_metrics.py), and a weights integrity auditor (audit_public_weights.py):
python research/20261116/verify_rank_metrics.py --check-recorded research/20261116/recorded_results.json python research/20261116/audit_public_weights.py --output public_weight_audit.json
These verify the recorded metric arithmetic and artifact integrity. They do not re-run model inference.
9.2 Re-execute the evaluation
- Obtain FashionIQ under its own terms and verify the annotation files against the SHA-256 values in Section 10.
- Obtain Qwen3-VL-Embedding-8B, Marqo FashionSigLIP, and the released CSMCIR checkpoint under their respective terms.
- Apply the evaluation contract in research/20261116/REPRODUCTION.md (gallery alignment, per-split candidate-only z-score normalization, reference exclusion, tie policy).
- The exact per-query rank lists behind the 15 November runs are published as research/20261115/DQU_CIR_official_VAL_and_original_DONIT_frozen_ranks_20261115.npz (SHA-256 d3994bc6d44498b1c2e0dcbb74dbc77f2e25d86090ce381e406c095b2a49deae), so the headline metrics can be recomputed from ranks without re-running inference. The evaluation runner re-executes both protocols on one NVIDIA A10G in about 76 seconds.
10. Artifact index
Published under research/20261115/ in DON-IT/donit-fashion-cir-v0.1 (15 November 2026 run):
| Artifact | Identity |
|---|---|
| Execution receipt, both protocols | SHA-256 bb2fcb29a9ad5cc3ff77de7c8eec36ee5ae247ac6e9f0e71ae3980f20f6af39d |
| Per-query rank lists | SHA-256 d3994bc6d44498b1c2e0dcbb74dbc77f2e25d86090ce381e406c095b2a49deae |
| Evaluation runner | SHA-256 6d2cc5dcfa9c98d65ffc25c444463e03c469165c20d0e79c309886a3e05048d2 |
| Qwen600 LoRA checkpoint | SHA-256 a0d0312c7406a4c1337165e9d80526d436dfe91c3f725bfb13ea0541adf89528 |
| Frozen Qwen original manifest | SHA-256 e849a72649885499bd9e52a6da102e8633717a151dff7fcaf4ec61a95d8615cb |
| FashionSigLIP frozen embedding archive | SHA-256 6ca4c228574a577b2e2f5d510cc8b9fb5f76b00b96ed4cf1becc3f4cfa766955 |
| Original gallery keys manifest | SHA-256 d3b084310b2333e7ec5f18ab08a21cb0fa25ac00c046a353af1944d292f98942 |
| cap.dress.val.json | SHA-256 5e5117d45695df9c3ca91fac3e0bc49e6422ae9c8b3793f0ac9b3ab83adb7de9 |
| cap.shirt.val.json | SHA-256 7b7ca3797b85dddfd83e74cdb4454cb63e3e7c09acef2590155efbacea6d7feb |
| cap.toptee.val.json | SHA-256 b4e09f428e6c255cc4b4ec46e77908305ab83075d82840c960be81af15cdc5eb |
| split.dress.val.json | SHA-256 21ff91d53c23859da91bfd49f3acc139b7f3a3dc944fc4e4a80194468cf14bab |
| split.shirt.val.json | SHA-256 b82effbf7352a6f828b38c45eb32e53d726bd4d6fd81d6290c81eeb2c68e3234 |
| split.toptee.val.json | SHA-256 ee42b2505275dd7a19b11ddb88c9264aab2d77f4e59b075fb1c7c3095d02f412 |
| v0.1 hybrid release | research/20261116/ in the same repository |
External references:
- FashionIQ dataset, metadata and validation images: Wu et al., CVPR 2021; XiaoxiaoGuo/fashion-iq @ f71b16c3.
- DQU-CIR: Wen et al., "Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval", SIGIR 2024; code iLearn-Lab/SIGIR24-DQU-CIR @ 49f53992.
- FactEmbed: KAIST CVLab, cvlab-kaist.github.io/FactEmbed (published results cited, not re-executed).
- CSMCIR: Qian et al., "CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval", Findings of ACL 2026; aclanthology.org/2026.findings-acl.216. Externally released checkpoint, used as a fixed component of the v0.1 hybrid.
11. Limitations
- Public validation, previously seen. FashionIQ validation was observed during earlier development. These are reproducible benchmark numbers on public data, not results on an untouched holdout.
- Comparisons are against published numbers, not re-executions. DQU-CIR and FactEmbed checkpoints are not publicly available and were not re-run; our comparisons use their published numbers.
- Scope. FashionIQ measures reference-plus-edit retrieval in three women's fashion categories, and overlap between the external backbones' pretraining data and FashionIQ cannot be ruled out. These results are not evidence about the live donit.me service.
12. What comes next
We are extending the same receipt-backed evaluation to further public retrieval benchmarks, and the systems measured here feed the search technology we are building at donit.me. This page is updated when a new dated receipt changes a number above; earlier versions and their receipts are preserved.
Research by DON IT. Validation images on this page are from the FashionIQ dataset (Wu et al., CVPR 2021), used for research reporting with attribution.