Research

DON IT exceeds the published result in composed image retrieval

We trained a retrieval system that takes a photograph of a garment and a short written modification, and ranks real product photographs so the right garment comes first. On the public FashionIQ benchmark, our primary system scores 62.66% macro R@10 on VAL-Split, the DQU-CIR authors' own protocol, exceeding the published result of 61.97%. On the complete-gallery Original-Split, our public hybrid v0.1 scores 59.57%, above FactEmbed's published 57.71%. Every score on this page carries a dated, hash-identified execution receipt, and the per-query rank lists and evaluation code are published for independent verification.

Version
2.0, 21 November 2026
Evaluated
15-16 November 2026
Artifacts verified
21 November 2026
Dataset
FashionIQ validation, 6,016 queries
Protocols
Original-Split and VAL-Split
Artifacts
huggingface.co/DON-IT/donit-fashion-cir-v0.1

1. What this looks like

Three real validation queries, one per category. The reference photograph and the human-written modification are the input. The photograph on the right is the annotated target garment, the one our primary system ranked first of the complete category gallery.

Reference dress: a black sleeveless dress
Reference
Human-written modification

"is white with black polka dots"

"is not see through"

Target dress: a white dress with black polka dots
Ranked first

dress. Ranked first of 3,817 candidate dresses, Original-Split.

Reference shirt: a blue printed t-shirt
Reference
Human-written modification

"is green with a four leaf clover"

"is green and has no text"

Target shirt: a green t-shirt with a four-leaf clover print
Ranked first

shirt. Ranked first of 6,346 candidate shirts, Original-Split.

Reference top: a green tank top
Reference
Human-written modification

"is a short sleeve v neck"

"is short sleeved and pink"

Target top: a pink short-sleeved v-neck top
Ranked first

toptee. Ranked first of 5,373 candidate tops and tees, Original-Split.

Images and modification captions: FashionIQ validation set (Wu et al., CVPR 2021), Section 3. Queries selected from the published per-query rank lists as cases where the annotated target ranked first under both protocols.

2. Results at a glance

Macro R@10; definitions, per-category values and caveats below.

ProtocolDON ITPublished reference
VAL-Split62.66%, 95% CI [61.41, 63.87]DQU-CIR 61.97%
Original-Split55.98%, 95% CI [54.70, 57.26]DQU-CIR 54.65%; FactEmbed 57.71%
Original-Split, hybrid v0.159.57%FactEmbed 57.71%, cross-publication

3. The task and the dataset

Composed image retrieval is the part of fashion search that understands intent. A shopper has seen something close to what they want and describes the change ("same dress, but in a lighter color"). The system must read the photograph, understand the language, and rank thousands of real garments so the right one appears first. A query is scored correct at rank K if the annotated target image appears in the top K.

FashionIQ (Wu et al., "Fashion IQ: A New Dataset towards Retrieving Images by Natural Language Feedback", CVPR 2021) is the standard public benchmark for this task. We evaluate on the original public validation split, using the original author metadata: repository XiaoxiaoGuo/fashion-iq, commit f71b16c3b09ef03b02c0fed160fa072fb3b8aa16. Categories: dress, shirt, toptee (tops and tees), all women's fashion. Queries: 6,016 human validation queries in total (dress 2,017; shirt 2,038; toptee 1,961), each a reference image plus one or two human-written modification captions.

4. How we evaluated

Two protocols, the same 6,016 queries, different candidate galleries. They measure different things and we report them separately throughout.

4.1 Original-Split (complete gallery)

Each category ranks its entire original same-category validation gallery, with the query's reference image excluded:

Table 1. Original-Split galleries.
CategoryQueriesCandidate gallery
dress2,0173,817
shirt2,0386,346
toptee1,9615,373
total6,01615,536

This is the harder protocol and the one most comparable to production retrieval.

4.2 VAL-Split (DQU-CIR author protocol)

The candidate gallery per category is the union of all reference and target image IDs appearing in the original author's validation triplets for that category, exactly as defined in the DQU-CIR authors' code (iLearn-Lab/SIGIR24-DQU-CIR, commit 49f53992447a9a84b897d9b2e2f2c91dd2cc8788, src/datasets.py::FashionIQ.get_test_data and src/test.py).

Table 2. VAL-Split galleries.
CategoryQueriesCandidate gallery
dress2,0172,628
shirt2,0383,089
toptee1,9612,902
total6,0168,619

The galleries are smaller than in the Original-Split (8,619 vs 15,536 candidates), so scores on the two protocols are not interchangeable. We label every number with its protocol.

4.3 Metric definitions

  • Recall@K (R@K) per query: 1 if the annotated target image is within the top K retrieved candidates, else 0.
  • Category R@K: the mean over that category's queries.
  • Macro R@K (the headline metric): the unweighted mean of the three category R@K values. No query-count weighting is applied.
  • Reference exclusion: the query's reference image is removed from its candidate set before ranking.
  • Tie ordering: deterministic and stable, as implemented in the evaluation runner; the complete per-query rank lists are published (Section 9) so every metric can be recomputed from the ranks alone.
  • Score combination: for fused systems, each component's scores are z-score normalized within the candidate set of the split being scored. The target identity is never used in scoring, normalization, or ranking.

5. Results

5.1 VAL-Split (DQU-CIR author protocol)

Evaluated
15 November 2026
Execution receipt
JSON
Per-query ranks
npz
Table 3. VAL-Split results.
SystemMacro R@10Macro R@50
DON IT primary (Section 8)62.66%80.57%
DQU-CIR, published (Wen et al., SIGIR 2024, Table 1)61.97% (+/- 0.28 seed SD)81.56%

Per-category R@10: dress 56.07%, shirt 64.18%, toptee 67.72%. Per-category R@50: dress 76.75%, shirt 80.52%, toptee 84.45%. Exact macro value: 0.6265816540877746. The DON IT point score exceeds DQU-CIR's published mean by +0.69 percentage points on the authors' own protocol. Section 7 states what this comparison establishes.

60%62.5%65%DON IT primaryDQU-CIR published
Figure 1. VAL-Split macro R@10. DON IT primary system with cluster-bootstrap 95% CI; DQU-CIR published mean ± seed SD.

5.2 Original-Split (complete gallery)

Same receipt, same frozen system, same day.

Table 4. Original-Split results.
SystemMacro R@10Macro R@50
DON IT primary (Section 8)55.98%75.24%
DQU-CIR, published (Wen et al., SIGIR 2024, Table 1)54.65% (+/- 0.38 seed SD)75.60%
FactEmbed, published (project page)57.71%76.87%

Per-category R@10: dress 51.17%, shirt 57.02%, toptee 59.77%. Per-category R@50: dress 72.58%, shirt 74.29%, toptee 78.84%. Exact macro value: 0.5598240183465593. The run reproduces the previously recorded value exactly. The DON IT point score exceeds DQU-CIR's published mean by +1.33 percentage points.

52%57%62%DON IT primaryDQU-CIR publishedFactEmbed publishedDON IT hybrid v0.1
Figure 2. Original-Split macro R@10. DON IT primary with cluster-bootstrap 95% CI; DQU-CIR published mean ± seed SD; FactEmbed and the DON IT hybrid v0.1 shown as point scores (no published absolute interval; hybrid from the public model card).
0%20%40%60%80%K=1K=5K=10K=50
DON IT primary
Figure 3. Recall@K of the primary system on the complete Original-Split gallery, K = 1, 5, 10, 50 (categorical axis).

5.3 Hybrid system, Original-Split (public release v0.1)

Our strongest published system, released on Hugging Face as DON-IT/donit-fashion-cir-v0.1 on 16 November 2026, combines the DON IT-trained components with a DON IT-trained residual student and the externally released CSMCIR checkpoint, using fixed rank fusion (50/50 reciprocal-rank fusion, rank constant 60). It was evaluated on the Original-Split protocol.

Table 5. Hybrid system v0.1, Original-Split.
SystemMacro R@10Macro R@50
DON IT hybrid (trained student + Qwen + CSMCIR)59.57%79.41%
Previous DON IT hybrid59.40%-
Official CSMCIR checkpoint, measured by us57.46%-
DON IT trained student + Qwen, without CSMCIR56.35%-
DON IT primary (Section 8)55.98%75.24%
FactEmbed, published57.71%76.87%

Per-category R@10 (hybrid): dress 55.13%, shirt 59.13%, toptee 64.46%. The gap over FactEmbed's published number is a cross-publication comparison, not a matched re-execution of FactEmbed. The 59.57% hybrid requires the external CSMCIR checkpoint at inference; the fully DON IT-trained configuration without CSMCIR measures 56.35%.

6. What DON IT training adds

Same receipt family, full 15,536-image gallery, 6,016 queries. Each row adds one DON IT-trained component to the frozen foundation:

Table 6. Ablation results, Original-Split.
ConfigurationR@1R@5R@10R@50
Qwen600, frozen (no DON IT training)23.73%42.93%51.52%71.46%
Frozen Qwen600 + trained FashionSigLIP25.75%45.30%53.91%73.52%
Trained Qwen600 alone25.18%44.80%54.01%73.47%
Trained Qwen600 + trained FashionSigLIP (primary)26.81%46.92%55.98%75.24%

DON IT training on the query encoder adds +2.49 points of macro R@10 over the frozen foundation; adding the trained composition head adds +4.46 points over frozen Qwen600 alone.

0%20%40%60%80%K=1K=5K=10K=50
Frozen Qwen600Frozen Qwen600 + trained SigLIP headTrained Qwen600 aloneTrained Qwen600 + trained SigLIP head (primary)
Figure 4. Ablation progression on Original-Split, K = 1, 5, 10, 50 (categorical axis, shared scale). Dashed lines are configurations without DON IT-trained weights.

7. What the comparisons establish

Confidence intervals use a reference-cluster bootstrap (queries sharing a reference image are resampled together), 4,315 clusters, 2,000 replicates, seed 20261115.

VAL-Split: the 95% CI for the DON IT absolute macro R@10 is [61.41%, 63.87%]. Our point score (62.66%) exceeds DQU-CIR's published mean (61.97%) by +0.69 points; the interval's lower bound sits below their published mean. DQU-CIR's authors publish training code but not their trained checkpoints, so we did not re-run their model and no paired per-query comparison exists. The claim we make: we exceed the published result on the authors' own protocol.

Original-Split: the 95% CI for the DON IT absolute macro R@10 is [54.70%, 57.26%]. The lower bound is above DQU-CIR's published mean (54.65%). The comparison is against their published mean, not a re-executed DQU-CIR.

Hybrid vs FactEmbed: 59.57% vs 57.71% published is +1.86 points, cross-publication only.

Two further full-gallery scores (56.42% and 56.52% macro R@10, Original-Split, receipts of 15 November 2026) are locked-weight re-evaluations of primary-system variants on previously seen public validation data. We record them as research-only, not benchmark results.

8. The systems under test

8.1 Primary system: Qwen600 + trained FashionSigLIP, predeclared 70/30 fusion

The system evaluated on both protocols on 15 November 2026:

Table 7. Primary system components and artifact identities.
ComponentDescriptionArtifact identity (SHA-256)
Query encoderQwen3-VL-Embedding-8B with a DON IT-trained 600-step LoRA adapter (Qwen600)a0d0312c7406a4c1337165e9d80526d436dfe91c3f725bfb13ea0541adf89528
Second scorerMarqo FashionSigLIP with a DON IT-trained composition headhead checkpoint e4e6089df6aae44b28cb73b815f14cec80c2d247a457c4c7912fbf55b1b6a825; frozen embedding archive 6ca4c228574a577b2e2f5d510cc8b9fb5f76b00b96ed4cf1becc3f4cfa766955
Fusionper-query z-scored score fusion, 70% Qwen600 / 30% FashionSigLIPweights fixed and declared before evaluation

Both checkpoints were frozen before the evaluation. No gradient updates, no training, and no checkpoint selection were performed as part of the reported runs.

9. Verification and reproduction

9.1 Verify our published numbers without a GPU

The public release at huggingface.co/DON-IT/donit-fashion-cir-v0.1 contains the execution receipt, per-query rank lists and evaluation runner behind the 15 November runs (research/20261115/), machine-readable recorded results (research/20261116/recorded_results.json), an independent metric verifier (verify_rank_metrics.py), and a weights integrity auditor (audit_public_weights.py):

python research/20261116/verify_rank_metrics.py --check-recorded research/20261116/recorded_results.json
python research/20261116/audit_public_weights.py --output public_weight_audit.json

These verify the recorded metric arithmetic and artifact integrity. They do not re-run model inference.

9.2 Re-execute the evaluation

  1. Obtain FashionIQ under its own terms and verify the annotation files against the SHA-256 values in Section 10.
  2. Obtain Qwen3-VL-Embedding-8B, Marqo FashionSigLIP, and the released CSMCIR checkpoint under their respective terms.
  3. Apply the evaluation contract in research/20261116/REPRODUCTION.md (gallery alignment, per-split candidate-only z-score normalization, reference exclusion, tie policy).
  4. The exact per-query rank lists behind the 15 November runs are published as research/20261115/DQU_CIR_official_VAL_and_original_DONIT_frozen_ranks_20261115.npz (SHA-256 d3994bc6d44498b1c2e0dcbb74dbc77f2e25d86090ce381e406c095b2a49deae), so the headline metrics can be recomputed from ranks without re-running inference. The evaluation runner re-executes both protocols on one NVIDIA A10G in about 76 seconds.

10. Artifact index

Published under research/20261115/ in DON-IT/donit-fashion-cir-v0.1 (15 November 2026 run):

Table 8. Artifact index.
ArtifactIdentity
Execution receipt, both protocolsSHA-256 bb2fcb29a9ad5cc3ff77de7c8eec36ee5ae247ac6e9f0e71ae3980f20f6af39d
Per-query rank listsSHA-256 d3994bc6d44498b1c2e0dcbb74dbc77f2e25d86090ce381e406c095b2a49deae
Evaluation runnerSHA-256 6d2cc5dcfa9c98d65ffc25c444463e03c469165c20d0e79c309886a3e05048d2
Qwen600 LoRA checkpointSHA-256 a0d0312c7406a4c1337165e9d80526d436dfe91c3f725bfb13ea0541adf89528
Frozen Qwen original manifestSHA-256 e849a72649885499bd9e52a6da102e8633717a151dff7fcaf4ec61a95d8615cb
FashionSigLIP frozen embedding archiveSHA-256 6ca4c228574a577b2e2f5d510cc8b9fb5f76b00b96ed4cf1becc3f4cfa766955
Original gallery keys manifestSHA-256 d3b084310b2333e7ec5f18ab08a21cb0fa25ac00c046a353af1944d292f98942
cap.dress.val.jsonSHA-256 5e5117d45695df9c3ca91fac3e0bc49e6422ae9c8b3793f0ac9b3ab83adb7de9
cap.shirt.val.jsonSHA-256 7b7ca3797b85dddfd83e74cdb4454cb63e3e7c09acef2590155efbacea6d7feb
cap.toptee.val.jsonSHA-256 b4e09f428e6c255cc4b4ec46e77908305ab83075d82840c960be81af15cdc5eb
split.dress.val.jsonSHA-256 21ff91d53c23859da91bfd49f3acc139b7f3a3dc944fc4e4a80194468cf14bab
split.shirt.val.jsonSHA-256 b82effbf7352a6f828b38c45eb32e53d726bd4d6fd81d6290c81eeb2c68e3234
split.toptee.val.jsonSHA-256 ee42b2505275dd7a19b11ddb88c9264aab2d77f4e59b075fb1c7c3095d02f412
v0.1 hybrid releaseresearch/20261116/ in the same repository

External references:

  • FashionIQ dataset, metadata and validation images: Wu et al., CVPR 2021; XiaoxiaoGuo/fashion-iq @ f71b16c3.
  • DQU-CIR: Wen et al., "Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval", SIGIR 2024; code iLearn-Lab/SIGIR24-DQU-CIR @ 49f53992.
  • FactEmbed: KAIST CVLab, cvlab-kaist.github.io/FactEmbed (published results cited, not re-executed).
  • CSMCIR: Qian et al., "CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval", Findings of ACL 2026; aclanthology.org/2026.findings-acl.216. Externally released checkpoint, used as a fixed component of the v0.1 hybrid.

11. Limitations

  1. Public validation, previously seen. FashionIQ validation was observed during earlier development. These are reproducible benchmark numbers on public data, not results on an untouched holdout.
  2. Comparisons are against published numbers, not re-executions. DQU-CIR and FactEmbed checkpoints are not publicly available and were not re-run; our comparisons use their published numbers.
  3. Scope. FashionIQ measures reference-plus-edit retrieval in three women's fashion categories, and overlap between the external backbones' pretraining data and FashionIQ cannot be ruled out. These results are not evidence about the live donit.me service.

12. What comes next

We are extending the same receipt-backed evaluation to further public retrieval benchmarks, and the systems measured here feed the search technology we are building at donit.me. This page is updated when a new dated receipt changes a number above; earlier versions and their receipts are preserved.

Research by DON IT. Validation images on this page are from the FashionIQ dataset (Wu et al., CVPR 2021), used for research reporting with attribution.