Project 06

Multimodal Semantic Image Search

CLIP retrieval is usually shipped straight off the shelf. This project fine-tunes it first, and measures whether that fine-tuning actually did anything, on a genuinely held-out slice of a 44,412-image fashion catalog rather than the training distribution.

The Approach

The vision encoder gets vision-only LoRA fine-tuning on the catalog's own product titles, 2 epochs, rank 16. Evaluated on a true id-level holdout, 10% of images never seen during training, scored against attribute-combo queries ("red tshirts") deliberately phrased differently from any training caption, it lifts Recall@10 from 14.0% to 17.2% and MRR from 0.704 to 0.745: a +23.0% relative Recall@10 uplift. The absolute number is modest, no hard-negative mining, a genuinely hard 44K-item target, but the relative uplift on a real holdout is what demonstrates the fine-tuning generalizes rather than memorizes.

The Results

The ANN benchmark asks a different question entirely: does the approximate index find what exact search would have found, not whether the embeddings are good. Flat, IVFFlat, and HNSW were compared on the same 44,412-vector corpus and 50 real text queries. HNSW's efSearch was tuned by measuring the recall/latency curve first, landing on 64: 14.1x faster than exact search (0.15ms vs. 2.11ms p50) at 93.6% of its recall, the honest operating point rather than an untuned default that would have made HNSW look strictly worse than IVFFlat.

Serving loads one CLIP model (LoRA-merged if a checkpoint exists), and the deployed container needed a second, less obvious fix to fit a 512MB free-tier host: casting the fp32 checkpoint to fp16 at load time doesn't free the fp32 bytes already allocated. Merging the LoRA weights and saving a native fp16 checkpoint fixed the real remaining OOM, verified by profiling the actual running container rather than guessing.

The Honest Parts

A known limitation, reported rather than hidden: CLIP-family models struggle with fine-grained compositional distinctions, a "red dress" query here also surfaces red nightdresses and camisoles, genuinely close in color and silhouette but not the specific category asked for, matching the general WINOGROUND finding that these models often match individual concepts without reliably composing them.

What I Learned

A true holdout is what makes a fine-tuning result mean anythingEvaluating on training-distribution queries would make almost any fine-tuning look like it helped. Holding out 10% of images at the id level, and phrasing eval queries differently from training captions, is what turns +23% relative Recall@10 into a claim about generalization rather than memorization.
Tune the knob before the benchmark, not after seeing the numberHNSW's default efSearch=16 would have made it look strictly worse than IVFFlat in the comparison table, a real but misleading picture. Measuring the recall/latency curve and picking 64 before running the final 50-query benchmark keeps the comparison honest, documented specifically so that distinction is checkable.
Two things that both look like 'recall' can be answering different questionsANN recall (does the index find what exact search would have found) and embedding recall (does the model retrieve what's actually correct) are different error types. Keeping them as separate named functions instead of one shared recall_at_k avoids silently comparing an index's approximation loss against a semantic relevance number.
A relative uplift on a real holdout beats an inflated absolute number17.2% Recall@10 looks unimpressive in isolation. Reporting it next to the pretrained baseline (14.0%) and explaining the eval design, true holdout, deliberately reworded queries, is what makes the honest number more convincing than a cherry-picked absolute one would have been.

Tech Stack

PyTorchCLIPLoRA/PEFTFAISSFastAPIDockerGradioW&B

Links

Live GitHub