Searching My Image Gallery by What the Pictures Show, With a 440M Model

I have a gallery called Lightbox that indexes every image my machines generate: renders, blog banners, book figures, slides I exported once and forgot about. It holds 1,342 frames, and only 22% of them kept the prompt that made them. The rest have a filename and a folder, and a filename like banner-final-1600x200.png does not say much. So mostly I found things by colour, folder and shape, which is a nice way of saying I scrolled.

Now there is a switch next to the search box: words or meaning. In meaning mode, a query ranks every frame by what the image shows. We picked 20 frames at random from the ones with no prompt and wrote a one-sentence description of each. Thirteen came back at rank 1 and eighteen in the top 5, out of 1,336 candidates. The fairest keyword search I could run against filenames and folders found three.

The model that does this is Google's EmbeddingGemma 2. I built it with Claude Code in one session, a week after the release. The model was the quick part. The time went to three other places.

One checkpoint, sized to the job

EmbeddingGemma 2 is a 740M-parameter embedder, Apache 2.0, that maps text, code, images, video and audio into one 768-dimension space.¹1Google DeepMind model card, released 2026-10-06. 24 layers, an 8,192-token context, and a default of 280 tokens per image. Built on Gemma 4. The detail that shaped this build is that the encoders are modular. You load the builds you need from the same checkpoint: 270M for text and code, 440M once you add vision, the full 740M with audio. All of them project into the same space.

That one property makes the architecture cheap. Images get embedded once, in a batch, with the 440M build on a GPU. Queries arrive one at a time and only need the text tower, so the 270M build answers them on a CPU. One thing we did check rather than assume: the text tower gives the same vectors whichever build runs it. Four queries embedded by the 270M build on the CPU in fp32 and by the 440M build on the GPU in bf16 agreed at 0.997 cosine or better. Whether words and images really share the space is what the eval further down tests.

None of this is a new idea. Immich has offered CLIP search over photo libraries for years, in Postgres with a vector extension.²2Immich docs, "Searching": contextual search runs on CLIP-family models through the VectorChord extension, alongside OCR, faces and metadata filters. What changed is that a small, open, commercially usable model now covers images and text with a text tower light enough for a CPU, and my gallery is not a photo app. Its images are mostly diagrams, banners and screenshots, which matters later.

Two paths, one space

The write path piggybacks on the indexer Lightbox already runs every 15 minutes. After the sweep, lightbox embed finds frames whose vector is missing or stale and runs their thumbnails through the 440M build. Not the originals. The thumbnails are 480-pixel WebP files the indexer already made, and using them means the embedder can only see frames the indexer admitted. The cost is that the smallest printed text may not survive the downscale; I have not measured how much that loses. That last part comes back below.

The first full run did 1,336 frames in 126 seconds on a DGX Spark. After that, a sweep with nothing new exits before it imports torch, so the steady-state cost is close to zero.

The vectors go into a table in the app's own SQLite file. Float32 blobs, one per frame, keyed by frame id, with an mtime:size signature so an edited file re-embeds. A delete trigger on the frame table takes the vector with it.

CREATE TABLE IF NOT EXISTS frame_vec (
  frame_id INTEGER PRIMARY KEY,
  sig      TEXT NOT NULL,      -- mtime:size when embedded
  model    TEXT NOT NULL,      -- google/embeddinggemma-2@768
  dim      INTEGER NOT NULL,
  vec      BLOB NOT NULL       -- float32, L2-normalised
);
CREATE TRIGGER IF NOT EXISTS frame_vec_ad AFTER DELETE ON frame BEGIN
  DELETE FROM frame_vec WHERE frame_id = old.id;
END;

No vector database. The whole matrix is 1,336 by 768 floats, about 4.1 MB. Ranking is one numpy matrix-vector product, and then the gallery's existing filters (colour, folder, shape, duplicates) apply to the ranked list. Qdrant published a good post the day the model shipped about squeezing EmbeddingGemma 2 vectors for 10 million documents, where full-size float32 vectors alone need 30.7 GB of RAM.³3Qdrant, "How Small Can Google's New EmbeddingGemma 2 Get?": 1-bit quantized full-size vectors kept 99.0% of baseline nDCG@10 at 30x less vector RAM; 256 dimensions with rescoring reached 77x less at 94.5%. That is a real problem. It is not mine. Reach for the database when the matrix stops fitting in your head.

The read path is smaller. The web server stays standard library plus numpy. It posts the query text to a small embedding service running the 270M build on the CPU, gets back 768 floats, scores every frame, and applies the filters after.

Where the time went

Getting a vector out of the model took a few minutes, and one of those was an error: the processor would not import until I installed torchvision, which the install instructions do not mention. The rest of the time went to three things the demo never shows.

A new table is a new leak path

Lightbox has a denylist. Some folders in my vault hold work and personal material, and the gallery has no login (it is reachable only on my private network), so those folders are never walked and never indexed. A selftest checks that no indexed row comes from an excluded folder, and that the exclusion is holding real images back rather than matching nothing.

The vector table is a derived store, and that denylist was written before it existed. I have been bitten by this exact shape before: a guard that lives inside one builder protects only what that builder builds, and every audit of the guard passes while a store next to it holds the thing you excluded. So the embedder reads thumbnails (which only exist for admitted frames), re-checks each row against the denylist anyway, and the selftest now checks the denylist against the vector table directly.

side quest A gate you have never seen fail is not a gate, so we planted a vector for a frame from an excluded folder in a copy of the database and ran the selftest against the copy. Both checks went red, the index one and the new vector one.

Vectors that fail without failing

The model card is direct about two silent failures. Activations overflow fp16, so in half precision you get NaN or degraded vectors and no exception. And if you shorten vectors to save space (the model supports Matryoshka truncation to 512, 256 or 128 dimensions), you have to re-normalise them, or every score still looks plausible and is wrong.⁴4The card's advice is to pass truncate_dim and normalize_embeddings=True to encode() rather than slicing by hand.

I take the fp16 warning seriously because I have seen the symptom. This summer a different model, the 8B embedder behind my notes search, ran in fp16 and returned NaN for about 40% of real queries. Every health check stayed green, because a check that asks "is the model loaded" cannot see a model that is loaded and producing garbage.

Fail loud, not over

If the query service is down, meaning search returns an error that says so. It does not fall back to word search. A word-search result under a "meaning" label reads as "the model looked and found nothing", which is a worse answer than no answer, and nobody would ever notice it was wrong.

How well it finds things, and what flatters the number

Two checks run in the selftest every time, and they point in opposite directions. For 20 frames that do have a prompt, the prompt goes in as a query: 15 of 20 frames come back in the top 10. Then the same prompts are scored against a random other frame: 0 of 20 land in the top 10. Chance alone would put about 0.15 of 20 there, so this is a sanity check rather than a strong control, but it is the check that fails on a model that ranks everything the same.

The test that matters is on frames with no prompt. Claude Code pulled 20 at random, laid them out on a contact sheet, looked at it, and wrote a one-sentence description per image before seeing any ranking. Then, as a baseline, one or two keywords per image, picked the same way, through word search.

Descriptions: 13 at rank 1, 18 in the top 5, median rank 1. Keywords through word search: 3 at rank 1. The keyword hits were the frames whose filename happened to say what they were (html, social security, pain points).

The same bare keywords through meaning search did worse than the sentences: 9 at rank 1 instead of 13. Meaning search wants a description, not a keyword. "family tree" put its target at 107; "family tree app profile page with a gravestone photo" put it at 43.

Now the caveats, which belong right here next to the number. This is n=20 and one run, so 13 of 20 at rank 1 is somewhere between about 43% and 82% with 95% confidence. Every query was written by a model looking at the image. And most of the descriptions named words printed in the image, like "401k catch-up contribution limits". All of those landed in the top 5. The four that only described the scene split: a clinician looking at a retina scan and a glowing mechanical hub came back at rank 1, while a sepia figure among paper scrolls landed at 19 and a genealogy profile with a headstone photo landed at 43.

Four scene-only queries is too few to call a pattern, but both misses in this sample were scene descriptions. For a gallery of banners, slides and diagrams, where I remember an image by its words, that is fine. For a photo library of beaches and dogs, run your own 20 queries before you trust it.

One more thing about the scores. They are cosine similarities, and in my gallery they are compressed: a good match tends to sit around 0.70 to 0.80 and most of the field well below that. The UI shows the score next to each frame, ranks by it, and never filters on it. A threshold that works for one query cuts the right answer off the next one.

The model runs only when you search

The first version kept the query service up all the time, holding about 2.4 GB of RAM to answer a search I make a few times a week. It now starts on demand. A systemd socket holds the port, the first meaning search starts the service, and a proxy stops it after 15 minutes without a query.side quest A cold search takes about 4 seconds while the model loads. After that, about 0.1 to 0.15 seconds.

Immich reached the same answer from the other direction: its machine-learning service unloads a model after 300 seconds of inactivity by default.⁵5Immich environment variables: MACHINE_LEARNING_MODEL_TTL, "Inactivity time (s) before a model is unloaded", default 300. Good company to be in.

What I would tell you to do

If you have a folder of images with no captions, this is an afternoon. Embed thumbnails, not originals. Store the vectors next to the rest of your data. Check every vector before you keep it. Write two retrieval controls that point in opposite directions, run a keyword baseline before you quote a number, then 20 honest queries of your own, and notice which queries are doing the winning. Skip the vector database until the matrix stops fitting in memory.

On my own test the gallery went from 3 of 20 by keyword to 18 of 20 in the top 5 by description, with a model small enough that the hard part was everything around it. Awesome trade.

Related reading on this site: MnemeshippedPractical Applications·May 29, 2026Mneme: Semantic Recall for Your Claude Code SessionsA local tool that turns months of Claude Code session JSONL into searchable memory. Why I built it, how the four-mode ladder works, and the twenty-minute setup. for semantic recall over Claude Code sessions with the same small-store instinct, DGX Lab Day 3shippedPractical Applications·Oct 24, 2025DGX Lab: Building a Complete RAG Infrastructure - From Ollama to Qdrant to AnythingLLM - Day 3Building a complete self-hosted RAG infrastructure with 8 integrated services on a single DGX workstation, supporting everything from medical AI fine-tuning to document Q&A. for the text-side retrieval stack this sits next to, and Building a Markdown RAG SystemshippedAI Systems & Architecture·Mar 21, 2024Building a Markdown RAG System: A Practical Guide to Document-Grounded AIDetailed walkthrough of building a Retrieval-Augmented Generation (RAG) system for markdown documents with lightweight implementation. for where my thinking on embeddings started.