On-Device Embeddings and RAG: What EmbeddingGemma 2 Changes for App Teams

What was announced

Google introduced EmbeddingGemma 2 on October 6, 2026, describing it as a lightweight multimodal embedding model for text, images, audio, and video. Its announcement lists a 740-million-parameter model, an 8K-token context window, and Matryoshka-style output truncation options. Those are vendor-reported specifications, not a guarantee of performance on a particular device or dataset.

For product teams, the important shift is architectural: embeddings and retrieval can move closer to the user and the source data. That can reduce network dependence and keep some content local. It does not automatically make an app private, faster, or cheaper; those outcomes depend on the device, index, workload, and update strategy.

Start with the retrieval requirement

Before selecting a model, define the retrieval task:

• Which inputs must be matched: text only, or combinations of text, images, audio, and video?
• Does the app need to work offline?
• What languages, domain vocabulary, and content lengths matter?
• How current must the indexed content be?
• What is the acceptable memory, storage, and battery budget?

Build a representative evaluation set from real user queries and documents. Compare candidate models on recall at the top results, ranking quality, latency, memory, and energy on target devices. For sensitive or high-impact use cases, include a review of missed results and false matches rather than relying on one aggregate benchmark.

Dimension changes are index migrations

Truncating vectors can reduce storage and search cost, but a smaller dimension may change nearest-neighbor rankings. Treat an embedding dimension or model change as a data migration:

1. Version the model, preprocessing, and vector dimension together.
2. Generate a parallel index from the same source snapshot.
3. Compare retrieval quality and operational costs on a fixed evaluation set.
4. Route a small share of traffic to the new index if the product permits it.
5. Switch only after validating results; retain a rollback path.

Never compare vectors from incompatible model versions in one index unless the model documentation explicitly guarantees compatibility.

Design a local-first retrieval boundary

Keep the embedding pipeline separate from answer generation. A local search may return relevant passages, but generated responses can still hallucinate or expose content beyond the caller’s permissions. Apply access control before retrieval, preserve source identifiers, and show citations or snippets so users can inspect where an answer came from.

If a cloud model receives retrieved passages, disclose that transfer in the product’s data flow and privacy documentation. Local embeddings do not make a cloud completion local.

Test devices, not just notebooks

Benchmark on the oldest supported hardware, not only a development workstation. Measure cold-start time, peak resident memory, battery impact, model download size, index growth, and search latency under realistic document counts. Test behavior under low memory, interrupted downloads, and offline operation. Quantized model size on one reference device should not be treated as a universal resource estimate.

When to keep retrieval server-side

Server-side indexing can be a better fit when documents change constantly, access policies are complex, the corpus is shared across teams, or client storage is constrained. A hybrid design can keep personal or offline data local while routing shared, permission-sensitive collections to a controlled backend.

The practical decision is not “local versus cloud” in isolation. It is a choice about where content is stored, embedded, retrieved, and generated—and which system can enforce freshness, privacy, and authorization at each step.

Sources

• Google DeepMind: EmbeddingGemma 2, an open lightweight multimodal embedding model (October 6, 2026)
• EmbeddingGemma 2 model card
• Google AI Edge developer guide