# DOMVS AVREA RAG export

This folder is the provider-neutral retrieval layer for the MD Atlas. It is
generated from the project database, all project `replit.md` files, the ebook
archive, and `orbitron-lemarche.md`.

## Files

- `RAG-MANIFEST.json` — schema, corpus counts, chunking rules, metadata filters,
  and coding/sales retrieval guidance.
- `RAG-DOCUMENTS.jsonl` — one normalized source document per line.
- `RAG-CHUNKS.jsonl` — heading-aware source chunks with heading paths and the
  original provider-neutral embedding fields.
- `RAG-VECTORS.jsonl` — the same chunks enriched with their Mistral vectors.
- `RAG-VECTOR-MONOLITH.html` — standalone HTML5 vector browser with all
  1,814 vectors embedded inline. It includes contextual chunk neighborhoods,
  similar-document links, citation copy, shareable query URLs, and a grounded
  chat that retrieves first and validates answer citations. It calls the
  Glow-Up proxy for query vectors and Mistral chat, then falls back to local
  lexical search when offline.
- `RAG-VECTOR-LAB.html` — fast browser client. It loads the interface and
  source metadata first, then lazy-loads `RAG-VECTORS.f32` only when semantic
  retrieval or related-document ranking is requested. Each result can expand
  its full Markdown source on demand. For grounded chat, a strong `replit.md`
  hit may additionally load that single document's full text; the chat still
  never receives the corpus or the raw vector file.
- `RAG-VECTORS.f32` — compact little-endian Float32 vector pack in the exact
  JSONL chunk order; `1,814 × 1,024 × 4` bytes.
- `RAG-SEARCH-INDEX.json` — browser-friendly full-text index used by the
  Atlas's enhanced manual search.

The Atlas and Vector Lab share the `atlas-collections` localStorage namespace.
Documents can be collected from the Reader, while retrieved chunks can be
saved directly from Vector Lab into the first active collection (or a default
`Research` collection).

Chat uses `mistral-large-latest` through `/mistral/chat`. It sends at most six
retrieved chunks (12,000 characters) plus at most one strong `replit.md` source
(60,000 characters, with an explicit truncation marker when necessary).

## Regenerate

From the workspace root:

```bash
python3 tools/build_rag_index.py
python3 tools/embed_rag_index.py
```

The command is deterministic apart from the manifest's `generated_at` field.
Embedding generation reuses cached vectors when chunk IDs and source text are
unchanged, so metadata-only index updates do not call the provider again.
It fails explicitly if a source document referenced by the project database is
missing.

## Ingestion contract

The current vector export was generated through
`https://glow-up-gemini.replit.app/mistral/embeddings` using a JSON body with
`input: [...]` and `model: "mistral-embed"`. The response uses the OpenAI-style
`data[].index` and `data[].embedding` fields and returns 1,024 dimensions.
Store the top-level metadata from `RAG-VECTORS.jsonl` as filterable fields.
Always validate the returned model and dimensions before caching vectors:
another proxy route can return a different 1,536-dimensional model even when
`mistral-embed` is requested.

- Coding assistant: `use_cases` contains `coding`
- Sales assistant: `use_cases` contains `sales`
- Narrow further with `cluster`, `status`, `rebuild_grade`, `tags`, or
  `tech_stack`

Use cosine similarity on normalized vectors and return the chunk text,
document title, source path, heading path, domain, and metadata to the chatbot
context. Keep the returned source path in the final answer so the assistant can
cite the underlying Markdown document. The browser chatbot only treats an
answer as validated when it includes numbered citations that point to retrieved
sources; otherwise it visibly marks the answer for source review.