| 1 | # ADR 0105: Hybrid Codebase Index (core + MCP) for C# stacks with Roslyn truth |
| 2 | |
| 3 | **Status:** Accepted · Implemented |
| 4 | **Date:** 2026-05-06 |
| 5 | |
| 6 | ## Related ADRs |
| 7 | |
| 8 | | ADR | Role | |
| 9 | |-----|------| |
| 10 | | [0039](0039-workspace-navigation-affordances.md) | Workspace navigation — multiple views and “current file + related” | |
| 11 | | [0040](0040-lsp-launch-line-settings-toml-presets-and-environment.md) | LSP (C# / Markdown) — command line in `settings.toml`: presets, optional keys, environment override | |
| 12 | | [0052](0052-agent-contract-cli-and-snapshot-tests.md) | Agent contract CLI (MCP parity) and snapshot tests | |
| 13 | | [0053](0053-semantic-map-control-flow-pfd.md) | Intent map and control flow on PFD | |
| 14 | | [0056](0056-semantic-map-pipeline-adoption.md) | Semantic map adoption of Skia composition pipeline | |
| 15 | | [0067](0067-graph-backed-surfaces-contract.md) | Graph-backed surfaces — shared contract for graph screen family | |
| 16 | | [0069](0069-markdown-preview-tool-surface-and-renderer-decoupling.md) | Markdown Preview — MFD instrument, renderer-first decoupling, no inline preview in document | |
| 17 | | [0079](0079-ide-display-system-ids-overlay-pipeline.md) | IDS vs CDS; AXAML index — not IDS | |
| 18 | | [0095](0095-workspace-solution-ide-health-stratification.md) | Three Health levels — Workspace, Solution, IDE (channel taxonomy) | |
| 19 | | [0097](0097-cockpit-compute-units-transport-to-channel-dto.md) | Cockpit compute units (CCU; LRU *Unit* analog) — layer between transport, meaning, and channel | |
| 20 | | [0098](0098-semantic-first-document-as-projection.md) | Semantics first; document and repository as projections (Semantic-First) | |
| 21 | | [0099](0099-ide-databus-typed-events-and-projections.md) | IDE DataBus — typed events and state projections | |
| 22 | | [0100](0100-project-constitution.md) | Project constitution | |
| 23 | | [0101](0101-licensing-and-commercialization-strategy.md) | Licensing and commercialization strategy | |
| 24 | | [0102](0102-data-acquisition-layer-boundary-and-contract.md) | Data Acquisition Layer — external interfaces and adapters boundary | |
| 25 | | [0106](0106-hybrid-codebase-index-cascadeide-integration-and-semantic-map.md) | Hybrid Codebase Index — CascadeIDE integration, freshness, Semantic Map | |
| 26 | |
| 27 | ## Summary |
| 28 | |
| 29 | - **Hybrid codebase index:** portable core + MCP; **Roslyn is truth for C#**. |
| 30 | - SQLite **FTS5** (keyword) + optional **vec** (semantic); fusion α/β. |
| 31 | - Scope: C#, Razor, AXAML, web stacks in one workspace; ADR 0106 — CIDE integration. |
| 32 | |
| 33 | --- |
| 34 | |
| 35 | <a id="adr0105-glossary"></a> |
| 36 | |
| 37 | ## Terms and abbreviations |
| 38 | |
| 39 | Working definitions **within this ADR**; algorithm details — per SQLite / chosen embedding provider documentation. |
| 40 | |
| 41 | | Term | Meaning here | |
| 42 | | --- | --- | |
| 43 | | **FTS** (*full-text search*) | Full-text search: index and queries over **tokens/words** inside document texts (file or chunk), not only exact field match or filename search. | |
| 44 | | **FTS5** | Fifth **SQLite** full-text module: FTS5 virtual tables, inverted index “term → document occurrences”, relevance-aware queries. In this ADR — primary **keyword** backend of layer B. | |
| 45 | | **Inverted index** | Structure “word/term → list of documents (and positions)” backing fast FTS; not to be confused with Roslyn **symbol graph**. | |
| 46 | | **BM25** (*Best Matching 25*, Okapi BM25 family) | Class of statistical **ranking functions** for full-text hits: balance “term frequent in this document” vs “term rare in corpus”. In SQLite FTS5 relevance uses auxiliary rank functions (including **`bm25()`**); in this ADR “keyword / BM25” means **full-text with such ranking**, not a separate engine outside SQLite. | |
| 47 | | **Keyword search** | Search by **word/phrase match** (via FTS), without required “understanding” of the query in different phrasings. | |
| 48 | | **Embedding** | Fixed-dimension vector from a model over text (code fragment, paragraph, query). Semantically similar texts ideally get **close** vectors in the chosen metric. | |
| 49 | | **Semantic / vector search** | Select fragments by **embedding proximity** of query and chunks (cosine similarity, etc.), not keyword match. In this ADR also **vec** (vector channel). | |
| 50 | | **Vector store** | Storage for vectors and metadata (chunk id, path, line range), with nearest-neighbor operations (ANN / full scan at small scale). | |
| 51 | | **sqlite-vec** | **SQLite** extension for vector storage and query; in this ADR — optional local vector store **beside** FTS, not replacing the keyword layer. | |
| 52 | | **Fusion** | Merge hit lists from **two channels** (here FTS and vec): score normalization, weighted sum or equivalent, final top‑N. See [§ fusion sketch](#adr0105-impl-sketch-fusion). | |
| 53 | | **Chunk** | Continuous file fragment indexed as one FTS/vec unit (line window, logical block, etc.); see [§ chunking](#adr0105-impl-sketch-chunking). | |
| 54 | | **MCP** | *Model Context Protocol* — transport and tool contract for agents/IDE; separate index MCP service in [§ deployment](#adr0105-deployment). | |
| 55 | | **DAL** | *Data Acquisition Layer* — layer for data from workspace and external world per [0102](0102-data-acquisition-layer-boundary-and-contract.md). | |
| 56 | | **CCU** | *Cockpit Compute Unit(s)* — packaging compute results into stable channel DTOs per [0097](0097-cockpit-compute-units-transport-to-channel-dto.md). | |
| 57 | |
| 58 | --- |
| 59 | |
| 60 | <a id="adr0105-context"></a> |
| 61 | |
| 62 | ## Context |
| 63 | |
| 64 | CascadeIDE is an MCP-first IDE: the agent must orient quickly in the codebase and assemble context in a small model window (or under a limited step/call budget). |
| 65 | |
| 66 | For **any** .NET/C# solution we already have a “source of truth” for precise semantic operations: |
| 67 | |
| 68 | - Roslyn (via roslyn-mcp and IDE wiring) for: diagnostics, go-to-definition, find-usages, rename, symbol-level navigation. |
| 69 | |
| 70 | But Roslyn does not fully solve: |
| 71 | |
| 72 | - fast “sense overview” and a “first map” of the solution without reading dozens of files; |
| 73 | - full-text and orientation over **Markdown**, configs, `.csproj` / `.sln` / `.slnx`, YAML/TOML, the **web layer** (**Razor/Blazor `.razor`**, HTML/CSS), **Avalonia (`.axaml`)** markup, and other artifacts **without** a Roslyn semantic model for those formats; |
| 74 | - for a **plain** C# project (including **CascadeIDE** itself) the same hybrid layer gives fast keyword/optional semantic over the **whole repository** — including **`.cs` as text** ([layer B](#adr0105-layer-b): FTS only, not symbols), while rename/impact stay on Roslyn; |
| 75 | - persistence across sessions: the “map” should live beside the project/IDE profile and not require re-training the agent every time. |
| 76 | |
| 77 | External solutions exist (e.g. SocratiCode) with hybrid search + graph + impact, but they add infrastructure load (Docker/Qdrant/Ollama) and license risk (AGPL) for product integration. |
| 78 | |
| 79 | Additionally: CascadeIDE is cross-platform (Avalonia). We do not want the critical navigation layer tied to Windows-only/drivers/Docker, but on Linux we may allow heavier backend options. |
| 80 | |
| 81 | --- |
| 82 | |
| 83 | <a id="adr0105-decision-summary"></a> |
| 84 | |
| 85 | ## Decision in one sentence |
| 86 | |
| 87 | Introduce a **two-layer navigation model**: **Roslyn is truth for C# semantics**, beside a **light hybrid index** over the **solution contour**: web artifacts (`.razor`, MD, HTML/CSS), **Avalonia `.axaml`** (and pairing heuristic with code-behind `.cs` when needed), configuration and companions (**including optional full-text on `.cs` as text**, without replacing symbol-level operations); keyword + optional semantics; minimal ops cost and cross-platform support. |
| 88 | |
| 89 | --- |
| 90 | |
| 91 | <a id="adr0105-goals"></a> |
| 92 | |
| 93 | ## Goals |
| 94 | |
| 95 | 1. **Reduce agent step count**: 1–2 calls → enough relevant context to decide. |
| 96 | 2. Provide a “first map” without “read 20 files”: top files/nodes/flows, entry points — for **Blazor/Web**, **Avalonia (AXAML + bindings/control names)**, and **plain C#**, including **developing CascadeIDE itself** on the same tool stack. |
| 97 | 3. Preserve **semantic correctness**: C# refactor-impact is Roslyn-based, not heuristic. |
| 98 | 4. Work **without mandatory Docker** (especially on Windows), with predictable local install/update. |
| 99 | 5. Be cross-platform (Windows/Linux/macOS), with optional backend accelerators on Linux. |
| 100 | |
| 101 | --- |
| 102 | |
| 103 | <a id="adr0105-non-goals"></a> |
| 104 | |
| 105 | ## Non-goals (first phase) |
| 106 | |
| 107 | - Full “polyglot dependency graph” across 18+ languages. |
| 108 | - Replacing Roslyn MCP: Roslyn remains the truth layer for C#. |
| 109 | - Mandatory vector DB/containers for baseline scenarios. |
| 110 | - “One graph that is always right”: graph/impact outside C# allows heuristics and needs verification. |
| 111 | |
| 112 | --- |
| 113 | |
| 114 | <a id="adr0105-architecture"></a> |
| 115 | |
| 116 | ## Architecture (by layer) |
| 117 | |
| 118 | <a id="adr0105-layer-a"></a> |
| 119 | |
| 120 | ### Layer A: Roslyn truth (C#) |
| 121 | |
| 122 | Use Roslyn for: |
| 123 | |
| 124 | - diagnostics / code actions; |
| 125 | - find usages / rename; |
| 126 | - symbol navigation; |
| 127 | - (where possible) call graph / entrypoints within a C# project. |
| 128 | |
| 129 | This layer is **precise** but “expensive” in workflow: the agent still needs to know *what to search for*. |
| 130 | |
| 131 | <a id="adr0105-layer-b"></a> |
| 132 | |
| 133 | ### Layer B: Hybrid index (artifacts around C#, web layer, Avalonia AXAML, optional `.cs` text) |
| 134 | |
| 135 | Index for files and fragments **outside** Roslyn symbolism or **as text** (not as a type graph): |
| 136 | |
| 137 | - `.razor`, `.razor.cs` (including partial / file pairing); |
| 138 | - `.md` / `.mdx`; |
| 139 | - `.html`, `.css`, `.scss` (including `@import`, classes/selectors); |
| 140 | - basic configs (`appsettings*.json`, `.editorconfig`, `*.props`, `*.targets`, `*.csproj`, `*.slnx`, pipeline YAML, `*.yml`, `*.toml`, etc.); |
| 141 | - **`.axaml`** (and typical code-behind `*.axaml.cs` if present): markup and attributes — **as text for FTS** and light heuristics (`x:Name`, `{Binding …}`, `Classes=`, `avares:` paths); **not** a substitute for an Avalonia XAML parser, **not** CDS/IDS semantics (see [0079 — CDS vs IDS](0079-ide-display-system-ids-overlay-pipeline.md#adr0079-cds-vs-ids)); |
| 142 | - **`*.cs` (index option):** **full-text/keyword only** (identifiers and strings match as text in the file); **rename/find-usages/impact** remain Roslyn-only. Tool responses must mark `.cs` hits as **text-ranked** so they are not mixed with symbol truth. |
| 143 | |
| 144 | The index provides: |
| 145 | |
| 146 | - **keyword / BM25**: config strings, CSS, Razor routes, `.cs`/`.axaml`/doc fragments; |
| 147 | - **optional semantic**: “by meaning” search (embeddings), without mandatory Docker. |
| 148 | |
| 149 | Index data: |
| 150 | |
| 151 | - stored locally (IDE profile or beside the project); |
| 152 | - updated incrementally (watcher + hash); |
| 153 | - explicit format versioning (so migration does not break UX). |
| 154 | |
| 155 | <a id="adr0105-storage"></a> |
| 156 | |
| 157 | #### Storage / backend (baseline) |
| 158 | |
| 159 | Recommended default (no Docker, cross-platform): |
| 160 | |
| 161 | - **Keyword/BM25**: SQLite **FTS5** (on-disk local DB) as fast full-text index. |
| 162 | - **Semantic vectors (optional)**: SQLite + **`sqlite-vec`** as local vector store (enabled only when semantics are on). |
| 163 | |
| 164 | The engine here is **classic SQLite** (e.g. `Microsoft.Data.Sqlite` or another provider to the same SQLite library), **not** [WitDatabase](https://github.com/dmitrat/WitDatabase) (`*.witdb`): Wit stays for CascadeIDE application data; the index file is a separate on-disk SQLite. |
| 165 | |
| 166 | Important: hybrid = **FTS (keyword)** + **vec (semantic)** as two independent sub-indexes merged at the service layer (ranking/fusion), not “one DB magic”. |
| 167 | |
| 168 | <a id="adr0105-layer-c"></a> |
| 169 | |
| 170 | ### Layer C: Composition (agent workflow, portable) |
| 171 | |
| 172 | Default agent scenario (outside a specific IDE): |
| 173 | |
| 174 | 1. Hybrid search (fast, cheap) → top-N fragments and map. |
| 175 | 2. Roslyn navigation for precise C# check/refactor. |
| 176 | 3. Point reads of files/fragments only after search. |
| 177 | |
| 178 | **Embedding this scenario in CascadeIDE** (buttons, channels, debounced reindex, CCU/DataBus, Semantic Map) — **[ADR 0106](0106-hybrid-codebase-index-cascadeide-integration-and-semantic-map.md)**. |
| 179 | |
| 180 | <a id="adr0105-deployment"></a> |
| 181 | |
| 182 | ### Deployment: library + separate MCP |
| 183 | |
| 184 | Package the index as a **shared library** (core: indexing, SQLite, request/response formats) and a **separate MCP server** (thin stdio layer + tool registration) so that: |
| 185 | |
| 186 | - search can be used **outside CascadeIDE** (other MCP IDE/agents, CLI, automation); |
| 187 | - the heavy process (watcher, SQLite files, optional embeddings) is isolated: restarts and updates do not mix with Avalonia/UI. |
| 188 | |
| 189 | CascadeIDE may use **the same core in-proc** or launch **the same MCP binary** as a child process — **tool ids and contracts** stay shared for both (cockpit placement details — [0106](0106-hybrid-codebase-index-cascadeide-integration-and-semantic-map.md)). |
| 190 | |
| 191 | --- |
| 192 | |
| 193 | <a id="adr0105-config-ux"></a> |
| 194 | |
| 195 | ## Configuration and UX invariants |
| 196 | |
| 197 | - **Off-by-default for infrastructure**: if semantic embeddings need an external provider, that must be opt-in. |
| 198 | - **Cross-platform**: same tool ids/contracts in MCP; difference only in backend provider. |
| 199 | - **Small-window operation**: tool responses should be “compact by default” (top-N, with path/range/score), with a separate command to expand. |
| 200 | |
| 201 | --- |
| 202 | |
| 203 | <a id="adr0105-impl-watchouts"></a> |
| 204 | |
| 205 | ## Implementation watchouts |
| 206 | |
| 207 | Operational points without which dogfood and production disappoint quickly: |
| 208 | |
| 209 | <a id="adr0105-impl-watchouts-volume"></a> |
| 210 | |
| 211 | 1. **Volume and noise.** FTS over all `*.cs` inflates the index and can **pollute top-N** with raw string hits. Need explicit **defaults and filters** in `settings.toml` (or equivalent): ignores/`gitignore` alignment, path masks, **ranking** (e.g. prioritize docs/configs over “raw” `.cs`, or the opposite — “code first” mode), ability to temporarily exclude `*.cs` from FTS without disabling the rest of the index. |
| 212 | |
| 213 | <a id="adr0105-impl-watchouts-freshness"></a> |
| 214 | |
| 215 | 2. **Freshness** on saves from **CascadeIDE**. Cheap increment and lag-free UX — **[ADR 0106](0106-hybrid-codebase-index-cascadeide-integration-and-semantic-map.md)**. MCP/core may use a watcher and incremental reindex; product tie-in with the editor session — in the IDE. |
| 216 | |
| 217 | <a id="adr0105-impl-watchouts-hit-kind"></a> |
| 218 | |
| 219 | 3. **MCP contract from the first prototype.** Search response structure needs a **stable hit-type field** (e.g. `hit_kind`: `text_fts` / `text_vector` / `symbol_followup_roslyn` or equivalent) so agent and human do not guess from free text. Changing field semantics later costs more than baking it in v0. |
| 220 | |
| 221 | --- |
| 222 | |
| 223 | <a id="adr0105-alternatives"></a> |
| 224 | |
| 225 | ## Alternatives and why not (for now) |
| 226 | |
| 227 | <a id="adr0105-alt-roslyn-grep"></a> |
| 228 | |
| 229 | ### A) “Roslyn + grep only” |
| 230 | |
| 231 | Pros: minimal infrastructure, high C# accuracy. |
| 232 | Cons: too many steps and file reads for agent scenarios; poor coverage of docs/config/web and **global** “where mentioned” across the repo without a heavy Roslyn-only sweep. |
| 233 | |
| 234 | <a id="adr0105-alt-socraticode"></a> |
| 235 | |
| 236 | ### B) Embed SocratiCode wholesale |
| 237 | |
| 238 | Pros: ready hybrid+graph+impact layer, fast “orientation” on a large repo. |
| 239 | Cons: |
| 240 | - ops: Docker/Qdrant/Ollama in baseline; |
| 241 | - graph correctness outside C# depends on heuristics; |
| 242 | - **AGPL license** — undesirable for product embedding (see [0101](0101-licensing-and-commercialization-strategy.md)). |
| 243 | |
| 244 | <a id="adr0105-alt-lsp-all"></a> |
| 245 | |
| 246 | ### C) LSP for everything (full polyglot) |
| 247 | |
| 248 | Pros: potential semantic accuracy per language. |
| 249 | Cons: too large operational and integration cost; does not solve “small window/few calls” without a separate index/ranking layer. |
| 250 | |
| 251 | --- |
| 252 | |
| 253 | <a id="adr0105-consequences"></a> |
| 254 | |
| 255 | ## Consequences |
| 256 | |
| 257 | <a id="adr0105-consequences-positive"></a> |
| 258 | |
| 259 | ### Positive |
| 260 | |
| 261 | - The agent gets a fast “first pass” over the solution **and** can **dogfood** the same index while developing **CascadeIDE** and other C# repos, not limited to “Blazor only”. |
| 262 | - Roslyn remains “truth” for dangerous operations (rename/impact/diagnostics). |
| 263 | - Docker becomes optional: Windows-friendly baseline; Linux may get extended modes. |
| 264 | |
| 265 | <a id="adr0105-consequences-risks"></a> |
| 266 | |
| 267 | ### Negative / risks |
| 268 | |
| 269 | - A new data layer (index) → versions, migrations, observability needed. |
| 270 | - Risk of false links in `.razor`/CSS/HTML heuristics → need “confidence” and explicit “hint” labeling. |
| 271 | - Indexing `.cs`/`.axaml` as text may **look like** “semantic find” → see [§ implementation watchouts](#adr0105-impl-watchouts) ([`hit_kind`](#adr0105-impl-watchouts-hit-kind), [ranking](#adr0105-impl-watchouts-volume)). |
| 272 | - Tools must stay compact or the hybrid index may spam context and hurt UX. |
| 273 | |
| 274 | --- |
| 275 | |
| 276 | <a id="adr0105-rollout-plan"></a> |
| 277 | |
| 278 | ## Rollout plan (portable core + MCP): status |
| 279 | |
| 280 | | Step | Content | ADR 0105 scope (implemented) | |
| 281 | | --- | --- | --- | |
| 282 | | 1 | MCP contracts (`search`, `status`, `reindex`, `explain-result`, version/`hit_kind`); core in library | ✅ **`hybrid-codebase-index`** repo | |
| 283 | | 2 | Keyword index, increment, ignores; optional FTS on `*.cs`; watcher tool | ✅ | |
| 284 | | 3 | Razor / AXAML: `.razor`↔`.razor.cs`, `.axaml`↔`.axaml.cs` pairs; heuristic headers `__hci_*` (directives, resources, bindings, tags) | ✅ (`HybridCodebaseIndex.Core` augment) | |
| 285 | | 4 | Embeddings opt-in (`settings.toml`), sqlite-vec optional | ✅ | |
| 286 | | 5 | IDE workflow + freshness on save | → **[ADR 0106](0106-hybrid-codebase-index-cascadeide-integration-and-semantic-map.md)** | |
| 287 | | 6 | Scope defaults, FTS chunking, FTS+vec fusion | ✅ `settings.toml` + hybrid search | |
| 288 | |
| 289 | --- |
| 290 | |
| 291 | <a id="adr0105-impl-sketch-scope-chunk-fusion"></a> |
| 292 | |
| 293 | ## Sketch: index scope, chunks, fusion (FTS + vec) |
| 294 | |
| 295 | Addendum to [rollout plan](#adr0105-rollout-plan): reasonable defaults at spike, without changing top-level architecture ([layer B](#adr0105-layer-b), [storage](#adr0105-storage)). |
| 296 | |
| 297 | <a id="adr0105-impl-sketch-scope"></a> |
| 298 | |
| 299 | ### Index scope |
| 300 | |
| 301 | - **Primary anchor:** active `.sln` / main `.csproj` of the CascadeIDE profile — same workspace contour as the Roslyn session. |
| 302 | - **Default extension:** paths under **workspace root**, minus aligned **`.gitignore`** (and if needed `.cursorignore` against agent noise) and a **hard denylist**: `bin/`, `obj/`, `node_modules/`, `.git/`, typical tool cache dirs. |
| 303 | - **Monorepo:** one index DB per **(workspace_root, solution_path)** pair; another solution in the same tree — separate index contour (switch by profile). Field **`extra_include_roots`** in `settings.toml` for sibling dirs (docs, external KB, etc.) — **opt-in**. |
| 304 | |
| 305 | <a id="adr0105-impl-sketch-chunking"></a> |
| 306 | |
| 307 | ### Chunking for FTS |
| 308 | |
| 309 | | Type | Strategy | |
| 310 | | --- | --- | |
| 311 | | Compact configs, small `.md`, `.razor` within size limit | One FTS document per file; upper document size limit (e.g. 256–512 KiB) — configurable. | |
| 312 | | Long `.md`, `.cs`, `.axaml` | Sliding **line windows** (guide: 80–120 lines, overlap 10–15); stable `chunk_id`: path + range (`start_line` / offset). | |
| 313 | | `.razor` | Prefer **logical boundaries** (`@code`, large markup blocks); if not cheap — same line windows. | |
| 314 | |
| 315 | **Freshness:** on edit rebuild only affected chunks; for small files whole-document rebuild is allowed. Tool response always gives **path and line range** (or offset) so agent and human open the point without guessing. |
| 316 | |
| 317 | <a id="adr0105-impl-sketch-fusion"></a> |
| 318 | |
| 319 | ### Fusion of keyword (FTS) and semantic (vec), v0 |
| 320 | |
| 321 | 1. Independently get **top‑K** from FTS and vec (internal K, guide 20–40; outward after merge — compact top‑N). |
| 322 | 2. **Normalize** scores within each channel (min-max or rank-based, e.g. `1/(rank+R)`). |
| 323 | 3. Merge unique chunks: final score **`S = α·S_fts + β·S_vec`**; if a chunk is missing in a channel — that channel’s contribution is **0**. |
| 324 | 4. **Default with vec on:** `α ≈ 0.65`, `β ≈ 0.35`; with vec off — FTS only. |
| 325 | 5. **Short query (1–2 tokens)** or low max `S_vec`: boost FTS or do not mix vec (keyword-dominant mode). |
| 326 | |
| 327 | In the DTO, keep **both contributions** (`fts_score`, `vec_score` when present) with `hit_kind` and final rank — so explainability (“why in top”) is preserved. Thresholds and weights in `settings.toml` without breaking response format on later iterations. |
| 328 | |