| 1 | <!-- English translation of adr/0054-benchmarking-methodology-and-baselines.md. Canonical Russian: ../../adr/0054-benchmarking-methodology-and-baselines.md --> |
| 2 | |
| 3 | # ADR 0054: Performance benchmarks and baseline metrics |
| 4 | |
| 5 | **Status:** Proposed |
| 6 | **Date:** 2026-04-17 |
| 7 | |
| 8 | ## Related ADRs |
| 9 | |
| 10 | | ADR | Role | |
| 11 | |-----|------| |
| 12 | | [0021](0021-pfd-mfd-cockpit-attention-model.md) | PFD/MFD - Cascade IDE Cockpit Attention Model | |
| 13 | | [0049](0049-skia-surface-rollout-over-avalonia-host.md) | Staged rollout of Skia-surfaces at Avalonia-host (CIDE-wide) | |
| 14 | | [0053](0053-semantic-map-control-flow-pfd.md) | Intent map and control flow on PFD (control flow) | |
| 15 | |
| 16 | --- |
| 17 | ## Context |
| 18 | |
| 19 | The observed difference in memory consumption between different tools (for example, CIDE vs Cursor) affects product decisions and optimization priorities. Currently, such comparisons are made sporadically and do not have a repeatable protocol. |
| 20 | |
| 21 | We need a unified way to measure and discuss performance without the “measured under different conditions” debate. |
| 22 | |
| 23 | --- |
| 24 | |
| 25 | ## Solution |
| 26 | |
| 27 | 1. Introduce a standard set of benchmark scripts for CIDE. |
| 28 | 2. Record baseline metrics in the CIDE repository as an engineering artifact. |
| 29 | 3. Use benchmarks as a gate for major UI/rendering/navigation changes. |
| 30 | |
| 31 | --- |
| 32 | |
| 33 | ## Scripts v1 |
| 34 | |
| 35 | - `idle`: the application is running, the solution is not loaded, 60 sec stabilization. |
| 36 | - `solution_open`: loaded generic `.sln`, without active build/test/debug. |
| 37 | - `editing`: medium-sized C# file open, normal navigation/scrolling. |
| 38 | - `semantic_map_file`: map in `file` level. |
| 39 | - `semantic_map_controlFlow`: map in the `controlFlow` level. |
| 40 | - `chat_active`: chat page is open, messaging is active (without long background tasks). |
| 41 | - `debug_session`: attach/launch + pause at breakpoint. |
| 42 | |
| 43 | --- |
| 44 | |
| 45 | ## Metrics v1 |
| 46 | |
| 47 | - Process memory: |
| 48 | - `WorkingSet (MB)` |
| 49 | - `PrivateBytes (MB)` |
| 50 | - CPU: |
| 51 | - average load per observation window |
| 52 | - peak load |
| 53 | - Response time: |
| 54 | - cold start time until the finished window |
| 55 | - loading time of the solution to a stable state |
| 56 | - Semantic map: |
| 57 | - map update time after file/carriage change |
| 58 | - number of nodes/edges in the resulting subgraph |
| 59 | |
| 60 | --- |
| 61 | |
| 62 | ## Measurement protocol |
| 63 | |
| 64 | - OS, build (`Debug`/`Release`), target runtime and config are recorded in the report. |
| 65 | - For each scenario, at least 3 runs, report in the form of median + max. |
| 66 | - Compare only scenarios measured on the same machine and in the same profile. |
| 67 | - If an external reference is used (eg Cursor), this is specified as a `reference` rather than a hard SLA. |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## Artifacts |
| 72 | |
| 73 | - In the repository: |
| 74 | - `docs/benchmarks/README.md` - how to run and read the results. |
| 75 | - `docs/benchmarks/baselines/*.json` — baseline by date/branch. |
| 76 | - `docs/benchmarks/reports/*.md` - human-readable reports. |
| 77 | - For CI (later): smoke benchmark on a limited set of scenarios. |
| 78 | |
| 79 | --- |
| 80 | |
| 81 | ## Consequences |
| 82 | |
| 83 | Pros: |
| 84 | - comparisons become repeatable and transparent; |
| 85 | - it’s easier to justify optimizations and regressions; |
| 86 | - the “fast/slow” discussion is based on facts. |
| 87 | |
| 88 | Cons: |
| 89 | - there is an operational burden on running and maintaining baseline; |
| 90 | - without discipline, the protocol quickly becomes outdated. |
| 91 | |
| 92 | --- |
| 93 | |
| 94 | ## Open questions |
| 95 | |
| 96 | - Is it necessary to separate the baseline for `Debug` and `Release` as two equal contours? |
| 97 | - What scenarios will become mandatory for PR-gate in CI (except smoke)? |
| 98 | - Is it necessary to auto-publish a mini-dashboard of baseline changes between commits? |