diff --git a/.gitignore b/.gitignore index af0730eb7fd..d3aceba741a 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ website/.hugo_build.lock website/public website/resources website/content/en/docs +website/static/diagrams +tools/diagram/d3.min.js e2e_integration_test* active-query-tracker dist/ diff --git a/docs/architecture-diagram.md b/docs/architecture-diagram.md new file mode 100644 index 00000000000..542c6ab6541 --- /dev/null +++ b/docs/architecture-diagram.md @@ -0,0 +1,34 @@ +--- +title: "Interactive Architecture Diagram" +linkTitle: "Interactive Diagram" +weight: 3 +slug: architecture-diagram +--- + +The [interactive architecture diagram](../tools/diagram/cortex-architecture.html) +draws the same system [Architecture](./architecture.md) describes in prose, with the +details attached to the picture instead of scattered through the text. It covers +the write path, the read path, the blocks lifecycle and the optional services — +ruler, alertmanager, compactor, store-gateway, query-scheduler and the caches. +Hover a connector and it names the protocol and the endpoint that hop actually +uses; select a component and it gives you the role, whether it is stateful, which +hash ring it joins, the endpoints it serves, its `-target` value and the file in +the Cortex tree that implements it. + +Three toggles cover the places where the topology genuinely forks, rather than +drawing one deployment and calling it typical: the query-frontend's own queue +versus a separate query-scheduler, the ruler evaluating rules in its own querier +stack versus delegating to the query-frontend with `-ruler.frontend-address`, and +the parquet queryable off versus on. There are also guided walkthroughs that step +through the write, read, rule-evaluation and blocks flows one hop at a time, a +table view of every component and flow, and a dark-mode toggle. + +The diagram's metadata is hand-maintained against the Cortex source rather than +generated from it, so the `src` path shown in each component's panel is the +authority — if a ring key, prefix or endpoint disagrees with the code, the code is +right and the diagram needs fixing. It also deliberately shows a few things the +prose does not yet cover, such as the OTLP ingest endpoint and the +parquet-converter, which is marked experimental for that reason. Its source lives +in [`tools/diagram/`](https://github.com/cortexproject/cortex/tree/master/tools/diagram). + +**[Open the interactive architecture diagram →](../tools/diagram/cortex-architecture.html)** diff --git a/docs/architecture.md b/docs/architecture.md index b532d83239a..1b4f602a087 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,9 +7,7 @@ slug: architecture Cortex consists of multiple horizontally scalable microservices. Each microservice uses the most appropriate technique for horizontal scaling; most are stateless and can handle requests for any users while some (namely the [ingesters](#ingester)) are semi-stateful and depend on consistent hashing. This document provides a basic overview of Cortex's architecture. -The following diagram does not include all the Cortex services, but does represent a typical deployment topology. - -

Cortex Architecture

+The [interactive architecture diagram](./architecture-diagram.md) draws the same system, and does include the services this document leaves out. Hover a connector for the protocol and endpoint of that hop; select a component for its hash ring, endpoints and source file. ## The role of Prometheus diff --git a/docs/blocks-storage/_index.md b/docs/blocks-storage/_index.md index bcec5ae48b3..c67dcd755eb 100644 --- a/docs/blocks-storage/_index.md +++ b/docs/blocks-storage/_index.md @@ -1,7 +1,7 @@ --- title: "Blocks Storage" linkTitle: "Blocks Storage" -weight: 3 +weight: 4 menu: --- diff --git a/tools/diagram/cortex-architecture.html b/tools/diagram/cortex-architecture.html new file mode 100644 index 00000000000..c32e53350d5 --- /dev/null +++ b/tools/diagram/cortex-architecture.html @@ -0,0 +1,1360 @@ + + + + + + +Cortex architecture — interactive diagram + + + + + + + + + + + + + +
+ +
+

Cortex architecture

+

+ The write path, read path and blocks lifecycle, as the code actually wires them up. + Hover a connector for its protocol and endpoint; select a component for its ring, + endpoints and source file. Nothing here is generated — the topology is + hand-maintained, so treat the source references in each panel as the authority. +

+
+ +
+
+ Show +
+
+ +
+ Query queue +
+ + +
+
+ +
+ Ruler evaluates via +
+ + +
+
+ +
+ Parquet queryable +
+ + +
+
+ +
+ Walk a flow +
+
+ +
+ View +
+ + + +
+
+
+ +
+
+ +
+ +
+ +
+ + + + +
+ + + + + + + diff --git a/tools/diagram/readme.md b/tools/diagram/readme.md index fddc3a0b765..17f296b1dac 100644 --- a/tools/diagram/readme.md +++ b/tools/diagram/readme.md @@ -1,3 +1,103 @@ -# Originals +# Architecture diagram sources -You can edit the original drawings using [https://www.draw.io/](https://www.draw.io/) +## `cortex-architecture.html` — interactive diagram + +Open it in a browser; `file://` works and there is no build step: + +```sh +open tools/diagram/cortex-architecture.html +``` + +Its only dependency is D3, which is not committed here — see +[below](#d3minjs--not-committed-comes-from-npm). + +It renders the write path, read path, blocks lifecycle and optional services as +an interactive [D3](https://d3js.org/) node-link diagram. Hover a connector for +its protocol and endpoint; select a component for its role, statefulness, hash +ring, endpoints and source file. There are toggles for the three places the +topology genuinely forks — the query-scheduler vs. the query-frontend's own +queue, the ruler's own querier stack vs. `-ruler.frontend-address`, and the +parquet queryable off vs. on — plus guided walkthroughs of the write, read, rule-evaluation and +blocks flows, a table view, and a dark-mode toggle. The page is light by +default, matching the rest of cortexmetrics.io; the OS colour-scheme setting is +deliberately not consulted. + +All seven caches are drawn separately because no two share a consumer set: the +results cache belongs to the query-frontend; the index cache to the store-gateway +alone; the chunks cache to the store-gateway and querier; the metadata cache to +those two plus the compactor; the two parquet caches to the querier and the +store-gateway's parquet stores, only with the parquet queryable enabled; and the +expanded postings cache to the ingester, in-process, configured under +`blocks-storage.tsdb.*` rather than with the bucket-store caches. + +### Editing it + +Components and connections live in the `NODES` and `EDGES` arrays near the top +of the single `