Skip to content

docs: document the Zitadel service identity and trim the generated specs #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Aloento merged 2 commits into from
Sep 23, 2026
Merged

docs: document the Zitadel service identity and trim the generated specs #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c7adc9a
docs: document the Zitadel service identity of the reporter
Aloento Sep 23, 2026
1f1c9f5
docs: drop the generated specifications and the duplicated module pages
Aloento Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ metrics-processor is there to address 2 primary needs:
- `src/` - Rust source code
- `doc/` - Documentation sources (mdbook)
- `tests/` - Integration and validation tests
- `specs/` - Feature specifications and implementation plans
- `playbooks/` - Operational playbooks

## Documentation
Expand Down Expand Up @@ -63,7 +62,6 @@ mdbook serve doc/
| [API Reference](doc/api/) | REST endpoints, authentication, examples |
| [Configuration](doc/configuration/) | Config schema, examples, validation |
| [Integration](doc/integration/) | TSDB interface, adding new backends |
| [Modules](doc/modules/) | Rust module documentation |
| [Guides](doc/guides/) | Troubleshooting, deployment |

| [Testing](doc/testing.md) | Testing guide, fixtures, coverage |
Expand Down
8 changes: 0 additions & 8 deletions doc/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,6 @@
- [Graphite Backend](integration/graphite.md)
- [Adding New Backends](integration/adding-backends.md)

# Modules
- [Overview](modules/overview.md)
- [API Module](modules/api.md)
- [Config Module](modules/config.md)
- [Types Module](modules/types.md)
- [Graphite Module](modules/graphite.md)
- [Common Module](modules/common.md)

# Operational Guides
- [Troubleshooting](guides/troubleshooting.md)
- [Deployment](guides/deployment.md)
Expand Down
280 changes: 179 additions & 101 deletions doc/api/authentication.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,153 +1,231 @@
# Authentication

This document describes the authentication mechanism used for integrating with the CloudMon Status Dashboard.
# Status Dashboard Authentication

## Overview

The metrics-processor uses JWT (JSON Web Token) authentication when reporting component status to the status-dashboard API. This is specifically used by the `cloudmon-metrics-reporter` component to securely communicate health status updates.
The `cloudmon-metrics-reporter` authenticates against the Status Dashboard with a Zitadel OIDC
service identity. MP does not implement the OAuth flow itself: the JWT Profile exchange is delegated
to the community [`zitadel`](https://crates.io/crates/zitadel) crate (`zitadel::credentials`).

## JWT Token Mechanism
The reporter loads the Zitadel machine user key file once at startup. For every report the crate

### Token Generation
1. discovers the token endpoint from `{issuer}/.well-known/openid-configuration`,
2. signs a short-lived JWT Profile assertion (RS256) with the private key of the key file,
3. exchanges it at the discovered token endpoint with the
`urn:ietf:params:oauth:grant-type:jwt-bearer` grant,

JWT tokens are generated using the HMAC-SHA256 algorithm with a shared secret key.
and MP only wraps the returned access token into an `Authorization: Bearer <access_token>` header.
No client secret is involved anywhere, and the Status Dashboard verifies the issued token against
the Zitadel issuer.

**Algorithm:** `HS256` (HMAC with SHA-256)
| Component | Responsibility |
|--------------------------|-------------------------------------------------------------------|
| MP (`src/oidc.rs`) | Load the key file once, call the crate per request, build headers |
| `zitadel` crate | OIDC discovery, JWKS fetch, assertion signing, token request |
| Status Dashboard backend | Token verification (`SD_OIDC_*` settings) |

**Token Structure:**
## Machine User Key File

The JWT token contains a simple claim structure:
The reporter authenticates as a Zitadel **machine user** (service user). The key file is downloaded
from the Zitadel Console (instance → *Users* → *Service Users* → select the machine user → *Keys* →
*New* → download the JSON key file):

```json
{
"stackmon": "dummy"
"type": "serviceaccount",
"keyId": "392067695547252958",
"key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----",
"userId": "392040635458125910"
}
```

**Signing Process:**

1. The shared secret is loaded from configuration (`status_dashboard.secret`)
2. An HMAC-SHA256 key is created from the secret bytes
3. Claims are signed with the key to produce the JWT token
4. The token is included in the `Authorization` header as a Bearer token
| Field | Meaning |
|----------|--------------------------------------------------------------------------------------------------------|
| `type` | Must be `serviceaccount`; any other value aborts the reporter at startup with the configuration key named |
| `keyId` | Key id sent as the `kid` header of the assertion, used by Zitadel to select the public key |
| `key` | PEM encoded RSA private key; Zitadel emits PKCS#1 (`BEGIN RSA PRIVATE KEY`), PKCS#8 is accepted too |
| `userId` | Machine user id, used as `iss` and `sub` of the assertion |

Notes:

- The path is configured through `status_dashboard.oidc_key_file`
(`MP_STATUS_DASHBOARD__OIDC_KEY_FILE`). The file is read and validated exactly once at startup, so
it does not have to stay available after the reporter started.
- Zitadel *application* keys (`type: application`, `clientId`) are not accepted: the `zitadel` crate
implements the machine user JWT Profile flow, which is the key kind this deployment uses. Such a
key file aborts startup with an error naming `MP_STATUS_DASHBOARD__OIDC_KEY_FILE`.
- Keep the key file out of the repository and out of configuration files, for example by mounting it
as a secret volume or by writing it to a container file system path.
- The key material, the signed assertion and the access token are never logged, and the private key
never appears in an error message. `OidcIdentity` redacts the loaded credentials in its `Debug`
output.

## Flow

1. The reporter resolves its service identity from the configuration
(`status_dashboard.oidc_issuer`, `oidc_key_file`, `oidc_scopes`). A missing or unusable value
aborts the reporter at startup and the error names the offending configuration key.
2. Before every report the crate signs a new assertion and exchanges it for an access token.
Tokens are never cached in-process, so every report carries a fresh token.
3. The access token is sent as `Authorization: Bearer <access_token>` on every Status Dashboard
call.
4. The Status Dashboard validates the token with `go-oidc`: the issuer must match `SD_OIDC_ISSUER`,
the audience must contain `SD_OIDC_CLIENT_ID`, and the signing key is resolved from the JWKS
endpoint by `kid`.
5. The reporter role is read from the project roles claim
(`urn:zitadel:iam:org:project:roles`, configurable on the backend via `SD_OIDC_ROLES_CLAIM`) and
mapped to the reporter role configured by `SD_RBAC_ROLES_REPORTERS`.

## Assertion

The assertion signed by the crate for the machine user looks like this:

### Configuration

Authentication is configured in the `status_dashboard` section of the configuration file:
```json
{
"alg": "RS256",
"kid": "392067695547252958",
"typ": "JWT"
}
```

```yaml
status_dashboard:
url: https://status-dashboard.example.com
secret: your-shared-secret-key
```json
{
"iss": "392040635458125910",
"sub": "392040635458125910",
"aud": "https://zitadel.example.com",
"iat": 1705929045,
"exp": 1705932645
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | Status dashboard base URL |
| `secret` | string | No | JWT signing secret. If not provided, requests are sent without authentication |
`iss` and `sub` are the `userId` of the machine user, the audience is the issuer URL and `exp` is one
hour after `iat`.

### Environment Variable Override
## Token Request

The secret can also be set via environment variable:
The reporter presents the assertion as the `assertion` parameter of the JWT bearer grant, without
any `Authorization` header and without a client id:

```bash
export MP_STATUS_DASHBOARD__SECRET="your-shared-secret-key"
curl -sS -X POST "https://zitadel.example.com/oauth/v2/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer" \
-d "assertion=<signed assertion>" \
-d "scope=<space joined scopes>"
```

Environment variables are merged with the configuration file, with environment variables taking precedence.
Notes:

## Token Usage
- The token endpoint is discovered per request from `{issuer}/.well-known/openid-configuration`, so
only the issuer is configured.
- The `zitadel` crate always adds the `openid` scope in front of the configured scopes and sends the
whole list space-joined as a single `scope` parameter, in the configured order.

### Authorization Header
### Required scopes

When making requests to the status-dashboard, the JWT token is included in the HTTP `Authorization` header using the Bearer scheme:
The configured scope list has to contain both of these scopes:

```
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdGFja21vbiI6ImR1bW15In0.<signature>
```
| Scope | Why it is needed |
|------------------------------------------------------------|-------------------------------------------------------------------------------------|
| `urn:zitadel:iam:org:project:role:sd_reporters` | Puts the reporter project role into the token, so the status page RBAC check passes |
| `urn:zitadel:iam:org:project:id:<projectId>:aud` | Makes the token audience the project id instead of the client id |

### Request Flow
`<projectId>` is the Zitadel project that both the Status Dashboard and the machine user are part
of. Zitadel puts the client id into the token audience by default, and the Status Dashboard rejects
that with `expected audience ... got [...]`; requesting the audience scope switches the audience to
the project id, so the backend has to be configured with `SD_OIDC_CLIENT_ID=<projectId>`.

1. **Configuration Load:** The reporter reads the `status_dashboard.secret` from configuration
2. **Token Generation:** If a secret is configured, a JWT token is generated at startup
3. **Request Authentication:** All POST requests to `/v1/component_status` include the Bearer token
4. **Server Validation:** The status-dashboard validates the token signature using the same shared secret

## Token Validation

On the server side (status-dashboard), tokens should be validated by:

1. Extracting the token from the `Authorization` header
2. Verifying the HMAC-SHA256 signature using the shared secret
3. Optionally checking the claims (currently contains `{"stackmon": "dummy"}`)
```yaml
status_dashboard:
oidc_scopes:
- "urn:zitadel:iam:org:project:role:sd_reporters"
- "urn:zitadel:iam:org:project:id:392066917738875090:aud"
```

## Security Considerations
`oidc_scopes` has no default, because the audience scope contains the project id of the deployment.
The reporter fails at startup when the list misses either scope and names
`MP_STATUS_DASHBOARD__OIDC_SCOPES` in the error.

### Secret Management
Verified against the pre-production instance with those two scopes: the access token carried
`aud = ["390700708019568682"]` and `groups = ["sd_reporters"]`, with `iss` set to the issuer URL.

- **Never commit secrets** to version control
- Use environment variables (`MP_STATUS_DASHBOARD__SECRET`) in production
- Rotate secrets periodically
- Use strong, randomly-generated secrets (minimum 32 characters recommended)
## Report Authentication

### Transport Security
```http
POST /v2/events HTTP/1.1
Host: status.example.com
Authorization: Bearer <access_token>
Content-Type: application/json

- Always use HTTPS for the status-dashboard URL in production
- The JWT token is sent in clear text in the Authorization header
- Without TLS, tokens could be intercepted and reused
{
"title": "System incident from monitoring system",
"description": "Object Storage Service is degraded",
"impact": 2,
"components": [218],
"start_date": "2024-01-22T10:30:44Z",
"system": true,
"type": "incident"
}
```

### Token Characteristics
## Permission Boundary

- **Stateless:** Tokens are self-contained and don't require server-side session storage
- **No Expiration:** Current implementation does not include expiration claims
- **Single Use Case:** Tokens are specifically for machine-to-machine authentication between reporter and status-dashboard
The Zitadel service identity used by the reporter is restricted to a single operation:

### Best Practices
- Allowed: `POST /v2/events` with `"system": true`
- Denied: component read endpoints and any event without `system: true`

1. **Use Strong Secrets:** Generate cryptographically secure random strings
```bash
openssl rand -base64 32
```
The Status Dashboard backend enforces this boundary and rejects reporter-scoped tokens on every
other route, independently of the roles present in the token.

2. **Environment Separation:** Use different secrets for development, staging, and production
## Backend Verification Points

3. **Audit Logging:** Log authentication failures on the status-dashboard for monitoring
| Check | Backend setting | Expectation |
|---------------|--------------------------|------------------------------------------------------------|
| Issuer | `SD_OIDC_ISSUER` | Matches `status_dashboard.oidc_issuer` |
| Audience | `SD_OIDC_CLIENT_ID` | The project id, requested through the audience scope |
| Signing key | JWKS by `kid` | Resolved from the issuer |
| Roles claim | `SD_OIDC_ROLES_CLAIM` | `urn:zitadel:iam:org:project:roles` |
| Reporter role | `SD_RBAC_ROLES_REPORTERS` | Role key requested through `oidc_scopes` |

4. **Network Isolation:** Where possible, restrict network access between components
The reporter does not send an audience of its own: Zitadel decides the audience of the access token,
so the backend has to be configured with the audience the requested scopes produce:

## Example Implementation
- Without the audience scope the token audience is the client id of the machine user, which the
backend rejects with `expected audience ... got [...]`.
- With `urn:zitadel:iam:org:project:id:<projectId>:aud` the audience is the project id, which is
what `SD_OIDC_CLIENT_ID` has to be set to.

### Generating a Token (Rust)
## Failure Handling

```rust
use hmac::{Hmac, Mac};
use jwt::SignWithKey;
use sha2::Sha256;
use std::collections::BTreeMap;
The reporter is fail-closed:

let secret = "your-shared-secret";
let key: Hmac<Sha256> = Hmac::new_from_slice(secret.as_bytes()).unwrap();
- A missing `oidc_issuer`, `oidc_key_file` or `oidc_scopes`, an unreadable or invalid key file, a key
type other than `serviceaccount`, an empty `keyId`/`key`/`userId` and a scope list without a role
scope or without the project audience scope abort the reporter at startup, and the error names the
configuration key that has to be fixed.
- A failing discovery or token request, and a token response without a usable `access_token`, abort
the report instead of sending it without authentication.
- The private key, the signed assertion and the access token are never part of an error message or
of the reporter output.

let mut claims = BTreeMap::new();
claims.insert("stackmon", "dummy");
## Configuration

let token_str = claims.sign_with_key(&key).unwrap();
let bearer = format!("Bearer {}", token_str);
```yaml
status_dashboard:
url: "https://status.example.com"
oidc_issuer: "https://zitadel.example.com"
oidc_key_file: "/etc/cloudmon/service-identity.json"
oidc_scopes:
# Reports the reporter role and makes the token audience the project id
- "urn:zitadel:iam:org:project:role:sd_reporters"
- "urn:zitadel:iam:org:project:id:392066917738875090:aud"
```

### Validating a Token (Pseudocode)
Environment variable equivalents:

```
function validate_token(authorization_header, secret):
# Extract token from "Bearer <token>"
token = extract_bearer_token(authorization_header)

# Verify signature
key = hmac_sha256_key(secret)
claims = verify_and_decode(token, key)

if claims is valid:
return true
else:
return false
```
| Environment Variable | Configuration path |
|--------------------------------------|----------------------------------|
| `MP_STATUS_DASHBOARD__URL` | `status_dashboard.url` |
| `MP_STATUS_DASHBOARD__OIDC_ISSUER` | `status_dashboard.oidc_issuer` |
| `MP_STATUS_DASHBOARD__OIDC_KEY_FILE` | `status_dashboard.oidc_key_file` |
| `MP_STATUS_DASHBOARD__OIDC_SCOPES` | `status_dashboard.oidc_scopes` |
6 changes: 3 additions & 3 deletions doc/architecture/data-flow.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ flowchart TD
Parse["Parse Response"]
Check{"Last value > 0?"}
Skip["Skip notification"]
Build["Build ComponentStatus"]
Build["Build IncidentData"]
Post["POST to Dashboard"]

Poll --> Parse
Expand All @@ -340,7 +340,7 @@ if let Some(last) = data.metrics.pop() {

```http
POST /v1/component_status
Authorization: Bearer <jwt_token>
Authorization: Bearer <access_token>
Content-Type: application/json

{
Expand Down Expand Up @@ -402,7 +402,7 @@ sequenceDiagram
API-->>Reporter: ServiceHealthResponse

alt Health status > 0
Reporter->>Reporter: Build ComponentStatus
Reporter->>Reporter: Build IncidentData
Reporter->>Dashboard: POST /v1/component_status
Dashboard-->>Reporter: 200 OK
end
Expand Down
Loading
Loading