Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 33 additions & 3 deletions python/docs/guides/pytest_plugin/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,11 +131,13 @@ Each kind has a home chosen for a specific workflow:

- **Pytest behavior** lives in `[tool.pytest.ini_options]` (log/offline/disabled/git/`*_step`/autouse/parametrize). A CLI flag exists for the ones with a real ad-hoc override workflow.
- **Connection** comes from the environment first, falling back to the ini keys; the API key is env-only so secrets stay out of committed files.
- **Report content** takes static defaults from `[tool.sift.pytest.report]` and per-run dynamic values from `SIFT_REPORT_*` env vars (CI builds, hardware cycling, anything `.env`-driven; pytest-dotenv loads `.env` for local dev).
- **Report content** takes static defaults from `[tool.sift.pytest.report]` and per-run dynamic values from `SIFT_REPORT_*` env vars (CI builds, hardware cycling, anything `.env`-driven; pytest-dotenv loads `.env` for local dev). `archive_on_create` also accepts a CLI flag and an ini key.

Precedence within a setting runs env > CLI flag > ini key > TOML > built-in
default. No setting exposes both env and CLI, so the chain isn't ambiguous in
practice.
default. For a boolean, an explicit `false` is a value: it overrides a `true`
from a lower-precedence source. `archive_on_create` uses every surface, so a
shared `pyproject.toml` can archive dev runs while production sets
`SIFT_REPORT_ARCHIVE_ON_CREATE=false`.

The plugin scans `SIFT_*` env vars and `[tool.sift.pytest.*]` keys at session
start; anything outside these tables fires a warning with a closest-match
Expand Down Expand Up @@ -179,10 +181,15 @@ suggestion, so typos like `SIFT_REPORT_SERIALNUM` surface immediately.
| Operator running the test. Defaults to the OS user. | `[tool.sift.pytest.report] system_operator` | `SIFT_REPORT_SYSTEM_OPERATOR` |
| Serial number of the unit under test. | `[tool.sift.pytest.report] serial_number` | `SIFT_REPORT_SERIAL_NUMBER` |
| Part number of the unit under test. | `[tool.sift.pytest.report] part_number` | `SIFT_REPORT_PART_NUMBER` |
| Archive the report right after creating it, so it drops out of the default Test Results views. An explicit false overrides a true from a lower-precedence source. | `[tool.sift.pytest.report] archive_on_create` | `SIFT_REPORT_ARCHIVE_ON_CREATE` |
| Free-form report metadata, as a TOML table of scalar values. For dynamic per-run keys, override the sift_report_metadata fixture in conftest. | `[tool.sift.pytest.report.metadata]` (table) | — |

<!-- END settings-reference -->

`archive_on_create` can also be set with `--sift-archive-on-create` or with
`sift_archive_on_create` in `[tool.pytest.ini_options]`. The table omits those
columns because the other report settings do not use them.

### Quick-start examples

```toml title="pyproject.toml"
Expand Down Expand Up @@ -243,6 +250,29 @@ SIFT_REPORT_SYSTEM_OPERATOR=$CI_ACTOR \
pytest tests/
```

### Archiving a run at creation

`archive_on_create` archives the report immediately after the plugin creates
it. Archived reports drop out of the default Test Results views. The plugin
creates the report, then archives it in a second call. If that call fails, the
plugin logs a warning and the test session continues with the report
unarchived.

```toml title="pyproject.toml"
[tool.sift.pytest.report]
archive_on_create = true
```

You can also pass `--sift-archive-on-create`, set `sift_archive_on_create`
under `[tool.pytest.ini_options]`, or set `SIFT_REPORT_ARCHIVE_ON_CREATE`.
Precedence is the environment variable, then the CLI flag, then the ini key,
then TOML. An explicit `false` overrides a `true` from a lower source, so
production can set `SIFT_REPORT_ARCHIVE_ON_CREATE=false` while the shared TOML
stays `true`.

The terminal summary prints `· archived` on the status row. Replay of that log
archives the uploaded report.

### `name` vs `test_case`

The two fields look similar but serve opposite purposes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1338,6 +1338,11 @@ def record_created(simulated_id: str, real_id: str) -> None:
real_report = await self._create_report_from_simulated(state.report)
real_report_id = real_report._id_or_error
record_created(state.report._id_or_error, real_report_id)
# Create has no is_archived field, so the collapsed flag goes out as an update.
if state.report.is_archived:
archive_update = TestReportUpdate(is_archived=True)
archive_update.resource_id = real_report_id
real_report = await self.update_test_report(archive_update, existing=real_report)

real_steps: list[TestStep] = []
for sim_step_id in state.steps_order:
Expand Down
80 changes: 77 additions & 3 deletions python/lib/sift_client/_internal/pytest_plugin/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ class Option:
- ``toml``: tuple path under ``[tool.sift...]``, e.g.
``("pytest", "report", "name")`` -> ``tool.sift.pytest.report.name``.
- ``env``: full env var name, e.g. ``"SIFT_API_KEY"``.
- ``value_type``: how to read env and TOML values. ``"bool"`` accepts
true/false words. Ini values use ``ini_type`` instead.
- ``surfaces``: the precedence order. The default puts env before cli.
Override it if a flag that the user types must outrank an environment
variable, as ``profile`` does.
Expand All @@ -93,6 +95,7 @@ class Option:
toml: tuple[str, ...] | None = None
env: str | None = None
merge: bool = False
value_type: str | None = None
surfaces: tuple[str, ...] = ("env", "cli", "ini", "toml")

@property
Expand All @@ -117,6 +120,8 @@ def __post_init__(self) -> None:
raise ValueError(f"Option({self.name!r}): ini_type requires ini")
if self.merge and not self.toml:
raise ValueError(f"Option({self.name!r}): merge=True needs toml")
if self.value_type not in (None, "bool"):
raise ValueError(f"Option({self.name!r}): value_type must be None or 'bool'")
if not any([self.cli, self.ini, self.toml, self.env]):
raise ValueError(f"Option({self.name!r}): declares no surfaces")
if self.category not in CATEGORIES:
Expand All @@ -132,8 +137,8 @@ def resolve(self, config: pytest.Config | None) -> Any:

The walk order is :attr:`surfaces`, which puts env before cli by default.
``getini`` returns the typed default for unset bool/list keys, so this
only returns ini values for booleans (always meaningful), non-empty
strings, and non-empty lists.
returns ini values for booleans, non-empty strings, and non-empty lists.
A ``None`` ini default counts as unset, so a lower value can still apply.
"""
return self.resolve_with_source(config)[0]

Expand All @@ -157,7 +162,11 @@ def _read_surface(self, surface: str, config: pytest.Config | None) -> Any:
if not self.env:
return None
env_value = os.getenv(self.env)
return env_value if env_value else None
if not env_value:
return None
if self.value_type == "bool":
return _coerce_bool(env_value, source=self.env)
return env_value
if config is None:
return None
if surface == "cli":
Expand All @@ -177,6 +186,9 @@ def _read_surface(self, surface: str, config: pytest.Config | None) -> Any:
if not self.toml:
return None
toml_value = _walk_toml(tool_sift(config), self.toml)
if self.value_type == "bool":
source = "tool.sift." + ".".join(self.toml)
return _coerce_bool(toml_value, source=source)
return toml_value if toml_value not in (None, "") else None

def resolve_merged(self, config: pytest.Config | None) -> dict[str, str | float | bool]:
Expand Down Expand Up @@ -208,6 +220,51 @@ def resolve_merged(self, config: pytest.Config | None) -> dict[str, str | float
return result


_BOOL_TRUE = frozenset({"1", "true", "t", "yes", "y", "on"})
_BOOL_FALSE = frozenset({"0", "false", "f", "no", "n", "off"})


def _parse_bool_token(raw: str) -> bool | None:
"""Parse a boolean token. ``None`` when ``raw`` is not a boolean word."""
token = raw.strip().lower()
if token in _BOOL_TRUE:
return True
if token in _BOOL_FALSE:
return False
return None


def _coerce_bool(value: Any, *, source: str) -> bool | None:
"""Coerce one surface's value to bool. Unset stays ``None``.

A string ``false`` is ``False``, not a truthy string. Anything that is not
a bool or a boolean word warns and counts as unset.
"""
if isinstance(value, bool):
return value
if value is None or value == "":
return None
if isinstance(value, str):
parsed = _parse_bool_token(value)
if parsed is not None:
return parsed
from sift_client.pytest_plugin import SiftPytestPluginWarning

log_event(
logger,
logging.WARNING,
"config.bool",
name=source,
value=repr(value),
)
warnings.warn(
f"Ignoring {source}={value!r}: expected true or false.",
SiftPytestPluginWarning,
stacklevel=2,
)
return None


def _walk_toml(data: dict[str, Any], path: tuple[str, ...]) -> Any:
"""Walk a parsed TOML tree along ``path``; return None on any missing key."""
cur: Any = data
Expand Down Expand Up @@ -445,6 +502,22 @@ def _walk_toml(data: dict[str, Any], path: tuple[str, ...]) -> Any:
env="SIFT_REPORT_PART_NUMBER",
toml=("pytest", "report", "part_number"),
)
# None, not False: an unset ini key must not hide a TOML true.
ARCHIVE_ON_CREATE_OPTION = Option(
name="archive_on_create",
category=CAT_REPORT,
help="Archive the report right after creating it, so it drops out of the "
"default Test Results views. An explicit false overrides a true from a "
"lower-precedence source.",
cli="--sift-archive-on-create",
cli_action="store_true",
ini="sift_archive_on_create",
ini_type="bool",
ini_default=None,
env="SIFT_REPORT_ARCHIVE_ON_CREATE",
toml=("pytest", "report", "archive_on_create"),
value_type="bool",
)
METADATA_OPTION = Option(
name="metadata",
category=CAT_REPORT,
Expand Down Expand Up @@ -478,6 +551,7 @@ def _walk_toml(data: dict[str, Any], path: tuple[str, ...]) -> Any:
SYSTEM_OPERATOR_OPTION,
SERIAL_NUMBER_OPTION,
PART_NUMBER_OPTION,
ARCHIVE_ON_CREATE_OPTION,
METADATA_OPTION,
)

Expand Down
3 changes: 3 additions & 0 deletions python/lib/sift_client/_internal/pytest_plugin/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from sift_client._internal.pytest_plugin.audit_log import log_event
from sift_client._internal.pytest_plugin.modes import is_offline
from sift_client._internal.pytest_plugin.options import (
ARCHIVE_ON_CREATE_OPTION,
GIT_METADATA_OPTION,
LOG_FILE_OPTION,
METADATA_OPTION,
Expand Down Expand Up @@ -474,6 +475,7 @@ def report_context_impl(
replay_log_file=not (disabled or offline),
metadata=report_metadata,
audit_log=audit_log,
archive_on_create=bool(ARCHIVE_ON_CREATE_OPTION.resolve(pytestconfig)),
# pytest tears this session-scoped fixture down during the LAST item's
# teardown phase but reports that phase's outcome afterwards, so a
# teardown failure on the final test is unknown here. The plugin's
Expand All @@ -494,6 +496,7 @@ def report_context_impl(
serial=report.serial_number or "-",
part=report.part_number or "-",
metadata=meta_kv,
archived=report.is_archived,
)
# What actually happens with the JSONL log, not the raw setting: the
# effective path (temp or pinned), or "disabled", plus whether the
Expand Down
9 changes: 5 additions & 4 deletions python/lib/sift_client/_internal/pytest_plugin/terminal.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,14 +162,15 @@ def write_report_summary(
status_word, status_markup = "PASSED", {"green": True, "bold": True}
# Offline results live only in the local log until replayed, so the status
# row calls that out instead of repeating the version (already in the header).
# Archived rides on this row too. The upload lines below are copied as commands.
report = context.report
archived = " · archived" if getattr(report, "is_archived", False) else ""
status_context = (
f"{mode_label(config)} · not uploaded"
f"{mode_label(config)} · not uploaded{archived}"
if offline
else f"{mode_label(config)} · sift-stack-py {sdk_version()}"
else f"{mode_label(config)} · sift-stack-py {sdk_version()}{archived}"
)

report = context.report

terminalreporter.write_sep(
"=", report_panel_title(report, terminalreporter), cyan=True, bold=True
)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
"""Replay keeps ``is_archived`` from a log written by ``archive_on_create``.

Batch replay collapses the log into one create. ``CreateTestReportRequest``
has no archive field, so the collapsed flag has to go out as a follow-up
update. Incremental replay sends the logged update as its own line.
"""

from __future__ import annotations

from datetime import datetime, timezone
from unittest.mock import AsyncMock, MagicMock

import pytest

from sift_client._internal.low_level_wrappers.test_results import (
TestResultsLowLevelClient as ResultsLowLevelClient,
)
from sift_client.sift_types.test_report import (
TestReport,
TestReportCreate,
TestReportUpdate,
TestStatus,
)

T0 = datetime(2026, 1, 1, tzinfo=timezone.utc)


def _make_report(id_: str, *, is_archived: bool = False) -> TestReport:
return TestReport(
id_=id_,
status=TestStatus.PASSED,
name="n",
test_system_name="s",
test_case="c",
start_time=T0,
end_time=T0,
metadata={},
is_archived=is_archived,
)


def _report_create() -> TestReportCreate:
return TestReportCreate(
status=TestStatus.IN_PROGRESS,
name="n",
test_system_name="s",
test_case="c",
start_time=T0,
end_time=T0,
)


def _install_create_spy(client: ResultsLowLevelClient) -> None:
"""Real creates return a stand-in. Simulate and log writes stay on the client."""
original = client.create_test_report

async def create_spy(*args, **kwargs):
if kwargs.get("simulate") or kwargs.get("log_file") is not None:
return await original(*args, **kwargs)
return _make_report("real-report")

client.create_test_report = create_spy # type: ignore[method-assign]


async def _write_archived_log(log_file, client: ResultsLowLevelClient) -> None:
report = await client.create_test_report(test_report=_report_create(), log_file=log_file)
update = TestReportUpdate(is_archived=True)
update.resource_id = report.id_
await client.update_test_report(update=update, log_file=log_file)


@pytest.mark.asyncio
async def test_batch_replay_archives_after_create(tmp_path):
"""The default upload path archives the report it just created."""
log_file = tmp_path / "archived.jsonl"
client = ResultsLowLevelClient(grpc_client=MagicMock())
await _write_archived_log(log_file, client)

original_update = client.update_test_report
real_updates: list[TestReportUpdate] = []
_install_create_spy(client)

async def update_spy(*args, **kwargs):
if kwargs.get("simulate") or kwargs.get("log_file") is not None:
return await original_update(*args, **kwargs)
real_updates.append(args[0])
return _make_report("real-report", is_archived=True)

client.update_test_report = update_spy # type: ignore[method-assign]

result = await client.import_log_file(log_file)

assert len(real_updates) == 1
assert real_updates[0].is_archived is True
assert real_updates[0].resource_id == "real-report"
assert result.report is not None
assert result.report.is_archived is True


@pytest.mark.asyncio
async def test_batch_replay_skips_archive_when_unset(tmp_path):
"""A log with no archive update does not send one."""
log_file = tmp_path / "plain.jsonl"
client = ResultsLowLevelClient(grpc_client=MagicMock())
await client.create_test_report(test_report=_report_create(), log_file=log_file)

client.update_test_report = AsyncMock() # type: ignore[method-assign]
_install_create_spy(client)

result = await client.import_log_file(log_file)

client.update_test_report.assert_not_called()
assert result.report is not None
assert result.report.is_archived is False


@pytest.mark.asyncio
async def test_incremental_replay_sends_archive_update(tmp_path):
"""Line-by-line replay forwards the logged archive update."""
log_file = tmp_path / "incremental.jsonl"
client = ResultsLowLevelClient(grpc_client=MagicMock())
await _write_archived_log(log_file, client)

archived = _make_report("real-report", is_archived=True)
client.create_test_report = AsyncMock(return_value=_make_report("real-report")) # type: ignore[method-assign]
client.update_test_report = AsyncMock(return_value=archived) # type: ignore[method-assign]

await client.import_log_file(log_file, incremental=True)

sent = client.update_test_report.await_args.kwargs["request"]
assert "is_archived" in sent.update_mask.paths
assert sent.test_report.is_archived is True
Loading
Loading