Producer context
Microsoft.Testing.Platform (MTP) produces CTRF for multiple test frameworks, so it cannot assume every framework assigns the same stability semantics to its native test UID.
Today the MTP CTRF reporter keeps the framework-provided UID under tests[].extra.uid rather than claiming it is a CTRF testId:
For MSTest specifically, that UID is a versioned hash of the assembly file name, fully-qualified test name, parameter types, and (for data-driven tests) the row index:
This is stable for many rebuilds, but a rename, assembly rename, or reordered dynamic data source can change it.
What the current spec resolves
The identity model is clear that:
testId identifies a logical test case and should be stable across runs within the producer's documented scope;executionId is unique for one execution of that logical test within a run;attemptId identifies an individual entry in retryAttempts;- identity fields are optional and opaque to consumers.
PR #57 and PR #62 provide useful general guidance, but we could not find guidance for parameterized cases or for reports transformed by retry merging.
Questions
- For a parameterized/data-driven method, is each argument row a distinct logical test case that should have its own
testId, or should all rows share the method's testId and be distinguished by parameters plus executionId?
- If rows are distinct test cases, should a generated
testId be based on canonical parameter values (stable across data-source reordering) rather than a row index? How should producers handle values that cannot be serialized deterministically or safely?
- Is a source rename/file move intentionally a new
testId, or is preserving identity across renames an expected producer responsibility?
- For retries executed as separate processes and later collapsed into one CTRF test object, should the raw per-process documents and the merged test share one
executionId, or should each raw document have its own execution identity and the merged document get another one?
- Is
attemptId intentionally limited to previous attempts in retryAttempts? The final attempt is represented by the test object, which has no attemptId; that makes it difficult to correlate the final attempt between a raw per-process report and a merged report.
Why this matters for MTP
Mapping an arbitrary framework UID to testId may overstate its stability, while omitting standardized identities prevents CTRF consumers from correlating parameterized rows and retry artifacts. We would like to document and implement the intended model rather than establish a .NET-specific convention.
Possible interpretations include treating every parameter row as a logical test case, or treating the method as the logical test and the row as an execution. We are not prescribing either; a normative recommendation and one worked parameterized/retry example would unblock the mapping.
Producer context
Microsoft.Testing.Platform (MTP) produces CTRF for multiple test frameworks, so it cannot assume every framework assigns the same stability semantics to its native test UID.
Today the MTP CTRF reporter keeps the framework-provided UID under
tests[].extra.uidrather than claiming it is a CTRFtestId:For MSTest specifically, that UID is a versioned hash of the assembly file name, fully-qualified test name, parameter types, and (for data-driven tests) the row index:
This is stable for many rebuilds, but a rename, assembly rename, or reordered dynamic data source can change it.
What the current spec resolves
The identity model is clear that:
testIdidentifies a logical test case and should be stable across runs within the producer's documented scope;executionIdis unique for one execution of that logical test within a run;attemptIdidentifies an individual entry inretryAttempts;PR #57 and PR #62 provide useful general guidance, but we could not find guidance for parameterized cases or for reports transformed by retry merging.
Questions
testId, or should all rows share the method'stestIdand be distinguished byparametersplusexecutionId?testIdbe based on canonical parameter values (stable across data-source reordering) rather than a row index? How should producers handle values that cannot be serialized deterministically or safely?testId, or is preserving identity across renames an expected producer responsibility?executionId, or should each raw document have its own execution identity and the merged document get another one?attemptIdintentionally limited to previous attempts inretryAttempts? The final attempt is represented by the test object, which has noattemptId; that makes it difficult to correlate the final attempt between a raw per-process report and a merged report.Why this matters for MTP
Mapping an arbitrary framework UID to
testIdmay overstate its stability, while omitting standardized identities prevents CTRF consumers from correlating parameterized rows and retry artifacts. We would like to document and implement the intended model rather than establish a .NET-specific convention.Possible interpretations include treating every parameter row as a logical test case, or treating the method as the logical test and the row as an execution. We are not prescribing either; a normative recommendation and one worked parameterized/retry example would unblock the mapping.