Reports

Source files: src/report/markdown.ts, src/report/json.ts, src/report/junit.ts

The report module produces persistent output after tests complete. Three formats serve different audiences. The Markdown report (latest.md) is for humans — it shows what failed, the expected versus observed behaviour, the file location, and a suggested resolution. The JSON report (ci-results.json) is for CI/CD pipelines — a machine-readable pass/fail signal with just enough data to identify failures and their locations. The JUnit XML report (junit-results.xml) is for integration with test frameworks and CI platforms that consume the standard JUnit format.

The latest.md file is overwritten on each run, giving a stable path that bookmarks and scripts can rely on. With the --timestamp flag, timestamped copies are also generated for historical tracking without losing the convenience of the stable path.

Markdown report

Generated by generateMarkdownReport(). Written to {output}/latest.md (and optionally a timestamped copy).

Structure

# Semantic Test Report

Run Timestamp: 2025-01-15T10:30:00.000Z
Project: semtest

## Summary

| Metric | Count |
|---|---|
| Total Tests | 12 |
| Passed | 10 |
| Failed | 2 |

---

## Test Results

### FAIL auth-check

Source: auth-middleware.spec.md
Status: FAIL

Expectation: ...
Observed: ...
Location: ...
Resolution: ...

---

## Validation Issues

- **duplicate-id**: ...

---

Sections

Section	Always present	Content
Header	Yes	Timestamp and project name
Summary table	Yes	Counts for total, passed, failed (+ errored, skipped, invalid if > 0)
Test Results	Yes	One subsection per test, grouped by directory. Passing tests excluded unless --include-passing
Validation Issues	Only if issues exist	Bulleted list of validation issues

Options

Option	Effect
includePassing	Passing tests appear in the Test Results section
validation	If provided and has issues, a Validation Issues section is appended

JSON report

Generated by generateJSONReport(). Written to {output}/ci-results.json.

CIResult shape

interface CIResult {
  status: "pass" | "fail" | "error";
  summary: {
    total: number;
    passed: number;
    failed: number;
    errored?: number;    // omitted if 0
    invalid?: number;    // omitted if 0
    skipped?: number;    // omitted if 0
  };
  tests: CITestEntry[];
  validation?: ValidationResult;  // omitted if not provided
}

CITestEntry shape

interface CITestEntry {
  id: string;
  sourceFile: string;
  status: TestStatus;
  group?: string;      // directory-based grouping
  location?: string;   // fail only
  error?: string;      // error only
}

The JSON report is intentionally minimal — it includes only the fields CI systems need to determine pass/fail and locate problems. Full diagnostic details are in the Markdown report. The group field is included when a test belongs to a subdirectory within the test root, enabling CI tools to organise results by group.

JUnit XML report

Generated by generateJUnitReport(). Written to {output}/junit-results.xml when --junit is passed or junit: true is set in config.

Structure

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="12" failures="2" errors="0" skipped="1" timestamp="...">
  <testsuite name="semantic-tests" tests="8" failures="1" errors="0" skipped="0" timestamp="...">
    <testcase name="auth-check" classname="auth-middleware.spec.md" />
    <testcase name="missing-validation" classname="config-schema.spec.md">
      <failure message="Config should validate runner names">...</failure>
    </testcase>
  </testsuite>
  <testsuite name="api" tests="4" failures="1" errors="0" skipped="1" timestamp="...">
    <testcase name="rate-limiting" classname="api-routes.spec.md">
      <skipped />
    </testcase>
  </testsuite>
</testsuites>

Mapping to JUnit elements

Status	JUnit element
pass	<testcase /> (self-closing, no child elements)
fail	<testcase> with <failure> child containing expectation, observed, location, resolution
error	<testcase> with <error> child containing the error message
skip / invalid	<testcase> with <skipped /> child

Tests are grouped into <testsuite> elements by their directory group. Tests without a group are placed in a suite named "semantic-tests". All XML content is properly escaped (control characters stripped, &, <, >, ", ' encoded).

Output paths

File	Path	When
Latest report	{output}/latest.md	Always
CI results	{output}/ci-results.json	Always
JUnit results	{output}/junit-results.xml	Only with --junit
Timestamped report	{output}/{timestamp}.md	Only with --timestamp
Debug logs	{output}/debug/{testname}.json	Only with --debug