Frontmatter

Source file: src/frontmatter/parser.ts

Frontmatter parsing allows individual test files to override global configuration. A spec file can specify its own LLM model key, timeout, tags for filtering, or permission-bypass setting — all via standard YAML frontmatter at the top of the file. This means a single test suite can mix different LLMs (e.g. use a faster model for simple checks and a more capable one for complex evaluations) without separate config files or CLI flags.

The parser uses gray-matter to extract the YAML block, then validates each field individually with Zod. The design is deliberately lenient — if one field is invalid (e.g. a misspelled model key), the other valid fields are still preserved rather than discarding all frontmatter.

ParsedSpec type

interface ParsedSpec {
  data: FrontmatterData;
  content: string;  // File content with frontmatter stripped
}

FrontmatterData type

interface FrontmatterData {
  tags?: string[];
  timeout?: number;
  llm?: ModelKey;
  skipPermissionsIfPossible?: boolean;
}

Supported fields

Field	Type	Description
tags	string[] or CSV string	Tags for --tag filtering. Accepts both YAML arrays and comma-separated strings
timeout	number	Per-test timeout in milliseconds, overrides config timeout
llm	ModelKey	Model key override, must be a valid registered key (see semtest list)
skipPermissionsIfPossible	boolean	Override the global permission-bypass setting for this test

CSV tag coercion

Tags can be written in two formats:

# YAML array
tags:
  - api
  - auth

# Comma-separated string (coerced to array)
tags: api, auth

Both produce ["api", "auth"]. The coercion is handled by a Zod preprocess step that splits comma-separated strings into arrays before validation.

Lenient validation

When frontmatter validation fails as a whole, the parser falls back to per-field validation:

1. Parse the full frontmatter object with frontmatterSchema.safeParse()
2. If it succeeds, return all fields
3. If it fails, validate each known field individually
4. Preserve valid fields, discard invalid ones
5. Print a warning listing the invalid fields

For invalid llm fields, the warning includes a hint: "Run \semtest list` to see available models”`.

Warning: invalid frontmatter fields (llm) in auth.spec.md, using global config defaults. Run `semtest list` to see available models

Unknown fields (not in the schema) are silently passed through via .passthrough() — they don’t cause warnings and don’t affect behaviour.

Integration with discovery

Frontmatter parsing happens during test discovery. When discoverTests() or resolveTests() reads a test file, it calls parseFrontmatter() to:

1. Extract YAML frontmatter and strip it from the content
2. Validate and return FrontmatterData
3. The stripped content is what gets sent to the LLM (frontmatter is not included in the prompt)

The resulting SemanticTest object carries both frontmatter (the parsed data) and tags (a convenience accessor for frontmatter.tags ?? []).

Example spec file with frontmatter

---
tags: api, critical
timeout: 120000
llm: gemini-2.5-pro
skipPermissionsIfPossible: true
---

# API Route Authentication

## Expectation

All API routes under `/api/protected/` should require a valid JWT token...