Test Case & Test Plan Schema (Canonical)

This is the source of truth for all testcases and testplans in the repository. Everything is stored as individual YAML files so they can be versioned with git.

Directory Structure

testcases/<project>/<module>/<group>/<testcase-id>.yaml
testplans/<project>/<plan-id>.yaml

Current projects: pbx, core (add new projects at the same level).

Rules

Every id must be globally unique across the entire system.
Testplans reference testcases by their id (never embed content).
Each step inside a testcase can be marked individually as automatable or not.
custom_fields is a free-form object for future extensibility.
Generated artifacts (from testplan export) live only under tests/generated/<plan-id>/ and are never considered source of truth.
Manual export of a testplan includes only the steps where automatable: false.

testcase.yaml (example)

id: pbx-extensions-check-create_modify_delete_sip_extensions
project: pbx
module: pbx
group: extensions
title: Create, modify and delete SIP extensions
description: Verify full CRUD lifecycle of SIP extensions via admin portal
priority: High
tags: [sip, provisioning]
references: []
dependencies: []                    # other global testcase ids that must run before this one

preconditions: []
postconditions: []

steps:
  - id: step-01
    action: Open Extensions page in admin portal
    expected: List of extensions is visible and searchable
    automatable: true
  - id: step-02
    action: Create new SIP extension with random number and register on physical phone
    expected: Extension appears in list and can register
    automatable: false
    reason_not_automatable: Requires physical phone hardware registration and audio validation
  - id: step-03
    action: Modify extension settings and verify changes
    expected: Changes are persisted and reflected in the system
    automatable: true

automation_status: partial          # automated | partial | manual (can be derived from steps)
file: tests/pbx/extensions.spec.ts  # TEMPORARY during transition only
custom_fields: {}

testplan.yaml (example)

id: pbx-regression-2026-05
name: PBX Regression Test Plan - May 2026
project: pbx
description: Full regression after v1.2.3
environment:
  base_url: https://pbx.local:4443
  tenant: lab-qa

testcases:
  - ref: pbx-extensions-check-create_modify_delete_sip_extensions
    severity: High
  - ref: core-repository-check-presence_of_release_notes_and_changelog
    severity: Medium
  - ref: pbx-calls-check-internal_extension_to_extension_audio_call
    severity: Critical
    must_run_after:
      - pbx-extensions-check-create_modify_delete_sip_extensions

execution_notes: |
  Run in isolated lab environment. Have physical phones ready for media tests.

custom_fields: {}

Export Behavior (when implemented)

From a composed testplan:

Manual deliverable: HTML containing only steps where automatable: false
Automated deliverable: Playwright .spec.ts placed in tests/generated/<plan-id>/ (usable directly by Playwright + MCP agent)

Transition Notes

Until migration is complete, the old editor/data/testcases.json and existing tests/*.spec.ts remain the temporary source of truth.
Once the YAML tree is populated and the editor reads from it, the YAML files become the single source of truth.
New testcases must be created as YAML files following this schema.

Versioning

All changes to these YAML files are tracked with normal git. No application-level versioning is required.

3.5 KiB Raw Blame History