Test Case & Test Plan Schema (Canonical)

This is the source of truth for all testcases and testplans in the repository. Everything is stored as individual YAML files so they can be versioned with git.

Directory Structure

testcases/<module>/<group>/<testcase-id>.yaml

Top level under testcases/ is the module (device/product type): core/, pbx/, gateway/, etc.
core/ = generic, cross-device concerns (documentation, security posture, installation, virtualization platform, etc.).
pbx/ = PBX-specific telephony concerns (extensions, calls, codecs, etc.).
Add new device types (e.g. gateway/, sbc/) as siblings of core/ and pbx/.

The old <project> level has been removed.

Rules

Every id must be globally unique across the entire system.
Testplans reference testcases by their id (never embed content).
The automation decision (automation_status) is made at testcase level, not per step.
Steps are intentionally simple (action + expected + optional notes).
automatable and notYet lists at testcase level are high-level summaries for reporting.
custom_fields is a free-form object for future extensibility.
Generated artifacts (from testplan export) live only under tests/generated/<plan-id>/ and are never considered source of truth.

testcase.yaml (example)

id: pbx-extensions-check-create_modify_delete_sip_extensions
module: pbx
group: extensions
title: Create, modify and delete SIP extensions
description: Verify full CRUD lifecycle of SIP extensions via admin portal
priority: High
tags: [sip, provisioning]
references: []
dependencies: []                    # other global testcase ids that must run before this one

preconditions: []
postconditions: []

steps:
  - action: Open Extensions page in admin portal
    expected: List of extensions is visible and searchable
  - action: Create new SIP extension with random number and register on physical phone
    expected: Extension appears in list and can register
    notes: Requires physical phone hardware registration and audio validation
  - action: Modify extension settings and verify changes
    expected: Changes are persisted and reflected in the system

automation_status: partial          # automated | partial | manual
reason_not_automatable: Requires physical phone for registration/audio validation (step 2)

automatable:
  - Portal UI steps for creating and editing extensions
notYet:
  - Physical phone registration and real audio validation

file: tests/pbx/extensions.spec.ts  # TEMPORARY during transition only
custom_fields: {}

testplan.yaml (example)

id: pbx-regression-2026-05
name: PBX Regression Test Plan - May 2026
description: Full regression after v1.2.3
environment:
  base_url: https://pbx.local:4443
  tenant: lab-qa

testcases:
  - ref: pbx-extensions-check-create_modify_delete_sip_extensions
    severity: High
  - ref: core-repository-check-presence_of_release_notes_and_changelog
    severity: Medium
  - ref: pbx-calls-check-internal_extension_to_extension_audio_call
    severity: Critical
    must_run_after:
      - pbx-extensions-check-create_modify_delete_sip_extensions

execution_notes: |
  Run in isolated lab environment. Have physical phones ready for media tests.

automation_summary:
  total: 3
  automated: 1
  partial: 1
  manual: 1
  automatable_percentage: 66.7

custom_fields: {}

Export Behavior (when implemented)

From a composed testplan:

Manual deliverable: HTML containing testcases where automation_status is manual or partial
Automated deliverable: Playwright .spec.ts placed in tests/generated/<plan-id>/ (usable directly by Playwright + MCP agent)

Automation Model (Simplified)

automation_status is the single source of truth for whether a testcase is automated.
Steps are kept simple on purpose (action + expected + optional free-text notes).
automatable / notYet lists at testcase level are summaries for reporting and planning.
Per-step automation flags were removed (decision moved to testcase level).

Transition Notes

Until migration is complete, the old editor/data/testcases.json and existing tests/*.spec.ts remain the temporary source of truth.
Once the YAML tree is populated and the editor reads from it, the YAML files become the single source of truth.
New testcases must be created as YAML files following this schema.

Versioning

All changes to these YAML files are tracked with normal git. No application-level versioning is required.

4.5 KiB Raw Blame History