121 lines
4.5 KiB
Markdown
121 lines
4.5 KiB
Markdown
# Test Case & Test Plan Schema (Canonical)
|
|
|
|
This is the **source of truth** for all testcases and testplans in the repository.
|
|
Everything is stored as individual YAML files so they can be versioned with git.
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
testcases/<module>/<group>/<testcase-id>.yaml
|
|
```
|
|
|
|
- Top level under `testcases/` is the **module** (device/product type): `core/`, `pbx/`, `gateway/`, etc.
|
|
- `core/` = generic, cross-device concerns (documentation, security posture, installation, virtualization platform, etc.).
|
|
- `pbx/` = PBX-specific telephony concerns (extensions, calls, codecs, etc.).
|
|
- Add new device types (e.g. `gateway/`, `sbc/`) as siblings of `core/` and `pbx/`.
|
|
|
|
The old `<project>` level has been removed.
|
|
|
|
## Rules
|
|
|
|
- Every `id` must be **globally unique** across the entire system.
|
|
- Testplans **reference** testcases by their `id` (never embed content).
|
|
- The automation decision (`automation_status`) is made **at testcase level**, not per step.
|
|
- Steps are intentionally simple (`action` + `expected` + optional `notes`).
|
|
- `automatable` and `notYet` lists at testcase level are high-level summaries for reporting.
|
|
- `custom_fields` is a free-form object for future extensibility.
|
|
- Generated artifacts (from testplan export) live only under `tests/generated/<plan-id>/` and are never considered source of truth.
|
|
|
|
## testcase.yaml (example)
|
|
|
|
```yaml
|
|
id: pbx-extensions-check-create_modify_delete_sip_extensions
|
|
module: pbx
|
|
group: extensions
|
|
title: Create, modify and delete SIP extensions
|
|
description: Verify full CRUD lifecycle of SIP extensions via admin portal
|
|
priority: High
|
|
tags: [sip, provisioning]
|
|
references: []
|
|
dependencies: [] # other global testcase ids that must run before this one
|
|
|
|
preconditions: []
|
|
postconditions: []
|
|
|
|
steps:
|
|
- action: Open Extensions page in admin portal
|
|
expected: List of extensions is visible and searchable
|
|
- action: Create new SIP extension with random number and register on physical phone
|
|
expected: Extension appears in list and can register
|
|
notes: Requires physical phone hardware registration and audio validation
|
|
- action: Modify extension settings and verify changes
|
|
expected: Changes are persisted and reflected in the system
|
|
|
|
automation_status: partial # automated | partial | manual
|
|
reason_not_automatable: Requires physical phone for registration/audio validation (step 2)
|
|
|
|
automatable:
|
|
- Portal UI steps for creating and editing extensions
|
|
notYet:
|
|
- Physical phone registration and real audio validation
|
|
|
|
file: tests/pbx/extensions.spec.ts # TEMPORARY during transition only
|
|
custom_fields: {}
|
|
```
|
|
|
|
## testplan.yaml (example)
|
|
|
|
```yaml
|
|
id: pbx-regression-2026-05
|
|
name: PBX Regression Test Plan - May 2026
|
|
description: Full regression after v1.2.3
|
|
environment:
|
|
base_url: https://pbx.local:4443
|
|
tenant: lab-qa
|
|
|
|
testcases:
|
|
- ref: pbx-extensions-check-create_modify_delete_sip_extensions
|
|
severity: High
|
|
- ref: core-repository-check-presence_of_release_notes_and_changelog
|
|
severity: Medium
|
|
- ref: pbx-calls-check-internal_extension_to_extension_audio_call
|
|
severity: Critical
|
|
must_run_after:
|
|
- pbx-extensions-check-create_modify_delete_sip_extensions
|
|
|
|
execution_notes: |
|
|
Run in isolated lab environment. Have physical phones ready for media tests.
|
|
|
|
automation_summary:
|
|
total: 3
|
|
automated: 1
|
|
partial: 1
|
|
manual: 1
|
|
automatable_percentage: 66.7
|
|
|
|
custom_fields: {}
|
|
```
|
|
|
|
## Export Behavior (when implemented)
|
|
|
|
From a composed testplan:
|
|
- **Manual deliverable**: HTML containing testcases where `automation_status` is `manual` or `partial`
|
|
- **Automated deliverable**: Playwright `.spec.ts` placed in `tests/generated/<plan-id>/` (usable directly by Playwright + MCP agent)
|
|
|
|
## Automation Model (Simplified)
|
|
|
|
- `automation_status` is the single source of truth for whether a testcase is automated.
|
|
- Steps are kept simple on purpose (`action` + `expected` + optional free-text `notes`).
|
|
- `automatable` / `notYet` lists at testcase level are **summaries** for reporting and planning.
|
|
- Per-step automation flags were removed (decision moved to testcase level).
|
|
|
|
## Transition Notes
|
|
|
|
- Until migration is complete, the old `editor/data/testcases.json` and existing `tests/*.spec.ts` remain the temporary source of truth.
|
|
- Once the YAML tree is populated and the editor reads from it, the YAML files become the single source of truth.
|
|
- New testcases must be created as YAML files following this schema.
|
|
|
|
## Versioning
|
|
|
|
All changes to these YAML files are tracked with normal git. No application-level versioning is required.
|