autotest/editor/tasks/automated-code-generation.md

# Automated Test Code Generation - Progress Tracker

**Project Goal**: Turn the Autotest Web Editor into the central "director" that can generate real, executable Playwright + MCP test files for all tests marked as fully automated.

**Current Decisions (Locked)**:
- Output folder: `tests/generated/{core,pbx}/`
- File naming: Same names as existing tests (e.g. `documentation.spec.ts`)
- Generation style: Template-based + LLM under the hood
- Granularity: One spec file per group
- First target group: `core-documentation`
- Human must always review & commit
- LLM context: Full group + focus on current test
- Update `testInfo.annotations` in generated files

**Status**: Active development — generator functional, first real code being produced (2026-05-22)

---

## Phases & Tasks

### Phase 0: Setup & Foundations

- [x] Create `editor/tasks/automated-code-generation.md` (this file)
- [x] Create folder `tests/generated/core/` and `tests/generated/pbx/`
- [ ] Decide on generator script location (`editor/scripts/generate-playwright-tests.ts`)
- [ ] Add `ts-node` or proper TypeScript execution if needed for scripts
- [x] Create basic `ActionCatalog` types in `editor/scripts/types/automated-generation.ts`
- [x] Create initial Action Catalog structure (`editor/scripts/action-catalog/`)

**Status**: Mostly done (script runner decision pending)

---

### Phase 1: Data Model Extensions

- [ ] Add new fields to the internal test model:
  - `fullyAutomated?: boolean`
  - `codeSteps?: Array<{ stepName: string, actionKey: string, params?: any }>`
  - `implementationNotes?: string`
  - `verifiedSnippets?: Array<{ stepName: string, code: string }>`
- [ ] Update `editor/data/testcases.json` schema (or handle missing fields gracefully)
- [x] Add UI section in the test modal for editing these new fields (basic codeSteps + params supported)
- [x] Add a filter "Only Fully Automated" in the main table / selection toolbar (selection works, visual row filter pending)

**Owner**: Frontend + Data
**Priority**: High

---

### Phase 2: Action Catalog

- [x] Design `action-catalog.json` structure
- [x] Create folder `editor/scripts/action-catalog/`
- [x] Implement TypeScript loader (`loadActionCatalog()`)
- [x] Add first batch of entries for `core-documentation` group
- [x] Support parameter substitution (e.g. `{{pattern}}`)

**Owner**: Backend / Scripts
**Priority**: High
**Status**: Foundation done, content population in progress

---

### Phase 3: Core Code Generator

- [x] Create generator script (now `.js`)
- [x] Support filtering by `status === "automated"` + `fullyAutomated`
- [x] Group tests by their `group` field
- [x] Generate one `.spec.ts` file per group in `tests/generated/`
- [x] Render `test.step()` blocks using the Action Catalog
- [x] Emit standard header + `testInfo.annotations`
- [x] Add basic CLI interface (`--group`, `--dry-run`)

**Priority**: High

---

### Phase 4: First Real Generation (core-documentation)

- [x] Populate enough catalog entries to generate `documentation.spec.ts` (2/4 tests done)
- [x] Run generator against `core-documentation` tests (dry-run verified, real write via UI works)
- [ ] Manually review the output
- [ ] Iterate on templates until the generated file is acceptable
- [ ] Commit first generated file (after review)

**Milestone**: First generated spec file

---

### Phase 5: Editor Integration

- [x] Add new button: **"Export Automated Playwright Tests"**
- [x] Add backend endpoint `POST /api/export/automated-tests`
- [x] Support real generation (writes files when called from UI)
- [ ] Add confirmation modal showing which groups will be generated
- [ ] Show link to the generated files after success

**Priority**: Medium

---

### Phase 6: LLM / MCP Assistance (Under the Hood)

- [ ] Create helper to build rich context (full group + current test + catalog + examples)
- [ ] Integrate LLM call (via MCP or direct client)
- [ ] Add fallback: when no catalog match → ask LLM
- [ ] Store accepted LLM suggestions back into `verifiedSnippets` or catalog (optional learning loop)

**Priority**: Medium

---

### Phase 7: Regeneration & Safety

- [ ] Define regeneration markers in generated files
- [ ] Implement update logic (preserve manual code above marker)
- [ ] Add `--force` and `--update-only` flags to the generator
- [ ] Document the regeneration contract

---

### Phase 8: Polish & Documentation

- [ ] Improve error messages and validation in generator
- [ ] Add tests for the generator (at least for `core-documentation`)
- [ ] Write `editor/docs/automated-code-generation.md`
- [ ] Update main `README.md` with new workflow
- [ ] Add "Fully Automated" badge/filter in the web editor

---

## Current Status

| Phase | Status       | Notes                              |
|-------|--------------|------------------------------------|
| 0     | In Progress  | Tracking file created              |
| 1     | Not Started  | Data model changes needed          |
| 2     | Not Started  | Action Catalog design pending      |
| 3     | Not Started  | Core generator not written yet     |
| 4     | Not Started  | First real generation              |
| 5     | Not Started  | Editor button not implemented      |
| 6     | Not Started  | LLM integration later              |
| 7     | Not Started  | Regeneration strategy              |
| 8     | Not Started  | Documentation                      |

**Next Immediate Task**: Continue Phase 4 — enrich remaining core-documentation tests and improve catalog actions.

---

## Notes & Decisions Log

- 2026-05-22: Confirmed output goes to `tests/generated/`
- 2026-05-22: First target group = `core-documentation`
- 2026-05-22: Template-based + LLM under the hood
- 2026-05-22: Human review & commit required

---

**Current Status (Updated 2026-05-22 evening)**

| Phase | Status          | Notes                                                                 |
|-------|-----------------|-----------------------------------------------------------------------|
| 0     | Done            | Tracking file + folders + basic types + JS generator created          |
| 1     | In Progress     | `fullyAutomated`, `implementationNotes`, `codeSteps` + params editor added. Two tests enriched. |
| 2     | In Progress     | Catalog + loader done. Good entries for core-documentation + substitution working. |
| 3     | In Progress     | Generator fully working in JS, supports group + real file writing via UI. |
| 4     | In Progress     | 2 out of 4 core-documentation tests now generate real code. |
| 5     | Done            | Export button + backend fully wired and functional. |
| 6     | Not Started     | LLM/MCP fallback layer                                                |
| 7     | Not Started     | Regeneration safety markers                                           |
| 8     | Not Started     | Documentation & polish                                                |

**Next Immediate Recommended Task**: Enrich remaining core-documentation tests and expand Action Catalog for a complete group generation.

---

**Last Updated**: 2026-05-22 (evening session)

---

## Session Summary (2026-05-22)

**Work completed in this session:**
- Added `fullyAutomated` checkbox + `implementationNotes` + basic `codeSteps` editor in the modal (Phase 1)
- Added "Only Fully Automated" bulk selection button (B)
- Created Action Catalog foundation + first entries for `core-documentation` (C)
- Created `generate-playwright-tests.ts` script (Phase 3 start)
- Added "Export Automated Tests" button in the UI + backend endpoint
- Updated progress tracker

**Current blockers / next work:**
- Generator currently only does dry-run. Needs to actually write files.
- Need to test generation for `core-documentation` group.
- `codeSteps` editor is still basic (can be improved).
**Current Focus**: Setting up the task tracker and preparing Phase 1

---

## Session Summary (2026-05-22 - Evening continuation)

**Work completed in this session:**
- Converted generator to plain JavaScript (`generate-playwright-tests.js`) + catalog loader for reliable execution without ts-node.
- Fixed `require` paths and made the generator fully functional.
- Improved `codeSteps` editor in the modal to support `params` (JSON input).
- Wired the "Export Automated Tests" button + backend to actually call the generator with `--group`.
- Enriched the first two `core-documentation` tests with real `fullyAutomated: true`, `implementationNotes`, and detailed `codeSteps` + params.
- Verified via `--dry-run` that the generator now produces clean, executable Playwright code with proper parameter substitution (e.g. `release-notes`, `cli-configuration`).
- Two tests in `core-documentation` now generate real steps instead of placeholders.

**Updated Status Table** (see below)

**Current blockers / open items:**
- Remaining two `core-documentation` tests still lack `codeSteps`.
- Action Catalog entries are still basic (many actions just do visibility checks).
- No regeneration marker / preserve-manual-sections logic yet.
- No visual row filter for "Only Fully Automated" (selection works, but table rows still show everything).

**Recommended next steps for tomorrow:**
1. Enrich the other two `core-documentation` tests (`check-accuracy_installation...` and `check-accuracy_trouble_shooting...`) with proper `codeSteps`.
2. Expand the Action Catalog with more realistic actions (file extraction, text parsing, writing outputs, etc.).
3. Test a real (non-dry-run) generation via the UI button and review the file in `tests/generated/core/documentation.spec.ts`.
4. (Optional) Add a proper "Only Fully Automated" visual filter to hide non-automated rows in the table.

---

## Current Status (Updated 2026-05-22 evening)

| Phase | Status          | Notes                                                                 |
|-------|-----------------|-----------------------------------------------------------------------|
| 0     | Done            | Tracking file + folders + basic types + JS generator created          |
| 1     | In Progress     | `fullyAutomated`, `implementationNotes`, `codeSteps` + params editor added. Two tests enriched. Filter button exists. |
| 2     | In Progress     | Catalog + loader done. Initial useful entries for core-documentation added. Substitution works. |
| 3     | In Progress     | Generator fully working (JS), supports `--group` + `--dry-run`, writes real files when called from UI. |
| 4     | In Progress     | First real generation for `core-documentation` partially done (2/4 tests enriched and generating cleanly). |
| 5     | Done            | "Export Automated Tests" button + backend endpoint implemented and tested. |
| 6     | Not Started     | LLM/MCP fallback layer                                                |
| 7     | Not Started     | Regeneration safety markers + preserve manual code                    |
| 8     | Not Started     | Documentation & polish                                                |

**Next Immediate Recommended Task (for tomorrow)**: Finish enriching all 4 tests in `core-documentation` and expand the Action Catalog so a complete, high-quality group can be generated.