juan/autotest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8feb88dfa6d1a78a8aaf23dae42d827e0d116b25
autotest/editor/tasks/automated-code-generation.md
T
juan 8feb88dfa6 vibe coded maxx
2026-05-22 13:46:02 +02:00

11 KiB
Raw Blame History

Automated Test Code Generation - Progress Tracker

Project Goal: Turn the Autotest Web Editor into the central "director" that can generate real, executable Playwright + MCP test files for all tests marked as fully automated.

Current Decisions (Locked):

  • Output folder: tests/generated/{core,pbx}/
  • File naming: Same names as existing tests (e.g. documentation.spec.ts)
  • Generation style: Template-based + LLM under the hood
  • Granularity: One spec file per group
  • First target group: core-documentation
  • Human must always review & commit
  • LLM context: Full group + focus on current test
  • Update testInfo.annotations in generated files

Status: Active development — generator functional, first real code being produced (2026-05-22)

Phases & Tasks

Phase 0: Setup & Foundations

  • Create editor/tasks/automated-code-generation.md (this file)
  • Create folder tests/generated/core/ and tests/generated/pbx/
  • Decide on generator script location (editor/scripts/generate-playwright-tests.ts)
  • Add ts-node or proper TypeScript execution if needed for scripts
  • Create basic ActionCatalog types in editor/scripts/types/automated-generation.ts
  • Create initial Action Catalog structure (editor/scripts/action-catalog/)

Status: Mostly done (script runner decision pending)

Phase 1: Data Model Extensions

  • Add new fields to the internal test model:
    • fullyAutomated?: boolean
    • codeSteps?: Array<{ stepName: string, actionKey: string, params?: any }>
    • implementationNotes?: string
    • verifiedSnippets?: Array<{ stepName: string, code: string }>
  • Update editor/data/testcases.json schema (or handle missing fields gracefully)
  • Add UI section in the test modal for editing these new fields (basic codeSteps + params supported)
  • Add a filter "Only Fully Automated" in the main table / selection toolbar (selection works, visual row filter pending)

Owner: Frontend + Data Priority: High

Phase 2: Action Catalog

  • Design action-catalog.json structure
  • Create folder editor/scripts/action-catalog/
  • Implement TypeScript loader (loadActionCatalog())
  • Add first batch of entries for core-documentation group
  • Support parameter substitution (e.g. {{pattern}})

Owner: Backend / Scripts Priority: High Status: Foundation done, content population in progress

Phase 3: Core Code Generator

  • Create generator script (now .js)
  • Support filtering by status === "automated" + fullyAutomated
  • Group tests by their group field
  • Generate one .spec.ts file per group in tests/generated/
  • Render test.step() blocks using the Action Catalog
  • Emit standard header + testInfo.annotations
  • Add basic CLI interface (--group, --dry-run)

Priority: High

Phase 4: First Real Generation (core-documentation)

  • Populate enough catalog entries to generate documentation.spec.ts (2/4 tests done)
  • Run generator against core-documentation tests (dry-run verified, real write via UI works)
  • Manually review the output
  • Iterate on templates until the generated file is acceptable
  • Commit first generated file (after review)

Milestone: First generated spec file

Phase 5: Editor Integration

  • Add new button: "Export Automated Playwright Tests"
  • Add backend endpoint POST /api/export/automated-tests
  • Support real generation (writes files when called from UI)
  • Add confirmation modal showing which groups will be generated
  • Show link to the generated files after success

Priority: Medium

Phase 6: LLM / MCP Assistance (Under the Hood)

  • Create helper to build rich context (full group + current test + catalog + examples)
  • Integrate LLM call (via MCP or direct client)
  • Add fallback: when no catalog match → ask LLM
  • Store accepted LLM suggestions back into verifiedSnippets or catalog (optional learning loop)

Priority: Medium

Phase 7: Regeneration & Safety

  • Define regeneration markers in generated files
  • Implement update logic (preserve manual code above marker)
  • Add --force and --update-only flags to the generator
  • Document the regeneration contract

Phase 8: Polish & Documentation

  • Improve error messages and validation in generator
  • Add tests for the generator (at least for core-documentation)
  • Write editor/docs/automated-code-generation.md
  • Update main README.md with new workflow
  • Add "Fully Automated" badge/filter in the web editor

Current Status

Phase Status Notes
0 In Progress Tracking file created
1 Not Started Data model changes needed
2 Not Started Action Catalog design pending
3 Not Started Core generator not written yet
4 Not Started First real generation
5 Not Started Editor button not implemented
6 Not Started LLM integration later
7 Not Started Regeneration strategy
8 Not Started Documentation

Next Immediate Task: Continue Phase 4 — enrich remaining core-documentation tests and improve catalog actions.

Notes & Decisions Log

  • 2026-05-22: Confirmed output goes to tests/generated/
  • 2026-05-22: First target group = core-documentation
  • 2026-05-22: Template-based + LLM under the hood
  • 2026-05-22: Human review & commit required

Current Status (Updated 2026-05-22 evening)

Phase Status Notes
0 Done Tracking file + folders + basic types + JS generator created
1 In Progress fullyAutomated, implementationNotes, codeSteps + params editor added. Two tests enriched.
2 In Progress Catalog + loader done. Good entries for core-documentation + substitution working.
3 In Progress Generator fully working in JS, supports group + real file writing via UI.
4 In Progress 2 out of 4 core-documentation tests now generate real code.
5 Done Export button + backend fully wired and functional.
6 Not Started LLM/MCP fallback layer
7 Not Started Regeneration safety markers
8 Not Started Documentation & polish

Next Immediate Recommended Task: Enrich remaining core-documentation tests and expand Action Catalog for a complete group generation.

Last Updated: 2026-05-22 (evening session)

Session Summary (2026-05-22)

Work completed in this session:

  • Added fullyAutomated checkbox + implementationNotes + basic codeSteps editor in the modal (Phase 1)
  • Added "Only Fully Automated" bulk selection button (B)
  • Created Action Catalog foundation + first entries for core-documentation (C)
  • Created generate-playwright-tests.ts script (Phase 3 start)
  • Added "Export Automated Tests" button in the UI + backend endpoint
  • Updated progress tracker

Current blockers / next work:

  • Generator currently only does dry-run. Needs to actually write files.
  • Need to test generation for core-documentation group.
  • codeSteps editor is still basic (can be improved). Current Focus: Setting up the task tracker and preparing Phase 1

Session Summary (2026-05-22 - Evening continuation)

Work completed in this session:

  • Converted generator to plain JavaScript (generate-playwright-tests.js) + catalog loader for reliable execution without ts-node.
  • Fixed require paths and made the generator fully functional.
  • Improved codeSteps editor in the modal to support params (JSON input).
  • Wired the "Export Automated Tests" button + backend to actually call the generator with --group.
  • Enriched the first two core-documentation tests with real fullyAutomated: true, implementationNotes, and detailed codeSteps + params.
  • Verified via --dry-run that the generator now produces clean, executable Playwright code with proper parameter substitution (e.g. release-notes, cli-configuration).
  • Two tests in core-documentation now generate real steps instead of placeholders.

Updated Status Table (see below)

Current blockers / open items:

  • Remaining two core-documentation tests still lack codeSteps.
  • Action Catalog entries are still basic (many actions just do visibility checks).
  • No regeneration marker / preserve-manual-sections logic yet.
  • No visual row filter for "Only Fully Automated" (selection works, but table rows still show everything).

Recommended next steps for tomorrow:

  1. Enrich the other two core-documentation tests (check-accuracy_installation... and check-accuracy_trouble_shooting...) with proper codeSteps.
  2. Expand the Action Catalog with more realistic actions (file extraction, text parsing, writing outputs, etc.).
  3. Test a real (non-dry-run) generation via the UI button and review the file in tests/generated/core/documentation.spec.ts.
  4. (Optional) Add a proper "Only Fully Automated" visual filter to hide non-automated rows in the table.

Current Status (Updated 2026-05-22 evening)

Phase Status Notes
0 Done Tracking file + folders + basic types + JS generator created
1 In Progress fullyAutomated, implementationNotes, codeSteps + params editor added. Two tests enriched. Filter button exists.
2 In Progress Catalog + loader done. Initial useful entries for core-documentation added. Substitution works.
3 In Progress Generator fully working (JS), supports --group + --dry-run, writes real files when called from UI.
4 In Progress First real generation for core-documentation partially done (2/4 tests enriched and generating cleanly).
5 Done "Export Automated Tests" button + backend endpoint implemented and tested.
6 Not Started LLM/MCP fallback layer
7 Not Started Regeneration safety markers + preserve manual code
8 Not Started Documentation & polish

Next Immediate Recommended Task (for tomorrow): Finish enriching all 4 tests in core-documentation and expand the Action Catalog so a complete, high-quality group can be generated.

Reference in New Issue View Git Blame Copy Permalink