diff options
Diffstat (limited to 'docs/design.md')
| -rw-r--r-- | docs/design.md | 329 |
1 files changed, 329 insertions, 0 deletions
diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..5f691d7 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,329 @@ +# wf — design reference + +Shared task workflow for many coding agents (Claude Code sessions) working in parallel on the +projects under one folder (`/projects`). This file is the reference for the TASKS.md format, +`workflow.toml`, every `wf` command and the release rules. The memory/CPU ledger `wf res` has its +own document: [resource-ledger.md](resource-ledger.md). Overview: [README](../README.md). + +## 1. Goal + +One workflow definition and one tool, used by every agent working in a subfolder of +`/projects`, instead of task scripts and workflow text copied (and drifting) per project. +Project `CLAUDE.md` files keep project specifics only. + +Success: +- An agent started in any project answers "continue" with the same steps. +- A workflow fix is made once, in one repo, under revision control. +- Task references cannot rot silently (`wf check` fails on them). +- Less always-loaded instruction text per project. +- A worker that hits a workflow problem reports it in one command and keeps working; fixes + reach all workers without breaking one that is mid-task. +- Routine task upkeep (list, find, add, reprioritise, flag, finish) is one short command + each; an agent never reads or rewrites TASKS.md whole to do it. + +Non-goals: +- No runner: the task tool never builds, tests, commits or pushes. + +## 2. Layout + +``` +/projects/ + CLAUDE.md -> workflow/shared/CLAUDE.md symlink + workflow/ this repo + shared/CLAUDE.md shared loop (section 6), loaded by every project + CLAUDE.md rules for sessions changing this repo + wf.py CLI entry + wf_res.py `wf res` (files, /proc, systemd); logic in wflib/res.py + wflib/tasks.py pure text: parse, insert, remove, archive + wflib/refs.py pure text: ids, links, anchors, doc sections + wflib/config.py find project root, load workflow.toml + wflib/check.py everything `wf check` reports + wflib/ledgers.py in-flight plan ledgers for `wf next` + wflib/search.py pure text: ranked search over tasks, archive, docs + wflib/migrate.py one-time converter from a numbered task list + wflib/res.py pure logic of the resource ledger + tests/test_*.py unittest, stdlib + inbox.md reports from workers, append-only, not in git (section 7) + templates/ workflow.toml, TASKS.md, archive.md for `wf init` + CHANGES.md release notes, newest first + <group>/<project>/ any depth ≤ 3; a project = a folder with workflow.toml +``` + +Loading: Claude Code reads `CLAUDE.md` from the working directory and every parent, so +sessions in a project or in its `.worktrees/*` get the shared file without a pointer. If a +symlink is not loaded, a real one-line file `/projects/CLAUDE.md` containing +`@workflow/shared/CLAUDE.md` does the same. The shared file sits in `shared/` so that a session +inside `/projects/public/workflow` does not load it twice. + +Each project `CLAUDE.md` starts with: +`Workflow: /projects/CLAUDE.md (shared loop, wf). Below = project specifics; this file wins on conflict.` + +The folder scanned by `wf projects` is the parent of this repo, or its grandparent when the parent holds no project (`WF_ROOT` overrides); the inbox +is `inbox.md` here (`WF_INBOX` overrides). The tasks of the workflow itself can live in a +separate (private) wf project; nothing in this repo needs them. + +Requirements: Linux, Python ≥ 3.11 (stdlib only). `wf res` also needs systemd user units with +cgroup v2. + +## 3. TASKS.md format + +```markdown +# Tasks — <project> + +<free prose: commands, pointers> + +## Awaiting your decision + +- **a-smoke-screens**: needs you present, screen unlocked. ... + +## Pending + +- **t-cave-seams** [P3] (<1h): Terrain: cave entrance seams. Close the slit beside the lintel. + - Steps: ... + - Tests (red first): ... + - Done: ... + - After: [[t-terrain-cave-render]] + Ref: DESIGN.md#terrain + +## Needs human + +- **t-playtest-feel** [P1] (<1h): Play-test feel + tuning. ... + - [ ] ... + +## Deferred + +- **t-android-port** [P3] (10h): ... +``` + +Rules: +- Item = a line `- **<id>**` at column 0 plus every following line that is blank or indented. + A flush-left non-item line ends the item list of the section (prose after items is kept). +- Id grammar: `t-[a-z0-9-]+` for tasks (Pending, Needs human, Deferred), `a-[a-z0-9-]+` for + Awaiting items. A task keeps its id when it moves between sections. + (Changed from the chat draft, which used `h-` for Needs human: a prefix tied to a section + would force an id change on a move.) +- Ids are unique across TASKS.md and the archive, and never reused. +- Task header: `- **<id>** [P0-P3] (<effort>) [status]: Title. Goal.` (old `(<effort>, interactive)` + still reads as `Sessions: owner`; `check` warns.) + Effort is one of `<1h`, `1h`, `5h`, `10h`, `100h`. Awaiting items have no priority or effort. +- Status, optional, fixed words: `(in progress: <branch or note>)`, `(blocked: [[a-id]])`. +- `After: [[id]], [[id]]` lists dependencies. `Ref:` lists `path` or `path#anchor`, comma + separated, relative to the project root; text in parentheses after a ref is a free note. +- `Model: haiku|sonnet|opus` (none = opus): the model for the task (docs/lanes.md §0). + `Sessions: parallel|solo|owner [— why]` (none = parallel; `- Sessions:` works too): + solo = only while no other session is live in the project (lib/submodule bump, repo-wide refactor, + shared runner), and while it is in progress no other session picks anything; owner = needs the owner + present (real display, playtest, choices): `next` picks it only with `--owner`. +- Slices of a task over 1h: own items `t-<parent>-1`, `t-<parent>-2` (or any id given with + `--id`), each `After:` the previous one. The parent item stays as the summary and lists its + slices (`Slices:` line); it is done when its last slice is done, and `next` does not pick it + while a slice is open. +- Pending holds open work only, in pick order. +- Sections other than the four named ones are prose: the tool keeps them and checks only + the `[[id]]` links in them. + +Archive file: newest first, one line per finished task: +`- 2026-09-29 **t-cave-seams** Terrain: cave entrance seams — <entry, ≤2 lines>`. +Older lines without an id stay as they are. + +## 4. workflow.toml + +In the project root. Its presence marks the project root for `wf`. + +```toml +format = 1 # required: TASKS format version +tasks = "TASKS.md" # required +archive = "tasks/archive.md" # required +docs = ["DESIGN.md", "docs/"] # files/dirs whose links and anchors `check` validates +verify = ["dotnet build X.slnx", "dotnet test X.slnx"] # printed by `done`, never run +done = ["Feature changed -> docs/CATALOG.md"] # printed by `done` +ledgers = ".superpowers/sdd" # optional: in-flight plan ledgers +ctx_hint = 100000 # optional: tokens; 0 = off (see below) +worktree_setup = ["mkdir -p out && ln -sfn \"$WF_MAIN/out/data\" out/data"] # optional: `wf setup` (below) +quick_gate = ["dotnet test X.slnx --filter ContractTests"] # optional: `wf gate` (below) +cloud = true # optional: project opts in to the cloud lane (default false) +cloud_include = ["out/data/"] # optional: git-ignored paths force-added to the cloud snapshot +cloud_note = "No GUI here." # optional: appended to the cloud prompt + +[anchors] # optional: index/spec anchor scheme +index = "DESIGN.md" # every task Ref anchor must be a heading here +index_section = "Subsystems" +specs = "docs/superpowers/specs" # explicit <a id> here needs an index entry +``` + +Only `format`, `tasks` and `archive` are required. Unknown keys are an error (catches typos). + +`worktree_setup`: shell commands `wf setup` runs inside a lane worktree (cwd = the worktree's project +folder, env `WF_MAIN` = the main tree's project folder), in order, stopping at the first non-zero exit +(`wf: …`, exit 1). They provide git-ignored inputs a fresh worktree lacks (e.g. link `out/…` from the +main tree) and must be idempotent: wf-worker runs it after every worktree create/reuse. + +`cloud` fit (cloud-lane spec §4.5): with `cloud = true` a task fits when runner-ready, not `Sessions: owner|solo`, +no `Cloud: no` line, effort <= `slice_above`, Model opus, and its title/body does not match the unfit regex list +in `wflib/cloud.py` (`wf` / `wf res` steps, GUI, LAN hosts, live server). Routing is per task: whoever adds it sets +`Cloud: yes|no` (`wf add/set --cloud`); `Cloud: yes` skips the regex list only (still opus only), `Cloud: no` +always excludes, no line = the rules above. `wf orch pick cloud` order (`cloud.pick_key`): `Cloud: yes` first, +then prio, then larger effort. `wf check` accepts the `Cloud:` line (yes|no, one per task). + +`quick_gate`: fast regression commands (minutes, not the full slow gate) `wf gate` runs in the current +tree's project folder (worktree or main; env `WF_MAIN`), stopping at the first red (`wf: …`, exit 1); +none configured → exit 0. wf-worker runs it before `wf done`, so a break is caught by the task that made +it, not blamed on a later task by the async slow gate. The project picks the commands. + +## 5. wf CLI + +Run as `python3 /projects/public/workflow/wf.py <cmd>` from anywhere inside a project. The +project root is the nearest ancestor of the working directory that holds `workflow.toml`; +`--project DIR` overrides. Output is plain text, exit 0 on success. + +Principle: every routine operation is one command with short output, so agents spend no +context on reading or rewriting task files. Hand edits are for body prose only. + +Read (never write): + +| Command | Does | +|---|---| +| `next [--brief]` | Awaiting items, Needs-human titles, in-flight ledgers, then the next task with `ctx` of its refs. `--brief`: the task only. Skips blocked items, open `After:`, parents with open slices, tasks held by another live session, `Sessions: owner` (unless `--owner`), `Sessions: solo` while another session is live. A solo task in progress by another live session → picks nothing, exit 1. Exit 1 if nothing is pickable. | +| `list [filters]` | One line per item: `id P2 1h status Title` (title cut to fit 100 columns). Default: Pending. Filters: `-s pending\|human\|awaiting\|deferred\|all`, `-p 0-3` (that priority or higher), `--ready` (pickable now), `--runner` (ready + Done line, not owner-bound, no open slices), `--blocked`, `--progress`, `--ref <path[#anchor]>`, `-n N`. Last line: counts per section. | +| `show <id>...` | The raw item(s), nothing resolved. | +| `ctx <id \| path#anchor>` | The item, its refs resolved, its dependencies and the items that depend on or link to it, then the notes of every area (`## Areas`) the item names (area name, code-map anchor or Paths entry as a whole word). For an anchor: that doc section plus the tasks that reference it. | +| `search <words>... [filters]` | Ranked hits over tasks, archive and `docs`: one line each, `id` or `path:line heading`, plus the matching line. Filters: `--tasks`, `--archive`, `--docs`, `-n N` (default 15). | +| `log [-n N] [words]` | Newest archive lines (default 10), optionally only those matching the words. | +| `projects` | Every project under the folder that holds the workflow repo (`/projects`) with a `workflow.toml` (depth ≤ 3): counts per section, next task id + title, number of `check` errors. | +| `check` | Validates; prints every problem; exit 1 on any error. | + +Write (each changes only the items named, and prints the changed header lines): + +| Command | Does | +|---|---| +| `add "<Title. Goal.>" -p N -e EFFORT [opts]` | New item. Id made from the title unless `--id` given or the text starts with `<id>: `; printed. Opts: `-s SECTION`, `--after ID,…`, `--ref REF,…`, `--parent ID` (slice: id `t-<parent>-N` or `--id`, `After:` the previous slice, parent's slice list updated), `--model M`, `--sessions solo|owner`, `-b` (body lines from stdin; never read unasked, a harness may hold stdin open). `add -` reads a whole item block from stdin. | +| `done <id>... [-m "<entry>"]` | Removes the task(s), puts the dated line(s) on top of the archive, prints the `verify` and `done` lists once. `-m` is required for a single task and applies to it; several ids without `-m` use each task's goal sentence. | +| `prio <id> <0-3>` | Sets the priority and moves the item to its place by the insert rule. | +| `move <id> <section>` or `move <id> --before\|--after <id>` | Moves between sections, or reorders inside one (refused if it breaks priority or `After:` order, unless `--force`). | +| `status <id> progress "<note>" \| blocked <a-id> \| clear` | Sets or clears the status words. | +| `set <id> [--title T] [--effort E] [--after IDS] [--ref REFS] [--model M] [--sessions S] [--done TEXT]` | Changes header fields and the `After:` / `Ref:` / `Model:` / `Sessions:` / `Done:` lines (`""` removes). `--interactive` = old spelling of `--sessions owner`. `--after ""` removes the line. `--title` keeps the goal, unless it has its own (`Title. Goal.`) or ends in `?`/`!`: then it replaces the whole text. | +| `rename <id> <new>` | Changes an open item's id (same kind, unused, not archived) and every `[[id]]` link in TASKS.md. | +| `note <id> "<line>"` | Appends one `- <line>` to the item body. | +| `body <id>` | Replaces the item body with stdin (header, `After:`, `Ref:` kept). | +| `tick <id> <n \| text>` | Checks the n-th (or the matching) `- [ ]` box of a Needs-human item. | +| `report "<what happened>" [--kind bug\|idea\|friction] [--cmd "<command>"]` | Appends one entry to the workflow `inbox.md` (section 7). Touches nothing in the project. | +| `batch N [--lanes L,…] [--prep] [--model M] [--for 6h] [--mem 4G] [--dry-run]` · `batch --status` | Starts an unattended batch orchestrator (`claude -p`, auto mode, no prompts, prompt `templates/batch-prompt.md`) as a `wf res run` job with `CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS` = `--for`; it writes `out/wf-batch-<stamp>.md`. `--status` = this project's newest batch job + newest summary file. `--prep` = prep batch (`templates/prep-prompt.md`): sonnet workers write Done (+Model) for up to N tasks without Done, unclear → awaiting; none → nothing started. See `docs/orchestrator.md`. | +| `init` | In a folder without `workflow.toml`: writes it, TASKS.md and the archive from the templates. | +| `migrate [--write]` | One-time converter from the numbered format. Without `--write`: prints the diff, the id map and notes, changes nothing. `--write` also raises `format` in `workflow.toml`. | + +Every write command accepts `--dry-run` (prints a unified diff, writes nothing) and ends by +comparing the full check before and after: a change that adds a problem is refused and +nothing is written; problems that were there before do not block other edits. + +Details: + +- **Pick rule (`next`)**: first Pending item, in file order, that has no `(blocked: …)` and + whose `After:` ids are all in the archive. Items it skipped are listed in one line each + with the reason. +- **Ref resolution (`ctx`, `next`)**: `path#anchor` → the section under the heading whose + slug or explicit `<a id>` equals the anchor, up to the next heading of the same or higher + level, cut at 80 lines with a `… (N more lines: <path>:<line>)` note. `path` alone → the + file's first heading and its `**Goal:**` line if it has one. With `[anchors]` set, an + index anchor also prints the spec section that carries the same `<a id>`. +- **Context hint (`next`, `done`)**: with `CLAUDE_CODE_SESSION_ID` set, the last request's prompt size + (input + cache write + cache read) from the last 1 MB of `<CLAUDE_CONFIG_DIR|~/.claude>/projects/*/<id>.jsonl`; + over `ctx_hint` → last line `context ~Nk tokens (> Mk): ask the owner to /clear, then continue`. No id, + no transcript → nothing. Subagents share the id, so the line says it is about the main session. +- **Ledgers (`next`)**: for each `<ledgers>/*/progress.md`: plan path from its first line, + done tasks, where to resume, count of `Ruling:` lines. +- **Insert rule (`add`, `prio`, `move <section>`)**: position in the section = after the + last item of the same or higher priority, and after every item named in its `After:`. + Body is re-indented to two spaces. A block given to `add -` must carry an id and, for + tasks, a priority; a leading `N.` from the old format is refused with a message. +- **Id from title (`add`)**: `t-` (`a-` for Awaiting) + slug of the title (the text up to + the first `. `), at most 40 characters, cut at a word; if taken (TASKS.md or archive), `-2`, `-3`, … is appended. +- **Search ranking**: words are matched case-insensitively as whole words or word prefixes. + Score per hit = words matched × weight of where (id or title 5, doc heading 4, header + goal 3, `Ref:`/body/doc text 1); all words matched ranks before some. Ties: open tasks, + then docs, then archive. No index file; everything is read per call. +- **Exact ids only** in write commands; a unique id prefix is accepted by read commands. + An unknown id prints the three nearest ids. +- **`done`** refuses when the id is unknown or is a parent with open slices. On an `a-` id + it removes the item without an archive line (`-m` is optional there) and clears the + `(blocked: [[a-id]])` status of the tasks that waited on it, listing them. +- **`check`** errors: + - duplicate id; id reused from the archive; bad id grammar; id prefix not allowed in its section + - task without priority or with an effort outside the vocabulary + - `[[id]]` that is neither in TASKS.md nor in the archive + - `(blocked: [[x]])` where `x` is not an open `a-` item + - `After:` cycle; Pending item placed before an open item it is `After:` + - `Ref:` path missing; anchor missing in that file + - old-format leftovers: numbered items + - item after a flush-left prose line inside a task section; malformed item header + - with `[anchors]`: task anchor without index heading; two index headings with the same + slug; index link to a missing file or anchor; explicit spec `<a id>` without index entry + - config: missing required key, unknown key, configured path that does not exist + In `docs` only `[[t-…]]` / `[[a-…]]` links are validated (other `[[x]]` may be the + project's own wiki links). + Warnings (exit stays 0): `#N` number refs in TASKS.md; task in progress with no branch/note; Awaiting item that no task + references and that is older than 30 days by `git blame`. +- **Writes** are whole-file: read, change in memory, write to a temp file in the same + directory, rename. If the file changed on disk between read and write (mtime + size), the + command stops without writing. `done` writes the archive first, then TASKS.md; if the + second write fails it reports that the archive line must be removed. +- **Errors** go to stderr as one line each, naming file and id; no tracebacks for expected + failures. + +## 6. Shared CLAUDE.md + +Text: `shared/CLAUDE.md` (the file is the authority). Sections: Style (concise everywhere; +compaction summaries, specs, plans stay full and exact), Cold start, Loop, Done, TASKS.md, +Context economy, Memory / CPU, Workflow problems. + +## 7. Improving the workflow without disturbing workers + +Problem: `wf.py` and the shared CLAUDE.md are live for every worker the moment they change. +A worker that edits them mid-task, or a half-finished change, would hit all the others. + +**Report (any worker, any time)** +- `wf report` appends to `inbox.md` with a single `O_APPEND` write, so parallel workers + never clash and nobody waits. Entry: date, project, kind, the text, the command if given, + wf version (git short hash). + `- 2026-09-29 proj-a friction: prio change needs two commands (cmd: wf prio …) @84decd1` +- The worker then works around the problem and continues its own task. It does not edit + anything under `/projects/public/workflow` and does not wait for a fix. +- `wf next` in any project prints one line when the inbox has entries, for the user's eyes: + `workflow inbox: 3 reports (triage: workflow session)`. + +**Triage and fix (a workflow session, started by the owner)** +- The workflow's own tasks live in a wf project of their own (e.g. a private + `/projects/workflow`), whose CLAUDE.md holds the rules of this section. Cold start there: + each inbox entry becomes a task (`wf add`, duplicates merged) or is rejected with a line in the + archive; the entry is then deleted from `inbox.md`. `inbox.md` is git-ignored; tasks and + archive carry the record. Bookkeeping there changes nothing a worker runs. +- Changes to `wf.py`, `wflib/`, `shared/CLAUDE.md` or templates happen in a git worktree + `/projects/public/workflow/.worktrees/<topic>` on a branch, never in the main tree. Workers keep + running the main tree's `master` meanwhile. +- Release = fast-forward merge to `master` in the main tree after: tests green, and + `wf check` run with the new code against every project listed by `wf projects` gives the + same result as before the change (or differences that are the intended ones). +- The user starts workflow sessions and sees the release; no worker does it on its own. + +**Compatibility rules for a release** +- Commands and options are only added. Renaming or removing one keeps the old spelling + working for one release, printing a one-line notice with the new spelling. +- A change to the TASKS.md format raises `format` and ships with a `wf migrate` step. + `wf` run in a project with an older `format` refuses write commands with + `TASKS format 1, wf needs 2: run wf migrate --write (idle project, one commit)`; read + commands keep working on the old format. +- Shared CLAUDE.md text changes reach a worker at its next session start; a rule change + must therefore not make files written under the old rule invalid. +- Every release adds a line to `CHANGES.md` (newest first): date, what changed, what a + project has to do, if anything. + +## 8. Testing + +- `wflib` functions are pure (text in, text out); unit tests use literal before/after texts + written by hand, never output of the code under test. +- CLI tests run `wf.py` with `--project <tempdir>` on fixture projects: pick rule, skip + reasons, insert position with priority and `After:`, `done` refusals, every `check` error + once, write-conflict stop, `list` filters, `prio`/`move` placement and refusals, `status`, + `set`, `note`, `body`, `tick`, search ranking order on a fixed corpus, `report` from two + processes at once (both entries whole), `format` gate, migrate on a fixture with the known oddities (stray `N.`, + `N/M` titles, `After:` by title, prose after items). +- Each new test is seen failing first, and again with the tested line broken by hand. |