workflow: initial public history

1 files changed, 329 insertions, 0 deletions

diff --git a/docs/design.md b/docs/design.md
new file mode 100644
index 0000000..5f691d7
--- /dev/null
+++ b/docs/design.md
@@ -0,0 +1,329 @@
+# wf — design reference
+
+Shared task workflow for many coding agents (Claude Code sessions) working in parallel on the
+projects under one folder (`/projects`). This file is the reference for the TASKS.md format,
+`workflow.toml`, every `wf` command and the release rules. The memory/CPU ledger `wf res` has its
+own document: [resource-ledger.md](resource-ledger.md). Overview: [README](../README.md).
+
+## 1. Goal
+
+One workflow definition and one tool, used by every agent working in a subfolder of
+`/projects`, instead of task scripts and workflow text copied (and drifting) per project.
+Project `CLAUDE.md` files keep project specifics only.
+
+Success:
+- An agent started in any project answers "continue" with the same steps.
+- A workflow fix is made once, in one repo, under revision control.
+- Task references cannot rot silently (`wf check` fails on them).
+- Less always-loaded instruction text per project.
+- A worker that hits a workflow problem reports it in one command and keeps working; fixes
+  reach all workers without breaking one that is mid-task.
+- Routine task upkeep (list, find, add, reprioritise, flag, finish) is one short command
+  each; an agent never reads or rewrites TASKS.md whole to do it.
+
+Non-goals:
+- No runner: the task tool never builds, tests, commits or pushes.
+
+## 2. Layout
+
+```
+/projects/
+  CLAUDE.md -> workflow/shared/CLAUDE.md   symlink
+  workflow/                            this repo
+    shared/CLAUDE.md                   shared loop (section 6), loaded by every project
+    CLAUDE.md                          rules for sessions changing this repo
+    wf.py                              CLI entry
+    wf_res.py                          `wf res` (files, /proc, systemd); logic in wflib/res.py
+    wflib/tasks.py                     pure text: parse, insert, remove, archive
+    wflib/refs.py                      pure text: ids, links, anchors, doc sections
+    wflib/config.py                    find project root, load workflow.toml
+    wflib/check.py                     everything `wf check` reports
+    wflib/ledgers.py                   in-flight plan ledgers for `wf next`
+    wflib/search.py                    pure text: ranked search over tasks, archive, docs
+    wflib/migrate.py                   one-time converter from a numbered task list
+    wflib/res.py                       pure logic of the resource ledger
+    tests/test_*.py                    unittest, stdlib
+    inbox.md                           reports from workers, append-only, not in git (section 7)
+    templates/                         workflow.toml, TASKS.md, archive.md for `wf init`
+    CHANGES.md                         release notes, newest first
+  <group>/<project>/                   any depth ≤ 3; a project = a folder with workflow.toml
+```
+
+Loading: Claude Code reads `CLAUDE.md` from the working directory and every parent, so
+sessions in a project or in its `.worktrees/*` get the shared file without a pointer. If a
+symlink is not loaded, a real one-line file `/projects/CLAUDE.md` containing
+`@workflow/shared/CLAUDE.md` does the same. The shared file sits in `shared/` so that a session
+inside `/projects/public/workflow` does not load it twice.
+
+Each project `CLAUDE.md` starts with:
+`Workflow: /projects/CLAUDE.md (shared loop, wf). Below = project specifics; this file wins on conflict.`
+
+The folder scanned by `wf projects` is the parent of this repo, or its grandparent when the parent holds no project (`WF_ROOT` overrides); the inbox
+is `inbox.md` here (`WF_INBOX` overrides). The tasks of the workflow itself can live in a
+separate (private) wf project; nothing in this repo needs them.
+
+Requirements: Linux, Python ≥ 3.11 (stdlib only). `wf res` also needs systemd user units with
+cgroup v2.
+
+## 3. TASKS.md format
+
+```markdown
+# Tasks — <project>
+
+<free prose: commands, pointers>
+
+## Awaiting your decision
+
+- **a-smoke-screens**: needs you present, screen unlocked. ...
+
+## Pending
+
+- **t-cave-seams** [P3] (<1h): Terrain: cave entrance seams. Close the slit beside the lintel.
+  - Steps: ...
+  - Tests (red first): ...
+  - Done: ...
+  - After: [[t-terrain-cave-render]]
+  Ref: DESIGN.md#terrain
+
+## Needs human
+
+- **t-playtest-feel** [P1] (<1h): Play-test feel + tuning. ...
+  - [ ] ...
+
+## Deferred
+
+- **t-android-port** [P3] (10h): ...
+```
+
+Rules:
+- Item = a line `- **<id>**` at column 0 plus every following line that is blank or indented.
+  A flush-left non-item line ends the item list of the section (prose after items is kept).
+- Id grammar: `t-[a-z0-9-]+` for tasks (Pending, Needs human, Deferred), `a-[a-z0-9-]+` for
+  Awaiting items. A task keeps its id when it moves between sections.
+  (Changed from the chat draft, which used `h-` for Needs human: a prefix tied to a section
+  would force an id change on a move.)
+- Ids are unique across TASKS.md and the archive, and never reused.
+- Task header: `- **<id>** [P0-P3] (<effort>) [status]: Title. Goal.` (old `(<effort>, interactive)`
+  still reads as `Sessions: owner`; `check` warns.)
+  Effort is one of `<1h`, `1h`, `5h`, `10h`, `100h`. Awaiting items have no priority or effort.
+- Status, optional, fixed words: `(in progress: <branch or note>)`, `(blocked: [[a-id]])`.
+- `After: [[id]], [[id]]` lists dependencies. `Ref:` lists `path` or `path#anchor`, comma
+  separated, relative to the project root; text in parentheses after a ref is a free note.
+- `Model: haiku|sonnet|opus` (none = opus): the model for the task (docs/lanes.md §0).
+  `Sessions: parallel|solo|owner [— why]` (none = parallel; `- Sessions:` works too):
+  solo = only while no other session is live in the project (lib/submodule bump, repo-wide refactor,
+  shared runner), and while it is in progress no other session picks anything; owner = needs the owner
+  present (real display, playtest, choices): `next` picks it only with `--owner`.
+- Slices of a task over 1h: own items `t-<parent>-1`, `t-<parent>-2` (or any id given with
+  `--id`), each `After:` the previous one. The parent item stays as the summary and lists its
+  slices (`Slices:` line); it is done when its last slice is done, and `next` does not pick it
+  while a slice is open.
+- Pending holds open work only, in pick order.
+- Sections other than the four named ones are prose: the tool keeps them and checks only
+  the `[[id]]` links in them.
+
+Archive file: newest first, one line per finished task:
+`- 2026-09-29 **t-cave-seams** Terrain: cave entrance seams — <entry, ≤2 lines>`.
+Older lines without an id stay as they are.
+
+## 4. workflow.toml
+
+In the project root. Its presence marks the project root for `wf`.
+
+```toml
+format  = 1                                # required: TASKS format version
+tasks   = "TASKS.md"                       # required
+archive = "tasks/archive.md"               # required
+docs    = ["DESIGN.md", "docs/"]           # files/dirs whose links and anchors `check` validates
+verify  = ["dotnet build X.slnx", "dotnet test X.slnx"]   # printed by `done`, never run
+done    = ["Feature changed -> docs/CATALOG.md"]          # printed by `done`
+ledgers = ".superpowers/sdd"               # optional: in-flight plan ledgers
+ctx_hint = 100000                          # optional: tokens; 0 = off (see below)
+worktree_setup = ["mkdir -p out && ln -sfn \"$WF_MAIN/out/data\" out/data"]  # optional: `wf setup` (below)
+quick_gate = ["dotnet test X.slnx --filter ContractTests"]  # optional: `wf gate` (below)
+cloud   = true                             # optional: project opts in to the cloud lane (default false)
+cloud_include = ["out/data/"]              # optional: git-ignored paths force-added to the cloud snapshot
+cloud_note = "No GUI here."                # optional: appended to the cloud prompt
+
+[anchors]                                  # optional: index/spec anchor scheme
+index = "DESIGN.md"                        # every task Ref anchor must be a heading here
+index_section = "Subsystems"
+specs = "docs/superpowers/specs"           # explicit <a id> here needs an index entry
+```
+
+Only `format`, `tasks` and `archive` are required. Unknown keys are an error (catches typos).
+
+`worktree_setup`: shell commands `wf setup` runs inside a lane worktree (cwd = the worktree's project
+folder, env `WF_MAIN` = the main tree's project folder), in order, stopping at the first non-zero exit
+(`wf: …`, exit 1). They provide git-ignored inputs a fresh worktree lacks (e.g. link `out/…` from the
+main tree) and must be idempotent: wf-worker runs it after every worktree create/reuse.
+
+`cloud` fit (cloud-lane spec §4.5): with `cloud = true` a task fits when runner-ready, not `Sessions: owner|solo`,
+no `Cloud: no` line, effort <= `slice_above`, Model opus, and its title/body does not match the unfit regex list
+in `wflib/cloud.py` (`wf` / `wf res` steps, GUI, LAN hosts, live server). Routing is per task: whoever adds it sets
+`Cloud: yes|no` (`wf add/set --cloud`); `Cloud: yes` skips the regex list only (still opus only), `Cloud: no`
+always excludes, no line = the rules above. `wf orch pick cloud` order (`cloud.pick_key`): `Cloud: yes` first,
+then prio, then larger effort. `wf check` accepts the `Cloud:` line (yes|no, one per task).
+
+`quick_gate`: fast regression commands (minutes, not the full slow gate) `wf gate` runs in the current
+tree's project folder (worktree or main; env `WF_MAIN`), stopping at the first red (`wf: …`, exit 1);
+none configured → exit 0. wf-worker runs it before `wf done`, so a break is caught by the task that made
+it, not blamed on a later task by the async slow gate. The project picks the commands.
+
+## 5. wf CLI
+
+Run as `python3 /projects/public/workflow/wf.py <cmd>` from anywhere inside a project. The
+project root is the nearest ancestor of the working directory that holds `workflow.toml`;
+`--project DIR` overrides. Output is plain text, exit 0 on success.
+
+Principle: every routine operation is one command with short output, so agents spend no
+context on reading or rewriting task files. Hand edits are for body prose only.
+
+Read (never write):
+
+| Command | Does |
+|---|---|
+| `next [--brief]` | Awaiting items, Needs-human titles, in-flight ledgers, then the next task with `ctx` of its refs. `--brief`: the task only. Skips blocked items, open `After:`, parents with open slices, tasks held by another live session, `Sessions: owner` (unless `--owner`), `Sessions: solo` while another session is live. A solo task in progress by another live session → picks nothing, exit 1. Exit 1 if nothing is pickable. |
+| `list [filters]` | One line per item: `id  P2  1h  status  Title` (title cut to fit 100 columns). Default: Pending. Filters: `-s pending\|human\|awaiting\|deferred\|all`, `-p 0-3` (that priority or higher), `--ready` (pickable now), `--runner` (ready + Done line, not owner-bound, no open slices), `--blocked`, `--progress`, `--ref <path[#anchor]>`, `-n N`. Last line: counts per section. |
+| `show <id>...` | The raw item(s), nothing resolved. |
+| `ctx <id \| path#anchor>` | The item, its refs resolved, its dependencies and the items that depend on or link to it, then the notes of every area (`## Areas`) the item names (area name, code-map anchor or Paths entry as a whole word). For an anchor: that doc section plus the tasks that reference it. |
+| `search <words>... [filters]` | Ranked hits over tasks, archive and `docs`: one line each, `id` or `path:line  heading`, plus the matching line. Filters: `--tasks`, `--archive`, `--docs`, `-n N` (default 15). |
+| `log [-n N] [words]` | Newest archive lines (default 10), optionally only those matching the words. |
+| `projects` | Every project under the folder that holds the workflow repo (`/projects`) with a `workflow.toml` (depth ≤ 3): counts per section, next task id + title, number of `check` errors. |
+| `check` | Validates; prints every problem; exit 1 on any error. |
+
+Write (each changes only the items named, and prints the changed header lines):
+
+| Command | Does |
+|---|---|
+| `add "<Title. Goal.>" -p N -e EFFORT [opts]` | New item. Id made from the title unless `--id` given or the text starts with `<id>: `; printed. Opts: `-s SECTION`, `--after ID,…`, `--ref REF,…`, `--parent ID` (slice: id `t-<parent>-N` or `--id`, `After:` the previous slice, parent's slice list updated), `--model M`, `--sessions solo|owner`, `-b` (body lines from stdin; never read unasked, a harness may hold stdin open). `add -` reads a whole item block from stdin. |
+| `done <id>... [-m "<entry>"]` | Removes the task(s), puts the dated line(s) on top of the archive, prints the `verify` and `done` lists once. `-m` is required for a single task and applies to it; several ids without `-m` use each task's goal sentence. |
+| `prio <id> <0-3>` | Sets the priority and moves the item to its place by the insert rule. |
+| `move <id> <section>` or `move <id> --before\|--after <id>` | Moves between sections, or reorders inside one (refused if it breaks priority or `After:` order, unless `--force`). |
+| `status <id> progress "<note>" \| blocked <a-id> \| clear` | Sets or clears the status words. |
+| `set <id> [--title T] [--effort E] [--after IDS] [--ref REFS] [--model M] [--sessions S] [--done TEXT]` | Changes header fields and the `After:` / `Ref:` / `Model:` / `Sessions:` / `Done:` lines (`""` removes). `--interactive` = old spelling of `--sessions owner`. `--after ""` removes the line. `--title` keeps the goal, unless it has its own (`Title. Goal.`) or ends in `?`/`!`: then it replaces the whole text. |
+| `rename <id> <new>` | Changes an open item's id (same kind, unused, not archived) and every `[[id]]` link in TASKS.md. |
+| `note <id> "<line>"` | Appends one `- <line>` to the item body. |
+| `body <id>` | Replaces the item body with stdin (header, `After:`, `Ref:` kept). |
+| `tick <id> <n \| text>` | Checks the n-th (or the matching) `- [ ]` box of a Needs-human item. |
+| `report "<what happened>" [--kind bug\|idea\|friction] [--cmd "<command>"]` | Appends one entry to the workflow `inbox.md` (section 7). Touches nothing in the project. |
+| `batch N [--lanes L,…] [--prep] [--model M] [--for 6h] [--mem 4G] [--dry-run]` · `batch --status` | Starts an unattended batch orchestrator (`claude -p`, auto mode, no prompts, prompt `templates/batch-prompt.md`) as a `wf res run` job with `CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS` = `--for`; it writes `out/wf-batch-<stamp>.md`. `--status` = this project's newest batch job + newest summary file. `--prep` = prep batch (`templates/prep-prompt.md`): sonnet workers write Done (+Model) for up to N tasks without Done, unclear → awaiting; none → nothing started. See `docs/orchestrator.md`. |
+| `init` | In a folder without `workflow.toml`: writes it, TASKS.md and the archive from the templates. |
+| `migrate [--write]` | One-time converter from the numbered format. Without `--write`: prints the diff, the id map and notes, changes nothing. `--write` also raises `format` in `workflow.toml`. |
+
+Every write command accepts `--dry-run` (prints a unified diff, writes nothing) and ends by
+comparing the full check before and after: a change that adds a problem is refused and
+nothing is written; problems that were there before do not block other edits.
+
+Details:
+
+- **Pick rule (`next`)**: first Pending item, in file order, that has no `(blocked: …)` and
+  whose `After:` ids are all in the archive. Items it skipped are listed in one line each
+  with the reason.
+- **Ref resolution (`ctx`, `next`)**: `path#anchor` → the section under the heading whose
+  slug or explicit `<a id>` equals the anchor, up to the next heading of the same or higher
+  level, cut at 80 lines with a `… (N more lines: <path>:<line>)` note. `path` alone → the
+  file's first heading and its `**Goal:**` line if it has one. With `[anchors]` set, an
+  index anchor also prints the spec section that carries the same `<a id>`.
+- **Context hint (`next`, `done`)**: with `CLAUDE_CODE_SESSION_ID` set, the last request's prompt size
+  (input + cache write + cache read) from the last 1 MB of `<CLAUDE_CONFIG_DIR|~/.claude>/projects/*/<id>.jsonl`;
+  over `ctx_hint` → last line `context ~Nk tokens (> Mk): ask the owner to /clear, then continue`. No id,
+  no transcript → nothing. Subagents share the id, so the line says it is about the main session.
+- **Ledgers (`next`)**: for each `<ledgers>/*/progress.md`: plan path from its first line,
+  done tasks, where to resume, count of `Ruling:` lines.
+- **Insert rule (`add`, `prio`, `move <section>`)**: position in the section = after the
+  last item of the same or higher priority, and after every item named in its `After:`.
+  Body is re-indented to two spaces. A block given to `add -` must carry an id and, for
+  tasks, a priority; a leading `N.` from the old format is refused with a message.
+- **Id from title (`add`)**: `t-` (`a-` for Awaiting) + slug of the title (the text up to
+  the first `. `), at most 40 characters, cut at a word; if taken (TASKS.md or archive), `-2`, `-3`, … is appended.
+- **Search ranking**: words are matched case-insensitively as whole words or word prefixes.
+  Score per hit = words matched × weight of where (id or title 5, doc heading 4, header
+  goal 3, `Ref:`/body/doc text 1); all words matched ranks before some. Ties: open tasks,
+  then docs, then archive. No index file; everything is read per call.
+- **Exact ids only** in write commands; a unique id prefix is accepted by read commands.
+  An unknown id prints the three nearest ids.
+- **`done`** refuses when the id is unknown or is a parent with open slices. On an `a-` id
+  it removes the item without an archive line (`-m` is optional there) and clears the
+  `(blocked: [[a-id]])` status of the tasks that waited on it, listing them.
+- **`check`** errors:
+  - duplicate id; id reused from the archive; bad id grammar; id prefix not allowed in its section
+  - task without priority or with an effort outside the vocabulary
+  - `[[id]]` that is neither in TASKS.md nor in the archive
+  - `(blocked: [[x]])` where `x` is not an open `a-` item
+  - `After:` cycle; Pending item placed before an open item it is `After:`
+  - `Ref:` path missing; anchor missing in that file
+  - old-format leftovers: numbered items
+  - item after a flush-left prose line inside a task section; malformed item header
+  - with `[anchors]`: task anchor without index heading; two index headings with the same
+    slug; index link to a missing file or anchor; explicit spec `<a id>` without index entry
+  - config: missing required key, unknown key, configured path that does not exist
+  In `docs` only `[[t-…]]` / `[[a-…]]` links are validated (other `[[x]]` may be the
+  project's own wiki links).
+  Warnings (exit stays 0): `#N` number refs in TASKS.md; task in progress with no branch/note; Awaiting item that no task
+  references and that is older than 30 days by `git blame`.
+- **Writes** are whole-file: read, change in memory, write to a temp file in the same
+  directory, rename. If the file changed on disk between read and write (mtime + size), the
+  command stops without writing. `done` writes the archive first, then TASKS.md; if the
+  second write fails it reports that the archive line must be removed.
+- **Errors** go to stderr as one line each, naming file and id; no tracebacks for expected
+  failures.
+
+## 6. Shared CLAUDE.md
+
+Text: `shared/CLAUDE.md` (the file is the authority). Sections: Style (concise everywhere;
+compaction summaries, specs, plans stay full and exact), Cold start, Loop, Done, TASKS.md,
+Context economy, Memory / CPU, Workflow problems.
+
+## 7. Improving the workflow without disturbing workers
+
+Problem: `wf.py` and the shared CLAUDE.md are live for every worker the moment they change.
+A worker that edits them mid-task, or a half-finished change, would hit all the others.
+
+**Report (any worker, any time)**
+- `wf report` appends to `inbox.md` with a single `O_APPEND` write, so parallel workers
+  never clash and nobody waits. Entry: date, project, kind, the text, the command if given,
+  wf version (git short hash).
+  `- 2026-09-29 proj-a friction: prio change needs two commands (cmd: wf prio …) @84decd1`
+- The worker then works around the problem and continues its own task. It does not edit
+  anything under `/projects/public/workflow` and does not wait for a fix.
+- `wf next` in any project prints one line when the inbox has entries, for the user's eyes:
+  `workflow inbox: 3 reports (triage: workflow session)`.
+
+**Triage and fix (a workflow session, started by the owner)**
+- The workflow's own tasks live in a wf project of their own (e.g. a private
+  `/projects/workflow`), whose CLAUDE.md holds the rules of this section. Cold start there:
+  each inbox entry becomes a task (`wf add`, duplicates merged) or is rejected with a line in the
+  archive; the entry is then deleted from `inbox.md`. `inbox.md` is git-ignored; tasks and
+  archive carry the record. Bookkeeping there changes nothing a worker runs.
+- Changes to `wf.py`, `wflib/`, `shared/CLAUDE.md` or templates happen in a git worktree
+  `/projects/public/workflow/.worktrees/<topic>` on a branch, never in the main tree. Workers keep
+  running the main tree's `master` meanwhile.
+- Release = fast-forward merge to `master` in the main tree after: tests green, and
+  `wf check` run with the new code against every project listed by `wf projects` gives the
+  same result as before the change (or differences that are the intended ones).
+- The user starts workflow sessions and sees the release; no worker does it on its own.
+
+**Compatibility rules for a release**
+- Commands and options are only added. Renaming or removing one keeps the old spelling
+  working for one release, printing a one-line notice with the new spelling.
+- A change to the TASKS.md format raises `format` and ships with a `wf migrate` step.
+  `wf` run in a project with an older `format` refuses write commands with
+  `TASKS format 1, wf needs 2: run wf migrate --write (idle project, one commit)`; read
+  commands keep working on the old format.
+- Shared CLAUDE.md text changes reach a worker at its next session start; a rule change
+  must therefore not make files written under the old rule invalid.
+- Every release adds a line to `CHANGES.md` (newest first): date, what changed, what a
+  project has to do, if anything.
+
+## 8. Testing
+
+- `wflib` functions are pure (text in, text out); unit tests use literal before/after texts
+  written by hand, never output of the code under test.
+- CLI tests run `wf.py` with `--project <tempdir>` on fixture projects: pick rule, skip
+  reasons, insert position with priority and `After:`, `done` refusals, every `check` error
+  once, write-conflict stop, `list` filters, `prio`/`move` placement and refusals, `status`,
+  `set`, `note`, `body`, `tick`, search ranking order on a fixed corpus, `report` from two
+  processes at once (both entries whole), `format` gate, migrate on a fixture with the known oddities (stray `N.`,
+  `N/M` titles, `After:` by title, prose after items).
+- Each new test is seen failing first, and again with the tested line broken by hand.