Generation Pipeline Framework

Role

The generation pipeline framework runs the artifact-producing phases defined by Report Generation. It owns task ordering, dependency readiness, concurrency limits, and common artifact checks. It does not own evidence extraction, project synthesis, or daily synthesis semantics.

Generation remains artifact-first: every phase invocation consumes the prepared workspace plus durable prerequisite artifacts, writes its own durable outputs, and returns success only after those outputs exist.

Task Model

The framework models phase invocations as task nodes:

This is a real DAG, not only three coarse phase barriers. Project synthesis for one project depends only on that project’s evidence tasks. Daily synthesis depends on all project synthesis tasks.

APIs

TaskSpec records the stable task id, kind, project/session scope, dependencies, expected inputs, and expected outputs. GenerationPlan is the immutable task graph built from the prepared workspace indexes.

Generation workflow APIs take a prepared workspace path. CLI and preparation code own date and reports-root resolution and the mapping to <reports-root>/work/<YYYY-MM-DD>; the generation package only inspects the workspace and its durable artifacts. The reports root is resolved once at the CLI boundary by prompt_diary.config.resolve_reports_root (--reports-root over PROMPT_DIARY_HOME over the stored config over the per-user data directory, the last supplied by prompt_diary.paths.platform_data_dir).

Dependencies normally require successful prerequisite tasks. Project synthesis is the exception: it waits for all evidence extraction attempts in that project to finish, but checks that each expected evidence card exists before starting. The project runner then validates every card’s complete turn coverage. An extraction task failure can continue into synthesis only if its durable card is nevertheless complete and valid; partial cards remain checkpoints for resumed extraction.

PhaseRunner is the narrow phase execution protocol:

async def run(*, workspace_path: Path, task: TaskSpec) -> TaskResult: ...

Each real phase implementation should live in its phase package and implement this protocol. The runner may use Codex, MCP tools, deterministic code, or mocks. The framework calls it only after dependencies are complete.

The three agent phase runners hold an injected AgentSessionFactory but do not own backend lifecycle. Backend ownership lives at the run scope: GenerateWorkspaceWorkflow enters one shared factory once per run (inside asyncio.run), and each agent assignment mints its own conversation off that shared backend via factory.runner(config). The assignment owns that conversation until its artifact-aware retry loop finishes, then closes it through AgentRunner.aclose(). The composition root cmds/generate.py::build_generation_workflow() constructs one CodexAgentSessionFactory, wraps it with the Prompt Diary content-language injector, passes the wrapper to the three agent phase runners, and sets it as the workflow’s agent_factory; the rendering runner is deterministic and takes no agent factory. The wrapper writes the generated workspace AGENTS.md and appends the same rendered language norm to every AgentConfig.developer_instructions before minting a conversation. GeneratePipelineRunner itself is agent-agnostic — it schedules tasks and calls PhaseRunner.run; backend and agent wiring are the workflow’s concern.

A phase runner therefore does not need to be an async context manager to obtain its backend: the shared AgentSessionFactory is entered once at the workflow scope, above the pipeline. The pipeline still enters any phase runner that is an async context manager (once per run), but that mechanism now serves only a runner’s own additional resources, not the agent backend.

Model and reasoning choices live in the bundled src/prompt_diary/generate/agent-settings.json. The agent phase runners load typed settings from this package resource and pass an explicit model and reasoning effort to each conversation. Daily synthesis selects settings separately for project summaries, the report title, engagement, and team learning. These are internal package settings, with no CLI, environment, or user-config overrides.

Each phase supplies a stable task contract as Codex base instructions and keeps variable input in its user message. Invocation-local settings suppress memory use and generation, skill catalogs, plugins, apps, delegation instructions, and project/environment context. CODEX_HOME, credentials, global AGENTS guidance, and persisted session logs remain inherited. Language rules are explicit developer instructions. These context controls were verified with Codex CLI 0.153.4; some are version-specific. Each phase directly exposes only its required Prompt Diary MCP tools, including on models that otherwise defer tool schemas behind code-mode discovery.

Generation uses the persisted client identity prompt_diary, allowing preparation to exclude its sessions independently of the report directory while retaining normal Codex logs for debugging. Stable prefixes can improve cache reuse; the pipeline does not fork template conversations or assume that provider caching guarantees a particular saving.

GenerateWorkspaceWorkflow is the shared workspace executor for both the full pipeline and one standalone phase task. run_generation_task is the lower-level task API used after declared prerequisites exist, which keeps phase development and debugging independent from the full pipeline.

GeneratePipelineRunner runs a full GenerationPlan. It schedules ready tasks, applies per-kind concurrency limits, marks dependents blocked after failed prerequisites, and validates that a successful task produced its declared outputs.

The scheduler does not retry failed tasks. Codex-backed phase runners own same-process agent retry inside a task through generate/agent_retry.py: they keep the current AgentRunner, re-read durable artifacts after each successful or failed turn, and send a phase-specific resume prompt when the artifact shows more work is needed. The default policy permits three consecutive no-progress attempts with exponential backoff from 1s up to 60s. If that budget is exhausted, the phase returns a failed task with an agent made no progress ... error. The helper closes the conversation after assignment completion, failure, or cancellation, including failures during artifact inspection or resume-prompt construction. It retains the conversation throughout retries. Deterministic rendering and non-agent failures remain outside this helper.

The Codex runner retains the active turn handle and consumes its public event stream until a terminal notification confirms completion, interruption, or failure. A timeout or caller cancellation sends an interrupt and waits for terminal confirmation before releasing the runner for another turn. Cleanup and backend shutdown have bounded grace periods; cancellation is propagated after cleanup. If termination cannot be confirmed, the shared backend is closed and disabled, which can also stop other tasks using it. Cached conversation threads cannot restart a disabled backend. A confirmed failed turn remains eligible for ordinary artifact-aware retry. Closing a Codex conversation unsubscribes its SDK thread and confirms that the server unloaded it before returning. Invocation-local thread_unload_delay_secs=0 lets completed assignments release their MCP and other runtime resources immediately while the backend remains available to other conversations. Persisted transcripts remain available for debugging and are not archived. If unload cannot be confirmed within the cleanup grace period, the shared backend is closed and disabled.

A full pipeline run succeeds when terminal deliverables succeed. Non-terminal tolerated failures remain visible on the run result. Project synthesis verifies that every prerequisite evidence card is complete; an extraction failure that leaves only partial evidence therefore blocks that project.

CLI

report generate runs the full pipeline for a target date, preparing the workspace first when it is missing.

Standalone phase commands require an existing prepared workspace and run one task after checking its declared prerequisites:

report generate evidence --date YYYY-MM-DD --project-key <project_key> --session-ref S0001
report generate project --date YYYY-MM-DD --project-key <project_key>
report generate daily --date YYYY-MM-DD
report generate render --date YYYY-MM-DD
report generate render --date YYYY-MM-DD --notion

The phase commands do not rerun earlier phases or prepare missing workspaces. They are development and repair entrypoints for the phase boundary rule. generate render writes the views from an existing daily-report.json; generate render --notion renders then publishes to Notion.

Evidence Extraction Runner

The evidence extraction phase runner creates a fresh agent conversation for each uncommitted source turn, in indexed order. Each receives the stable extractor contract as base instructions and the current turn’s scope in a separate user message. An optional previous_context locator supplies the preceding human turn’s start_line and end_line in the same prepared file, even when it is unindexed inherited history or falls outside the report date. It supplies no turn_ref and is used for narrow MCP context reads behind continuations or corrections; only current-turn lines may support citations. For older prepared workspaces without the locator, the runner falls back to the preceding indexed turn. Earlier extraction conversations and their transcript reads are not carried forward.

After each turn the runner verifies the result by reading the evidence card from the workspace directly. It never trusts the assistant’s text response. An uncommitted turn — one where the card on disk does not reflect the expected turn — is retried on the same agent conversation until that turn is committed or the no-progress budget is exhausted. The retry counter is scoped to the current assigned turn and resets when the runner advances to the next committed turn. Retries send short assignment reminders on the same conversation without appending the original prompt again. The helper closes that conversation before the runner advances to the next source turn.

At task start the runner validates any existing evidence card against the current prepared session. A complete card requires no agent calls. A valid partial card is retained, and only missing turns are extracted; committed chains also survive a later turn’s failure. Invalid cards are reset. Every attempt checks the same schema, scope, and citation rules before accepting a committed turn. Project synthesis requires complete evidence cards, so partial evidence cannot silently become an evidence gap. prepare --force replaces the whole workspace and therefore invalidates its generated artifacts.

The runner builds a workspace-aware agent factory once per run. For the Codex backend the factory registers the package MCP server (report mcp serve) with the prepared workspace path in the PROMPT_DIARY_WORKSPACE environment variable. A Codex-spawned stdio MCP server does not inherit the calling thread’s working directory, so the MCP write_evidence tool resolves its workspace from that variable, falling back to cwd. The agent runs non-interactively (approval_mode="auto_review", sandbox="workspace-write") using the system codex binary on PATH.

Project And Daily Agent Retry

Project synthesis uses the same helper with the current uncovered-turn count as its progress marker. A retry continues on the same runner with the current uncovered-turn list; progress means that list strictly shrinks, and completion means every indexed turn is covered by valid work items. A valid partial project-synthesis.json survives failure and restart. A restarted conversation receives the committed refs and coverage together with evidence for only the remaining turns, and continues with unused work-item refs. A complete envelope needs no agent calls; an invalid envelope is reset at task start.

Daily synthesis still uses one fresh agent conversation per pass: each project summary, report title, engagement assessment, and team-learning pass gets its own runner. A pass retries on that same runner until its target slot is written in daily-report.json or the no-progress budget is exhausted. If a turn fails after writing the slot, the artifact inspection treats the pass as complete. A retry names the unwritten slot without repeating the pass’s source content.

Progress

The scheduler emits TaskStarted/TaskFinished events and threads a ProgressReporter into each phase runner’s run(...); the evidence runner emits TurnAdvanced per turn. See Progress Reporting.

Boundaries

The framework checks only generic output existence. Phase-local validation belongs to the phase runner before it returns success. For example, evidence extraction should validate evidence card structure, daily synthesis should validate daily-report.json, and the rendering phase should validate the rendered views.

An absent evidence card is a missing prerequisite artifact and prevents the project task from starting. An incomplete or invalid card fails the project runner’s prerequisite checks. Other failed dependencies block their dependent tasks.