content-factory/tasks/007-agent-job-queue-and-runner.md

Task 007: Agent Job Queue And Runner MVP

Development description: Implement the durable job path from backend to runner, including queued jobs, isolated workspaces, CLI command execution, log capture, output validation, retry, and cancellation basics.

Implementation Details

• Backend:
  • POST /api/agent-jobs/test-codex
  • GET /api/agent-jobs
  • GET /api/agent-jobs/{job_id}
  • POST /api/agent-jobs/{job_id}/retry
  • POST /api/agent-jobs/{job_id}/cancel
• Queue:
  • Use Redis-backed Celery, Dramatiq, or RQ.
  • Persist QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED.
• Runner:
  • Creates one workspace per job.
  • Writes structured input files.
  • Executes allowed command.
  • Captures stdout, stderr, exit code, duration.
  • Validates output JSON against the expected schema.
  • Uploads job artifacts to object storage where appropriate.
• Test Codex job:
  • Must support a demo-safe fake runner mode for CI/local tests.
  • Real Codex CLI execution is configurable for environments with runner auth.

Public Interface

• Admin can enqueue a test runner job.
• UI can show job status and logs.
• Failed jobs can be retried where allowed.

Acceptance Criteria

• TDD pre-requirement: before implementation, write one failing backend-to-fake-runner integration test through public job APIs; implement minimal queue/runner behavior to make it green and record evidence in Result.
• Backend can create and persist an agent job.
• Runner claims a queued job and marks it running.
• Runner stores stdout, stderr, exit code, duration, and final status.
• Runner validates output JSON and marks schema failures as FAILED_SCHEMA_VALIDATION.
• Cancelled jobs do not update article domain state.
• Retry creates a new attempt or new job with traceable parent metadata.
• Demo stack works without real Codex credentials by using fake runner mode.

Verification

• Run backend/runner integration tests with fake runner.
• Run a Docker Compose test job from Admin UI or API.
• Verify logs and status in UI.

Result

• Status: Pending execution.
• TDD plan: To be filled during execution.
• Red evidence: To be filled during execution.
• Green evidence: To be filled during execution.
• Refactor notes: To be filled during execution.
• Verification output: To be filled during execution.