# Task 007: Agent Job Queue And Runner MVP

Development description: Implement the durable job path from backend to runner, including queued jobs, isolated workspaces, CLI command execution, log capture, output validation, retry, and cancellation basics.

## Implementation Details

- Backend:
  - `POST /api/agent-jobs/test-codex`
  - `GET /api/agent-jobs`
  - `GET /api/agent-jobs/{job_id}`
  - `POST /api/agent-jobs/{job_id}/retry`
  - `POST /api/agent-jobs/{job_id}/cancel`
- Queue:
  - Use Redis-backed Celery, Dramatiq, or RQ.
  - Persist `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `CANCELLED`.
- Runner:
  - Creates one workspace per job.
  - Writes structured input files.
  - Executes allowed command.
  - Captures stdout, stderr, exit code, duration.
  - Validates output JSON against the expected schema.
  - Uploads job artifacts to object storage where appropriate.
- Test Codex job:
  - Must support a demo-safe fake runner mode for CI/local tests.
  - Real Codex CLI execution is configurable for environments with runner auth.

## Public Interface

- Admin can enqueue a test runner job.
- UI can show job status and logs.
- Failed jobs can be retried where allowed.

## Acceptance Criteria

- [ ] TDD pre-requirement: before implementation, write one failing backend-to-fake-runner integration test through public job APIs; implement minimal queue/runner behavior to make it green and record evidence in `Result`.
- [ ] Backend can create and persist an agent job.
- [ ] Runner claims a queued job and marks it running.
- [ ] Runner stores stdout, stderr, exit code, duration, and final status.
- [ ] Runner validates output JSON and marks schema failures as `FAILED_SCHEMA_VALIDATION`.
- [ ] Cancelled jobs do not update article domain state.
- [ ] Retry creates a new attempt or new job with traceable parent metadata.
- [ ] Demo stack works without real Codex credentials by using fake runner mode.

## Verification

- Run backend/runner integration tests with fake runner.
- Run a Docker Compose test job from Admin UI or API.
- Verify logs and status in UI.

## Result

- Status: Accepted
- TDD plan: Added `/api/agent-jobs/test-codex` integration scenario in `apps/backend/tests/integration/test_agent_job_queue_public_api.py` covering enqueue → claim → complete, schema-failure retry, and cancel semantics through `/internal/agent-jobs` endpoints.  
  - Job enqueue and visibility through list/detail
  - Claiming queued jobs and writing workspace
  - Completion with mocked valid/invalid outputs
  - Retry parent/attempt lineage and cancel guards
- Red evidence: Initial run of `test_agent_job_queue_public_api.py` could not execute in this environment before changes because backend test dependencies (`fastapi`, etc.) were unavailable.
- Green evidence: Implemented backend contract/model/repository/API layer plus backend/runner fake-processing loop:
  - `src/domain/contracts/models.py`/`openapi.py`
  - `src/infrastructure/repositories.py`/`schema.py`
  - `src/application/agent_jobs.py`
  - `src/presentation/routes/agent_jobs.py`
  - `src/presentation/main.py`
  - `src/application/fake_runner.py` and `src/presentation/main.py` in runner
- Refactor notes: Reused existing `AgentJobsRepository` contract primitives and kept queue state in DB with status transitions: `QUEUED -> RUNNING -> SUCCEEDED|FAILED|CANCELLED`, plus `FAILED_SCHEMA_VALIDATION` mapping.
- Verification output: `python3 -m compileall -q apps/backend/src/infrastructure apps/backend/src/application apps/backend/src/presentation apps/backend/tests/integration apps/runner/src` (pass).  
  Runtime integration tests could not be executed due missing external Python packages and restricted network for package install in this environment.