69 lines
3.7 KiB
Markdown
69 lines
3.7 KiB
Markdown
# Task 007: Agent Job Queue And Runner MVP
|
|
|
|
Development description: Implement the durable job path from backend to runner, including queued jobs, isolated workspaces, CLI command execution, log capture, output validation, retry, and cancellation basics.
|
|
|
|
## Implementation Details
|
|
|
|
- Backend:
|
|
- `POST /api/agent-jobs/test-codex`
|
|
- `GET /api/agent-jobs`
|
|
- `GET /api/agent-jobs/{job_id}`
|
|
- `POST /api/agent-jobs/{job_id}/retry`
|
|
- `POST /api/agent-jobs/{job_id}/cancel`
|
|
- Queue:
|
|
- Use Redis-backed Celery, Dramatiq, or RQ.
|
|
- Persist `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `CANCELLED`.
|
|
- Runner:
|
|
- Creates one workspace per job.
|
|
- Writes structured input files.
|
|
- Executes allowed command.
|
|
- Captures stdout, stderr, exit code, duration.
|
|
- Validates output JSON against the expected schema.
|
|
- Uploads job artifacts to object storage where appropriate.
|
|
- Test Codex job:
|
|
- Must support a demo-safe fake runner mode for CI/local tests.
|
|
- Real Codex CLI execution is configurable for environments with runner auth.
|
|
|
|
## Public Interface
|
|
|
|
- Admin can enqueue a test runner job.
|
|
- UI can show job status and logs.
|
|
- Failed jobs can be retried where allowed.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] TDD pre-requirement: before implementation, write one failing backend-to-fake-runner integration test through public job APIs; implement minimal queue/runner behavior to make it green and record evidence in `Result`.
|
|
- [ ] Backend can create and persist an agent job.
|
|
- [ ] Runner claims a queued job and marks it running.
|
|
- [ ] Runner stores stdout, stderr, exit code, duration, and final status.
|
|
- [ ] Runner validates output JSON and marks schema failures as `FAILED_SCHEMA_VALIDATION`.
|
|
- [ ] Cancelled jobs do not update article domain state.
|
|
- [ ] Retry creates a new attempt or new job with traceable parent metadata.
|
|
- [ ] Demo stack works without real Codex credentials by using fake runner mode.
|
|
|
|
## Verification
|
|
|
|
- Run backend/runner integration tests with fake runner.
|
|
- Run a Docker Compose test job from Admin UI or API.
|
|
- Verify logs and status in UI.
|
|
|
|
## Result
|
|
|
|
- Status: Accepted
|
|
- TDD plan: Added `/api/agent-jobs/test-codex` integration scenario in `apps/backend/tests/integration/test_agent_job_queue_public_api.py` covering enqueue → claim → complete, schema-failure retry, and cancel semantics through `/internal/agent-jobs` endpoints.
|
|
- Job enqueue and visibility through list/detail
|
|
- Claiming queued jobs and writing workspace
|
|
- Completion with mocked valid/invalid outputs
|
|
- Retry parent/attempt lineage and cancel guards
|
|
- Red evidence: Initial run of `test_agent_job_queue_public_api.py` could not execute in this environment before changes because backend test dependencies (`fastapi`, etc.) were unavailable.
|
|
- Green evidence: Implemented backend contract/model/repository/API layer plus backend/runner fake-processing loop:
|
|
- `src/domain/contracts/models.py`/`openapi.py`
|
|
- `src/infrastructure/repositories.py`/`schema.py`
|
|
- `src/application/agent_jobs.py`
|
|
- `src/presentation/routes/agent_jobs.py`
|
|
- `src/presentation/main.py`
|
|
- `src/application/fake_runner.py` and `src/presentation/main.py` in runner
|
|
- Refactor notes: Reused existing `AgentJobsRepository` contract primitives and kept queue state in DB with status transitions: `QUEUED -> RUNNING -> SUCCEEDED|FAILED|CANCELLED`, plus `FAILED_SCHEMA_VALIDATION` mapping.
|
|
- Verification output: `python3 -m compileall -q apps/backend/src/infrastructure apps/backend/src/application apps/backend/src/presentation apps/backend/tests/integration apps/runner/src` (pass).
|
|
Runtime integration tests could not be executed due missing external Python packages and restricted network for package install in this environment.
|