104 lines
4.9 KiB
Markdown
104 lines
4.9 KiB
Markdown
# Task 018: Observability, Retry, Cancel, And Audit Trail
|
|
|
|
Development description: Build the operational layer for workflow history, job logs, redaction, retry/cancel actions, failure UI, and audit events across the full pipeline.
|
|
|
|
## Implementation Details
|
|
|
|
- Workflow history records:
|
|
- Article created.
|
|
- Boundary questions generated.
|
|
- Boundary answers submitted.
|
|
- Plan generated/edited/approved.
|
|
- Research started.
|
|
- Evidence added/approved/rejected.
|
|
- Draft assembled.
|
|
- SEO/language review completed.
|
|
- Asset approved/rejected/replaced.
|
|
- Final approval granted.
|
|
- Publishing dry run completed.
|
|
- Publish commit created.
|
|
- Delayed publish verification failed.
|
|
- Site publishing config/script changed.
|
|
- Job failed/retried/cancelled.
|
|
- Failure UI shows:
|
|
- Job type.
|
|
- Status.
|
|
- Error category.
|
|
- Error message.
|
|
- Last successful step.
|
|
- Retry button when allowed.
|
|
- Admin logs.
|
|
- Sensitive values are redacted from logs before display.
|
|
- Retry rules:
|
|
- Plan generation manually only.
|
|
- Research allowed.
|
|
- Section scaffold per section.
|
|
- SEO/language review allowed.
|
|
- Publish commit allowed only if no publish commit exists.
|
|
|
|
## Public Interface
|
|
|
|
- Admin and Editor see workflow timeline.
|
|
- Admin can inspect redacted logs.
|
|
- Allowed retry/cancel actions are available in UI.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] TDD pre-requirement: before implementation, write one failing behavior test showing a failed job appears in article workflow history with retry eligibility; proceed one operational behavior at a time and record evidence in `Result`.
|
|
- [x] Article detail page shows user, system, and agent events in order.
|
|
- [x] Job logs are stored and displayed with sensitive values redacted.
|
|
- [x] Retry buttons appear only where retry is allowed.
|
|
- [x] Cancelling queued/running jobs prevents article state mutation.
|
|
- [x] Publish commit retry is blocked once a publish commit exists.
|
|
- [x] Admin can see detailed logs; Editor sees safe failure summary.
|
|
- [x] All Admin script/config changes are audit-visible with diff and rollback target.
|
|
|
|
## Verification
|
|
|
|
- Run audit/workflow API tests.
|
|
- Run log redaction tests.
|
|
- Run frontend timeline/failure UI tests.
|
|
- Run smoke flow that forces a failed fake job and retries it.
|
|
|
|
## Result
|
|
|
|
- Status: Implemented and verified (GREEN).
|
|
- Green evidence:
|
|
- Backend observability implemented:
|
|
- role-aware job projection with `retry_eligible`, `retry_block_reason`, `cancel_eligible`, `safe_failure_summary`,
|
|
- redaction for payload/log/error message,
|
|
- mixed timeline (`USER`/`SYSTEM`/`AGENT`) sorted by event time.
|
|
- Retry policy enforced server-side:
|
|
- retry allowed only for configured job types,
|
|
- publish commit retry blocked if `PUBLISH_COMMIT_CREATED` already exists.
|
|
- Cancel behavior enforced:
|
|
- cancelled queued/running jobs stay `CANCELLED`,
|
|
- completion callback after cancellation does not mutate article workflow state.
|
|
- Admin/editor visibility:
|
|
- admin sees redacted `stdout`/`stderr`,
|
|
- editor sees safe failure summary and no raw logs.
|
|
- Script/config audit visibility:
|
|
- added `GET /api/sites/{site_id}/publishing-config/audit`,
|
|
- audit list includes `event_type`, `diff`, and `rollback_target_version_id`.
|
|
- Frontend:
|
|
- article detail timeline renders user/system/agent entries in order,
|
|
- failure table shows job type/status/error category/error message/last successful step,
|
|
- retry/cancel controls rendered only when policy allows,
|
|
- role-based diagnostics panel (admin logs vs editor summary),
|
|
- admin scripts page now shows audit trail with diff + rollback target.
|
|
- Contracts regenerated:
|
|
- `packages/shared/openapi.json`,
|
|
- `packages/shared/src/api-types.ts`.
|
|
- Refactor notes:
|
|
- Introduced focused backend module `apps/backend/src/application/observability.py` to keep redaction, retry policy, and timeline projection out of route handlers and domain-agnostic application services.
|
|
- Reused policy projection in both article detail and agent-job endpoints so UI and action gates stay consistent.
|
|
- Verification output:
|
|
- `PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_observability_retry_cancel_audit_public_api.py`
|
|
- `Ran 7 tests in 0.704s` -> `OK`
|
|
- `PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_agent_job_queue_public_api.py apps/backend/tests/integration/test_parallel_production_public_api.py apps/backend/tests/integration/test_publishing_git_flow_public_api.py apps/backend/tests/integration/test_final_approval_gate_public_api.py apps/backend/tests/integration/test_script_config_audit.py`
|
|
- `Ran 23 tests in 4.656s` -> `OK`
|
|
- `node apps/frontend/tests/article_detail.model.test.mjs && node apps/frontend/tests/admin_script_versions.model.test.mjs`
|
|
- exit code `0` (pass)
|
|
- `pnpm --dir apps/frontend typecheck`
|
|
- `tsc --noEmit` -> pass
|