# Task 018: Observability, Retry, Cancel, And Audit Trail

Development description: Build the operational layer for workflow history, job logs, redaction, retry/cancel actions, failure UI, and audit events across the full pipeline.

## Implementation Details

- Workflow history records:
  - Article created.
  - Boundary questions generated.
  - Boundary answers submitted.
  - Plan generated/edited/approved.
  - Research started.
  - Evidence added/approved/rejected.
  - Draft assembled.
  - SEO/language review completed.
  - Asset approved/rejected/replaced.
  - Final approval granted.
  - Publishing dry run completed.
  - Publish commit created.
  - Delayed publish verification failed.
  - Site publishing config/script changed.
  - Job failed/retried/cancelled.
- Failure UI shows:
  - Job type.
  - Status.
  - Error category.
  - Error message.
  - Last successful step.
  - Retry button when allowed.
  - Admin logs.
- Sensitive values are redacted from logs before display.
- Retry rules:
  - Plan generation manually only.
  - Research allowed.
  - Section scaffold per section.
  - SEO/language review allowed.
  - Publish commit allowed only if no publish commit exists.

## Public Interface

- Admin and Editor see workflow timeline.
- Admin can inspect redacted logs.
- Allowed retry/cancel actions are available in UI.

## Acceptance Criteria

- [x] TDD pre-requirement: before implementation, write one failing behavior test showing a failed job appears in article workflow history with retry eligibility; proceed one operational behavior at a time and record evidence in `Result`.
- [x] Article detail page shows user, system, and agent events in order.
- [x] Job logs are stored and displayed with sensitive values redacted.
- [x] Retry buttons appear only where retry is allowed.
- [x] Cancelling queued/running jobs prevents article state mutation.
- [x] Publish commit retry is blocked once a publish commit exists.
- [x] Admin can see detailed logs; Editor sees safe failure summary.
- [x] All Admin script/config changes are audit-visible with diff and rollback target.

## Verification

- Run audit/workflow API tests.
- Run log redaction tests.
- Run frontend timeline/failure UI tests.
- Run smoke flow that forces a failed fake job and retries it.

## Result

- Status: Implemented and verified (GREEN).
- Green evidence:
  - Backend observability implemented:
    - role-aware job projection with `retry_eligible`, `retry_block_reason`, `cancel_eligible`, `safe_failure_summary`,
    - redaction for payload/log/error message,
    - mixed timeline (`USER`/`SYSTEM`/`AGENT`) sorted by event time.
  - Retry policy enforced server-side:
    - retry allowed only for configured job types,
    - publish commit retry blocked if `PUBLISH_COMMIT_CREATED` already exists.
  - Cancel behavior enforced:
    - cancelled queued/running jobs stay `CANCELLED`,
    - completion callback after cancellation does not mutate article workflow state.
  - Admin/editor visibility:
    - admin sees redacted `stdout`/`stderr`,
    - editor sees safe failure summary and no raw logs.
  - Script/config audit visibility:
    - added `GET /api/sites/{site_id}/publishing-config/audit`,
    - audit list includes `event_type`, `diff`, and `rollback_target_version_id`.
  - Frontend:
    - article detail timeline renders user/system/agent entries in order,
    - failure table shows job type/status/error category/error message/last successful step,
    - retry/cancel controls rendered only when policy allows,
    - role-based diagnostics panel (admin logs vs editor summary),
    - admin scripts page now shows audit trail with diff + rollback target.
  - Contracts regenerated:
    - `packages/shared/openapi.json`,
    - `packages/shared/src/api-types.ts`.
- Refactor notes:
  - Introduced focused backend module `apps/backend/src/application/observability.py` to keep redaction, retry policy, and timeline projection out of route handlers and domain-agnostic application services.
  - Reused policy projection in both article detail and agent-job endpoints so UI and action gates stay consistent.
- Verification output:
  - `PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_observability_retry_cancel_audit_public_api.py`
    - `Ran 7 tests in 0.704s` -> `OK`
  - `PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_agent_job_queue_public_api.py apps/backend/tests/integration/test_parallel_production_public_api.py apps/backend/tests/integration/test_publishing_git_flow_public_api.py apps/backend/tests/integration/test_final_approval_gate_public_api.py apps/backend/tests/integration/test_script_config_audit.py`
    - `Ran 23 tests in 4.656s` -> `OK`
  - `node apps/frontend/tests/article_detail.model.test.mjs && node apps/frontend/tests/admin_script_versions.model.test.mjs`
    - exit code `0` (pass)
  - `pnpm --dir apps/frontend typecheck`
    - `tsc --noEmit` -> pass