content-factory/tasks/018-observability-retry-cancel-audit.md

Task 018: Observability, Retry, Cancel, And Audit Trail

Development description: Build the operational layer for workflow history, job logs, redaction, retry/cancel actions, failure UI, and audit events across the full pipeline.

Implementation Details

• Workflow history records:
  • Article created.
  • Boundary questions generated.
  • Boundary answers submitted.
  • Plan generated/edited/approved.
  • Research started.
  • Evidence added/approved/rejected.
  • Draft assembled.
  • SEO/language review completed.
  • Asset approved/rejected/replaced.
  • Final approval granted.
  • Publishing dry run completed.
  • Publish commit created.
  • Delayed publish verification failed.
  • Site publishing config/script changed.
  • Job failed/retried/cancelled.
• Failure UI shows:
  • Job type.
  • Status.
  • Error category.
  • Error message.
  • Last successful step.
  • Retry button when allowed.
  • Admin logs.
• Sensitive values are redacted from logs before display.
• Retry rules:
  • Plan generation manually only.
  • Research allowed.
  • Section scaffold per section.
  • SEO/language review allowed.
  • Publish commit allowed only if no publish commit exists.

Public Interface

• Admin and Editor see workflow timeline.
• Admin can inspect redacted logs.
• Allowed retry/cancel actions are available in UI.

Acceptance Criteria

• TDD pre-requirement: before implementation, write one failing behavior test showing a failed job appears in article workflow history with retry eligibility; proceed one operational behavior at a time and record evidence in Result.
• Article detail page shows user, system, and agent events in order.
• Job logs are stored and displayed with sensitive values redacted.
• Retry buttons appear only where retry is allowed.
• Cancelling queued/running jobs prevents article state mutation.
• Publish commit retry is blocked once a publish commit exists.
• Admin can see detailed logs; Editor sees safe failure summary.
• All Admin script/config changes are audit-visible with diff and rollback target.

Verification

• Run audit/workflow API tests.
• Run log redaction tests.
• Run frontend timeline/failure UI tests.
• Run smoke flow that forces a failed fake job and retries it.

Result

• Status: Implemented and verified (GREEN).
• Green evidence:
  • Backend observability implemented:
    • role-aware job projection with retry_eligible, retry_block_reason, cancel_eligible, safe_failure_summary,
    • redaction for payload/log/error message,
    • mixed timeline (USER/SYSTEM/AGENT) sorted by event time.
  • Retry policy enforced server-side:
    • retry allowed only for configured job types,
    • publish commit retry blocked if PUBLISH_COMMIT_CREATED already exists.
  • Cancel behavior enforced:
    • cancelled queued/running jobs stay CANCELLED,
    • completion callback after cancellation does not mutate article workflow state.
  • Admin/editor visibility:
    • admin sees redacted stdout/stderr,
    • editor sees safe failure summary and no raw logs.
  • Script/config audit visibility:
    • added GET /api/sites/{site_id}/publishing-config/audit,
    • audit list includes event_type, diff, and rollback_target_version_id.
  • Frontend:
    • article detail timeline renders user/system/agent entries in order,
    • failure table shows job type/status/error category/error message/last successful step,
    • retry/cancel controls rendered only when policy allows,
    • role-based diagnostics panel (admin logs vs editor summary),
    • admin scripts page now shows audit trail with diff + rollback target.
  • Contracts regenerated:
    • packages/shared/openapi.json,
    • packages/shared/src/api-types.ts.
• Refactor notes:
  • Introduced focused backend module apps/backend/src/application/observability.py to keep redaction, retry policy, and timeline projection out of route handlers and domain-agnostic application services.
  • Reused policy projection in both article detail and agent-job endpoints so UI and action gates stay consistent.
• Verification output:
  • PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_observability_retry_cancel_audit_public_api.py
    • Ran 7 tests in 0.704s -> OK
  • PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_agent_job_queue_public_api.py apps/backend/tests/integration/test_parallel_production_public_api.py apps/backend/tests/integration/test_publishing_git_flow_public_api.py apps/backend/tests/integration/test_final_approval_gate_public_api.py apps/backend/tests/integration/test_script_config_audit.py
    • Ran 23 tests in 4.656s -> OK
  • node apps/frontend/tests/article_detail.model.test.mjs && node apps/frontend/tests/admin_script_versions.model.test.mjs
    • exit code 0 (pass)
  • pnpm --dir apps/frontend typecheck
    • tsc --noEmit -> pass