# Task 010: Research Fetch And S3 Manifest Development description: Implement explicit auditable web research with source retrieval metadata, source-section snapshots, object-storage uploads, and Postgres manifest records. ## Implementation Details - Backend endpoint: - `POST /api/articles/{article_id}/research/start` - `GET /api/articles/{article_id}/research` - Research starts only after approved plan. - Research uses an explicit search/fetch script, not only free-form agent browsing. - For discovered sources, store in object storage: - Relevant content sections. - Page metadata. - Search path to the article. - Agent/source-selection criteria. - Page `` metadata. - Server IP address. - Domain WHOIS owner when available. - In Postgres store only: - Research run manifest. - S3/object keys. - Source URLs. - Content hashes. - Artifact types. - Summary metadata required for UI and validation. - Retention is indefinite for v1. - Provide deterministic fake search/fetch fixtures for tests and demo. ## Public Interface - Editor starts research for an approved article plan. - UI shows research run status and discovered source summary. - Backend exposes manifest-linked evidence summaries. ## Acceptance Criteria - [x] TDD pre-requirement: before implementation, write one failing behavior test that starts research from an approved plan and verifies a manifest with object keys is produced; proceed one research behavior at a time and record evidence in `Result`. - [x] Research cannot start before plan approval. - [x] Research run creates an agent/job record and article status `RESEARCH_RUNNING`. - [x] Source-section artifacts are uploaded to object storage. - [x] Manifest records include object keys, hashes, URLs, artifact types, and metadata references. - [x] Re-running research creates a new manifest instead of overwriting old artifacts. - [x] UI can show source URL, title, domain, type, summary, retrieval timestamp, and artifact link. - [x] Demo stack can run with deterministic fake research results. ## Verification - Run research integration tests with fake search/fetch. - Verify object storage contains source-section artifact files. - Verify Postgres manifest records point to those files. - Run Docker Compose research smoke flow. ## Result - Status: Accepted. - TDD plan: Added `apps/backend/tests/integration/test_research_manifest_public_api.py` before implementation. The test builds an approved plan, starts research, verifies a manifest with object keys/hashes/source URLs/artifact types/metadata, checks uploaded local object files, lists research runs, and verifies rerun creates a new manifest prefix. - Red evidence: Initial run failed with `201 != 202`; `POST /api/articles/{article_id}/research/start` only returned a queued `RESEARCH` job and no manifest/artifacts. - Green evidence: Implemented deterministic fake research fetch, local/S3 object storage adapter, research manifest repository, `ResearchStartResponse`, `ResearchListResponse`, `GET /research`, manifest-linked detail output, and FSD frontend research page at `/articles/[articleId]/research`. - Refactor notes: Moved research-start behavior out of the Task 009 placeholder and made it produce an auditable manifest immediately for deterministic demo/tests. `OBJECT_STORAGE_LOCAL_ROOT` enables isolated integration tests; Docker Compose uses the existing S3-compatible MinIO env. - Verification output: `PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_research_manifest_public_api.py apps/backend/tests/integration/test_plan_generation_review_public_api.py apps/backend/tests/integration/test_schema_storage_contracts.py apps/backend/tests/contracts/test_generated_contract_artifacts.py` passed. `pnpm --filter @pipeline/frontend test:ui` passed. `pnpm --filter @pipeline/frontend typecheck` passed. `tests/smoke/research-flow.sh` and `tests/smoke/plan-flow.sh` passed with Docker Compose after Docker socket escalation.