3.9 KiB
3.9 KiB
Task 010: Research Fetch And S3 Manifest
Development description: Implement explicit auditable web research with source retrieval metadata, source-section snapshots, object-storage uploads, and Postgres manifest records.
Implementation Details
- Backend endpoint:
POST /api/articles/{article_id}/research/startGET /api/articles/{article_id}/research
- Research starts only after approved plan.
- Research uses an explicit search/fetch script, not only free-form agent browsing.
- For discovered sources, store in object storage:
- Relevant content sections.
- Page metadata.
- Search path to the article.
- Agent/source-selection criteria.
- Page
<head>metadata. - Server IP address.
- Domain WHOIS owner when available.
- In Postgres store only:
- Research run manifest.
- S3/object keys.
- Source URLs.
- Content hashes.
- Artifact types.
- Summary metadata required for UI and validation.
- Retention is indefinite for v1.
- Provide deterministic fake search/fetch fixtures for tests and demo.
Public Interface
- Editor starts research for an approved article plan.
- UI shows research run status and discovered source summary.
- Backend exposes manifest-linked evidence summaries.
Acceptance Criteria
- TDD pre-requirement: before implementation, write one failing behavior test that starts research from an approved plan and verifies a manifest with object keys is produced; proceed one research behavior at a time and record evidence in
Result. - Research cannot start before plan approval.
- Research run creates an agent/job record and article status
RESEARCH_RUNNING. - Source-section artifacts are uploaded to object storage.
- Manifest records include object keys, hashes, URLs, artifact types, and metadata references.
- Re-running research creates a new manifest instead of overwriting old artifacts.
- UI can show source URL, title, domain, type, summary, retrieval timestamp, and artifact link.
- Demo stack can run with deterministic fake research results.
Verification
- Run research integration tests with fake search/fetch.
- Verify object storage contains source-section artifact files.
- Verify Postgres manifest records point to those files.
- Run Docker Compose research smoke flow.
Result
- Status: Accepted.
- TDD plan: Added
apps/backend/tests/integration/test_research_manifest_public_api.pybefore implementation. The test builds an approved plan, starts research, verifies a manifest with object keys/hashes/source URLs/artifact types/metadata, checks uploaded local object files, lists research runs, and verifies rerun creates a new manifest prefix. - Red evidence: Initial run failed with
201 != 202;POST /api/articles/{article_id}/research/startonly returned a queuedRESEARCHjob and no manifest/artifacts. - Green evidence: Implemented deterministic fake research fetch, local/S3 object storage adapter, research manifest repository,
ResearchStartResponse,ResearchListResponse,GET /research, manifest-linked detail output, and FSD frontend research page at/articles/[articleId]/research. - Refactor notes: Moved research-start behavior out of the Task 009 placeholder and made it produce an auditable manifest immediately for deterministic demo/tests.
OBJECT_STORAGE_LOCAL_ROOTenables isolated integration tests; Docker Compose uses the existing S3-compatible MinIO env. - Verification output:
PYTHONPATH=/private/tmp/pupline-backend-deps python3 -m unittest apps/backend/tests/integration/test_research_manifest_public_api.py apps/backend/tests/integration/test_plan_generation_review_public_api.py apps/backend/tests/integration/test_schema_storage_contracts.py apps/backend/tests/contracts/test_generated_contract_artifacts.pypassed.pnpm --filter @pipeline/frontend test:uipassed.pnpm --filter @pipeline/frontend typecheckpassed.tests/smoke/research-flow.shandtests/smoke/plan-flow.shpassed with Docker Compose after Docker socket escalation.