62 lines
2.5 KiB
Markdown
62 lines
2.5 KiB
Markdown
# Task 010: Research Fetch And S3 Manifest
|
|
|
|
Development description: Implement explicit auditable web research with source retrieval metadata, source-section snapshots, object-storage uploads, and Postgres manifest records.
|
|
|
|
## Implementation Details
|
|
|
|
- Backend endpoint:
|
|
- `POST /api/articles/{article_id}/research/start`
|
|
- `GET /api/articles/{article_id}/research`
|
|
- Research starts only after approved plan.
|
|
- Research uses an explicit search/fetch script, not only free-form agent browsing.
|
|
- For discovered sources, store in object storage:
|
|
- Relevant content sections.
|
|
- Page metadata.
|
|
- Search path to the article.
|
|
- Agent/source-selection criteria.
|
|
- Page `<head>` metadata.
|
|
- Server IP address.
|
|
- Domain WHOIS owner when available.
|
|
- In Postgres store only:
|
|
- Research run manifest.
|
|
- S3/object keys.
|
|
- Source URLs.
|
|
- Content hashes.
|
|
- Artifact types.
|
|
- Summary metadata required for UI and validation.
|
|
- Retention is indefinite for v1.
|
|
- Provide deterministic fake search/fetch fixtures for tests and demo.
|
|
|
|
## Public Interface
|
|
|
|
- Editor starts research for an approved article plan.
|
|
- UI shows research run status and discovered source summary.
|
|
- Backend exposes manifest-linked evidence summaries.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] TDD pre-requirement: before implementation, write one failing behavior test that starts research from an approved plan and verifies a manifest with object keys is produced; proceed one research behavior at a time and record evidence in `Result`.
|
|
- [ ] Research cannot start before plan approval.
|
|
- [ ] Research run creates an agent/job record and article status `RESEARCH_RUNNING`.
|
|
- [ ] Source-section artifacts are uploaded to object storage.
|
|
- [ ] Manifest records include object keys, hashes, URLs, artifact types, and metadata references.
|
|
- [ ] Re-running research creates a new manifest instead of overwriting old artifacts.
|
|
- [ ] UI can show source URL, title, domain, type, summary, retrieval timestamp, and artifact link.
|
|
- [ ] Demo stack can run with deterministic fake research results.
|
|
|
|
## Verification
|
|
|
|
- Run research integration tests with fake search/fetch.
|
|
- Verify object storage contains source-section artifact files.
|
|
- Verify Postgres manifest records point to those files.
|
|
- Run Docker Compose research smoke flow.
|
|
|
|
## Result
|
|
|
|
- Status: Pending execution.
|
|
- TDD plan: To be filled during execution.
|
|
- Red evidence: To be filled during execution.
|
|
- Green evidence: To be filled during execution.
|
|
- Refactor notes: To be filled during execution.
|
|
- Verification output: To be filled during execution.
|