feat: complete dashcam catalog ingestion demo

ingestion/README.md

@@ -0,0 +1,228 @@
+# Dashcam catalog ingestion
+
+This stack runs in the same cloud as the Next.js application - one Swarm/Portainer environment, not a separate server. The ingestion side owns PostgreSQL, crawler state, raw snapshots, moderation state, and export artifacts. The `ingestion-worker` writes the approved catalog to a shared export volume, and the Next.js build/app reads those artifacts directly from that volume - no network call and no token. Integrity is guaranteed by the manifest `sha256` and a shared Zod schema validated at read/build time.
+
+> Note: co-locating ingestion and the web app on one shared export volume supersedes the earlier separate-server design (decision id `videoreg-nnv.3`).
+
+## Services
+
+`docker-compose.yml` starts:
+
+- `postgres` - source of truth for ingestion state.
+- `ingestion-worker` - HTTP-only crawler/parser/normalizer; also writes export artifacts to the shared export volume.
+- `ingestion-admin` - minimal operational endpoints and operator UI, internal/localhost only.
+
+The regular crawl path does not use a browser, stealth mode, proxy rotation, captcha bypass, or aggressive parallel loading.
+
+For Portainer-based deployment, deploy the same combined Swarm stack described
+in [`deploy/portainer`](../deploy/portainer/README.md).
+
+CI/CD builds and deploys both runtime images through `.gitea/workflows/master.yml`.
+
+## Server prerequisites
+
+- Docker with Compose v2.
+- Outbound HTTPS access to `market.yandex.ru` and, when enabled, `www.mvideo.ru`.
+- Persistent Docker volumes for `postgres-data` and `ingestion-data` plus the
+  shared catalog export directory.
+
+Do not expose Postgres or the admin server publicly. Keep them bound to `127.0.0.1` and use a private network for operator access.
+
+## First deploy in the application environment
+
+Deploy this stack in the same environment as the Next.js app. From the repository root:
+
+```bash
+cp ingestion/.env.example ingestion/.env
+```
+
+Edit `ingestion/.env`:
+
+```bash
+POSTGRES_PASSWORD=<strong-postgres-password>
+INGESTION_ADMIN_USERNAME=admin
+INGESTION_ADMIN_PASSWORD=<strong-admin-password>
+```
+
+The `ingestion-worker` writes the validated `manifest.json` and
+`artifacts/<exportId>/catalog.json` to `INGESTION_EXPORT_DIR` (`/app/exports`
+inside local Compose). Local Compose maps it to `data/catalog-exports`; the web
+app reads that directory directly.
+
+Start the stack:
+
+```bash
+docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
+```
+
+Check status:
+
+```bash
+docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml ps
+docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml logs -f ingestion-worker
+curl http://127.0.0.1:4101/health
+```
+
+Open the ingestion admin UI:
+
+```text
+http://127.0.0.1:4101/admin
+```
+
+The UI is served by `ingestion-admin` itself and requires Basic Auth using
+`INGESTION_ADMIN_USERNAME` and `INGESTION_ADMIN_PASSWORD`. It is intentionally
+separate from the Next.js application and uses admin API endpoints under
+`/admin/api`.
+
+For a single public host/standard HTTPS port, configure Traefik to route by path:
+
+```text
+Host(`example.com`) && PathPrefix(`/admin`) -> ingestion-admin:4101
+Host(`example.com`) && PathPrefix(`/`)      -> Next.js:3000
+```
+
+Set a higher priority on the `/admin` router than the `/` router. Do not strip
+the `/admin` prefix; the admin app serves HTML/assets at `/admin/*` and its API
+at `/admin/api/*`.
+
+## Local smoke run without PostgreSQL
+
+For parser/debug checks you can run local HTTP-only smoke commands. They do not
+write to PostgreSQL and do not write snapshots.
+
+```bash
+npm run --silent ingestion:smoke-yandex
+npm run --silent ingestion:smoke-mvideo
+```
+
+Useful overrides:
+
+```bash
+INGESTION_SMOKE_LIMIT=5 \
+INGESTION_SMOKE_DELAY_MS=5000 \
+YANDEX_MARKET_DASHCAM_CATEGORY_URL="https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269" \
+npm run --silent ingestion:smoke-yandex > /tmp/yandex-market-smoke.json
+
+INGESTION_SMOKE_LIMIT=5 \
+INGESTION_SMOKE_DELAY_MS=5000 \
+MVIDEO_DASHCAM_CATEGORY_URL="https://www.mvideo.ru/product-list-page?q=видеорегистраторы" \
+npm run --silent ingestion:smoke-mvideo > /tmp/mvideo-smoke.json
+```
+
+Configured category seed env vars:
+
+- `YANDEX_MARKET_DASHCAM_CATEGORY_URL` - active source, used by the worker and smoke run.
+- `MVIDEO_DASHCAM_CATEGORY_URL` - enables MVideo discovery and product parsing in
+  the worker. Leave it empty to disable that source.
+
+If a source returns a CAPTCHA, 403, or 429, the worker records the reason and
+pauses that source in PostgreSQL. It does not switch user agents, use a browser
+fallback, rotate proxies, or otherwise attempt to bypass the challenge. The
+queue and moderation UI keep the paused state visible to the operator.
+
+Progress goes to stderr, so stdout remains valid JSON. Each product includes:
+
+- `sources[]` with source name, URL, source product id, fetched time, title, and `stableContentHash`;
+- source card `title`;
+- `brand`, source `model`, `canonicalModel`, and `canonicalName`;
+- `technicalSpecs.normalized` for canonical keys used by matching/export;
+- `technicalSpecs.raw` for the fuller source characteristics with section groups, original labels, source values, and normalized key/value when known;
+- `moderation.conflicts`, currently empty in smoke mode because there is only one source per product;
+- `internalSignals.aggregateRating`, kept out of public export.
+
+## Operational endpoints
+
+Admin endpoints are intentionally small:
+
+```bash
+curl http://127.0.0.1:4101/health
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  http://127.0.0.1:4101/admin/api/queue
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  http://127.0.0.1:4101/admin/api/moderation
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  http://127.0.0.1:4101/admin/api/moderation/products?status=needs_review
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  http://127.0.0.1:4101/admin/api/manufacturers
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  http://127.0.0.1:4101/admin/api/manufacturers/spawnson
+curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
+  -X PATCH http://127.0.0.1:4101/admin/api/manufacturers/spawnson \
+  -H "content-type: application/json" \
+  -d '{"trustStatus":"untrusted","notes":"No stable manufacturer source found"}'
+```
+
+Manufacturer records are created automatically from parsed product brands. Use
+`trustStatus=untrusted` for brands that should not be trusted as authoritative
+manufacturer sources during moderation.
+
+When two sources produce the same canonical name, ingestion links their source
+records to one canonical product and queues `same_product` for review. Raw specs
+remain source-scoped; differing normalized values appear as source conflicts.
+This is an exact canonical-name match only, so variants are never merged by a
+fuzzy or image-based heuristic.
+
+The same manufacturer records can be edited in the ingestion admin UI at
+`http://127.0.0.1:4101/admin`.
+
+Product cards can be approved or rejected from the `Products` view in that UI.
+
+## Publishing an approved export
+
+The public export includes only approved canonical products. It excludes prices, sellers, availability, review texts, review authors, and aggregate rating.
+
+For the current MVP, moderation is represented in PostgreSQL. Approving a product means setting `products.moderation_status = 'approved'` and approving selected cached image artifacts in `product_images` once image storage policy is accepted. Products without approved image artifacts can still be exported with an empty `images` array.
+
+Generate the latest export:
+
+```bash
+docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml run --rm ingestion-worker \
+  npm run ingestion:export-catalog
+```
+
+The job writes to the shared export volume:
+
+- `manifest.json`
+- `artifacts/<exportId>/catalog.json`
+
+These land directly on the shared export volume; nothing serves them over HTTP.
+
+## Importing into the Next.js/application
+
+The import reads the export artifacts directly from the shared export volume - no network call and no token. Point it at the mounted export directory and run:
+
+```bash
+DASHCAM_CATALOG_EXPORT_DIR=/path/to/exports npm run catalog:import
+```
+
+The import command:
+
+- reads `manifest.json` from the export directory;
+- validates `schemaVersion`, `exportId`, `createdAt`, `sha256`, and `itemCount`;
+- reads the catalog artifact from the same directory;
+- verifies the artifact SHA-256;
+- validates the payload through the shared Zod schema;
+- atomically updates `data/videoreg.json`.
+
+The Next.js application never connects to the ingestion PostgreSQL database; its only catalog contact is the export artifact on the shared volume.
+
+## Backups
+
+Back up both Docker volumes:
+
+- `postgres-data` - canonical ingestion database.
+- `ingestion-data` - raw snapshots.
+- `data/catalog-exports` locally, or `catalog_exports` in Swarm - approved
+  catalog artifacts shared with web.
+
+Losing `ingestion-data` removes audit/debug snapshots even if the database remains available.
+
+## Updating
+
+Pull the new repository state and rebuild:
+
+```bash
+docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
+```
+
+Migrations run automatically before `ingestion-worker` and `ingestion-admin` start. They are tracked in the `schema_migrations` table.