videoreg.ru/ingestion/README.md

Dashcam catalog ingestion

This stack runs in the same cloud as the Next.js application - one Swarm/Portainer environment, not a separate server. The ingestion side owns PostgreSQL, crawler state, raw snapshots, moderation state, and export artifacts. The ingestion-worker writes the approved catalog to a shared export volume, and the Next.js build/app reads those artifacts directly from that volume - no network call and no token. Integrity is guaranteed by the manifest sha256 and a shared Zod schema validated at read/build time.

Note: co-locating ingestion and the web app on one shared export volume supersedes the earlier separate-server design (decision id videoreg-nnv.3).

Services

docker-compose.yml starts:

• postgres - source of truth for ingestion state.
• ingestion-worker - HTTP-only crawler/parser/normalizer; also writes export artifacts to the shared export volume.
• ingestion-admin - minimal operational endpoints and operator UI, internal/localhost only.

The regular crawl path does not use a browser, stealth mode, proxy rotation, captcha bypass, or aggressive parallel loading.

For Portainer-based deployment, deploy the same combined Swarm stack described in deploy/portainer.

CI/CD builds and deploys both runtime images through .gitea/workflows/master.yml.

Server prerequisites

• Docker with Compose v2.
• Outbound HTTPS access to market.yandex.ru and, when enabled, www.mvideo.ru.
• Persistent Docker volumes for postgres-data and ingestion-data plus the shared catalog export directory.

Do not expose Postgres or the admin server publicly. Keep them bound to 127.0.0.1 and use a private network for operator access.

First deploy in the application environment

Deploy this stack in the same environment as the Next.js app. From the repository root:

cp ingestion/.env.example ingestion/.env

Edit ingestion/.env:

POSTGRES_PASSWORD=<strong-postgres-password>
INGESTION_ADMIN_USERNAME=admin
INGESTION_ADMIN_PASSWORD=<strong-admin-password>

The ingestion-worker writes the validated manifest.json and artifacts/<exportId>/catalog.json to INGESTION_EXPORT_DIR (/app/exports inside local Compose). Local Compose maps it to data/catalog-exports; the web app reads that directory directly.

Start the stack:

docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build

Check status:

docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml ps
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml logs -f ingestion-worker
curl http://127.0.0.1:4101/health

Open the ingestion admin UI:

http://127.0.0.1:4101/admin

The UI is served by ingestion-admin itself and requires Basic Auth using INGESTION_ADMIN_USERNAME and INGESTION_ADMIN_PASSWORD. It is intentionally separate from the Next.js application and uses admin API endpoints under /admin/api.

For a single public host/standard HTTPS port, configure Traefik to route by path:

Host(`example.com`) && PathPrefix(`/admin`) -> ingestion-admin:4101
Host(`example.com`) && PathPrefix(`/`)      -> Next.js:3000

Set a higher priority on the /admin router than the / router. Do not strip the /admin prefix; the admin app serves HTML/assets at /admin/* and its API at /admin/api/*.

Local smoke run without PostgreSQL

For parser/debug checks you can run local HTTP-only smoke commands. They do not write to PostgreSQL and do not write snapshots.

npm run --silent ingestion:smoke-yandex
npm run --silent ingestion:smoke-mvideo

Useful overrides:

INGESTION_SMOKE_LIMIT=5 \
INGESTION_SMOKE_DELAY_MS=5000 \
YANDEX_MARKET_DASHCAM_CATEGORY_URL="https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269" \
npm run --silent ingestion:smoke-yandex > /tmp/yandex-market-smoke.json

INGESTION_SMOKE_LIMIT=5 \
INGESTION_SMOKE_DELAY_MS=5000 \
MVIDEO_DASHCAM_CATEGORY_URL="https://www.mvideo.ru/product-list-page?q=видеорегистраторы" \
npm run --silent ingestion:smoke-mvideo > /tmp/mvideo-smoke.json

Configured category seed env vars:

• YANDEX_MARKET_DASHCAM_CATEGORY_URL - active source, used by the worker and smoke run.
• MVIDEO_DASHCAM_CATEGORY_URL - enables MVideo discovery and product parsing in the worker. Leave it empty to disable that source.

If a source returns a CAPTCHA, 403, or 429, the worker records the reason and pauses that source in PostgreSQL. It does not switch user agents, use a browser fallback, rotate proxies, or otherwise attempt to bypass the challenge. The queue and moderation UI keep the paused state visible to the operator.

Progress goes to stderr, so stdout remains valid JSON. Each product includes:

• sources[] with source name, URL, source product id, fetched time, title, and stableContentHash;
• source card title;
• brand, source model, canonicalModel, and canonicalName;
• technicalSpecs.normalized for canonical keys used by matching/export;
• technicalSpecs.raw for the fuller source characteristics with section groups, original labels, source values, and normalized key/value when known;
• moderation.conflicts, currently empty in smoke mode because there is only one source per product;
• internalSignals.aggregateRating, kept out of public export.

Operational endpoints

Admin endpoints are intentionally small:

curl http://127.0.0.1:4101/health
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/queue
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/moderation
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/moderation/products?status=needs_review
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/manufacturers
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/manufacturers/spawnson
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  -X PATCH http://127.0.0.1:4101/admin/api/manufacturers/spawnson \
  -H "content-type: application/json" \
  -d '{"trustStatus":"untrusted","notes":"No stable manufacturer source found"}'

Manufacturer records are created automatically from parsed product brands. Use trustStatus=untrusted for brands that should not be trusted as authoritative manufacturer sources during moderation.

When two sources produce the same canonical name, ingestion links their source records to one canonical product and queues same_product for review. Raw specs remain source-scoped; differing normalized values appear as source conflicts. This is an exact canonical-name match only, so variants are never merged by a fuzzy or image-based heuristic.

The same manufacturer records can be edited in the ingestion admin UI at http://127.0.0.1:4101/admin.

Product cards can be approved or rejected from the Products view in that UI.

Publishing an approved export

The public export includes only approved canonical products. It excludes prices, sellers, availability, review texts, review authors, and aggregate rating.

For the current MVP, moderation is represented in PostgreSQL. Approving a product means setting products.moderation_status = 'approved' and approving selected cached image artifacts in product_images once image storage policy is accepted. Products without approved image artifacts can still be exported with an empty images array.

Generate the latest export:

docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml run --rm ingestion-worker \
  npm run ingestion:export-catalog

The job writes to the shared export volume:

• manifest.json
• artifacts/<exportId>/catalog.json

These land directly on the shared export volume; nothing serves them over HTTP.

Importing into the Next.js/application

The import reads the export artifacts directly from the shared export volume - no network call and no token. Point it at the mounted export directory and run:

DASHCAM_CATALOG_EXPORT_DIR=/path/to/exports npm run catalog:import

The import command:

• reads manifest.json from the export directory;
• validates schemaVersion, exportId, createdAt, sha256, and itemCount;
• reads the catalog artifact from the same directory;
• verifies the artifact SHA-256;
• validates the payload through the shared Zod schema;
• atomically updates data/videoreg.json.

The Next.js application never connects to the ingestion PostgreSQL database; its only catalog contact is the export artifact on the shared volume.

Backups

Back up both Docker volumes:

• postgres-data - canonical ingestion database.
• ingestion-data - raw snapshots.
• data/catalog-exports locally, or catalog_exports in Swarm - approved catalog artifacts shared with web.

Losing ingestion-data removes audit/debug snapshots even if the database remains available.

Updating

Pull the new repository state and rebuild:

docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build

Migrations run automatically before ingestion-worker and ingestion-admin start. They are tracked in the schema_migrations table.