# Dashcam catalog ingestion

This stack runs in the same cloud as the Next.js application - one Swarm/Portainer environment, not a separate server. The ingestion side owns PostgreSQL, crawler state, raw snapshots, moderation state, and export artifacts. The `ingestion-worker` writes the approved catalog to a shared export volume, and the Next.js build/app reads those artifacts directly from that volume - no network call and no token. Integrity is guaranteed by the manifest `sha256` and a shared Zod schema validated at read/build time.

> Note: co-locating ingestion and the web app on one shared export volume supersedes the earlier separate-server design (decision id `videoreg-nnv.3`).

## Services

`docker-compose.yml` starts:

- `postgres` - source of truth for ingestion state.
- `ingestion-worker` - HTTP-only crawler/parser/normalizer; also writes export artifacts to the shared export volume.
- `ingestion-admin` - minimal operational endpoints and operator UI, internal/localhost only.

The regular crawl path does not use a browser, stealth mode, proxy rotation, captcha bypass, or aggressive parallel loading.

For Portainer-based deployment, deploy the same combined Swarm stack described
in [`deploy/portainer`](../deploy/portainer/README.md).

CI/CD builds and deploys both runtime images through `.gitea/workflows/master.yml`.

## Server prerequisites

- Docker with Compose v2.
- Outbound HTTPS access to `market.yandex.ru` and, when enabled, `www.mvideo.ru`.
- Persistent Docker volumes for `postgres-data` and `ingestion-data` plus the
  shared catalog export directory.

Do not expose Postgres or the admin server publicly. Keep them bound to `127.0.0.1` and use a private network for operator access.

## First deploy in the application environment

Deploy this stack in the same environment as the Next.js app. From the repository root:

```bash
cp ingestion/.env.example ingestion/.env
```

Edit `ingestion/.env`:

```bash
POSTGRES_PASSWORD=<strong-postgres-password>
INGESTION_ADMIN_USERNAME=admin
INGESTION_ADMIN_PASSWORD=<strong-admin-password>
```

The `ingestion-worker` writes the validated `manifest.json` and
`artifacts/<exportId>/catalog.json` to `INGESTION_EXPORT_DIR` (`/app/exports`
inside local Compose). Local Compose maps it to `data/catalog-exports`; the web
app reads that directory directly.

Start the stack:

```bash
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
```

Check status:

```bash
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml ps
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml logs -f ingestion-worker
curl http://127.0.0.1:4101/health
```

Open the ingestion admin UI:

```text
http://127.0.0.1:4101/admin
```

The UI is served by `ingestion-admin` itself and requires Basic Auth using
`INGESTION_ADMIN_USERNAME` and `INGESTION_ADMIN_PASSWORD`. It is intentionally
separate from the Next.js application and uses admin API endpoints under
`/admin/api`.

For a single public host/standard HTTPS port, configure Traefik to route by path:

```text
Host(`example.com`) && PathPrefix(`/admin`) -> ingestion-admin:4101
Host(`example.com`) && PathPrefix(`/`)      -> Next.js:3000
```

Set a higher priority on the `/admin` router than the `/` router. Do not strip
the `/admin` prefix; the admin app serves HTML/assets at `/admin/*` and its API
at `/admin/api/*`.

## Local smoke run without PostgreSQL

For parser/debug checks you can run local HTTP-only smoke commands. They do not
write to PostgreSQL and do not write snapshots.

```bash
npm run --silent ingestion:smoke-yandex
npm run --silent ingestion:smoke-mvideo
```

Useful overrides:

```bash
INGESTION_SMOKE_LIMIT=5 \
INGESTION_SMOKE_DELAY_MS=5000 \
YANDEX_MARKET_DASHCAM_CATEGORY_URL="https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269" \
npm run --silent ingestion:smoke-yandex > /tmp/yandex-market-smoke.json

INGESTION_SMOKE_LIMIT=5 \
INGESTION_SMOKE_DELAY_MS=5000 \
MVIDEO_DASHCAM_CATEGORY_URL="https://www.mvideo.ru/product-list-page?q=видеорегистраторы" \
npm run --silent ingestion:smoke-mvideo > /tmp/mvideo-smoke.json
```

Configured category seed env vars:

- `YANDEX_MARKET_DASHCAM_CATEGORY_URL` - active source, used by the worker and smoke run.
- `MVIDEO_DASHCAM_CATEGORY_URL` - enables MVideo discovery and product parsing in
  the worker. Leave it empty to disable that source.

If a source returns a CAPTCHA, 403, or 429, the worker records the reason and
pauses that source in PostgreSQL. It does not switch user agents, use a browser
fallback, rotate proxies, or otherwise attempt to bypass the challenge. The
queue and moderation UI keep the paused state visible to the operator.

Progress goes to stderr, so stdout remains valid JSON. Each product includes:

- `sources[]` with source name, URL, source product id, fetched time, title, and `stableContentHash`;
- source card `title`;
- `brand`, source `model`, `canonicalModel`, and `canonicalName`;
- `technicalSpecs.normalized` for canonical keys used by matching/export;
- `technicalSpecs.raw` for the fuller source characteristics with section groups, original labels, source values, and normalized key/value when known;
- `moderation.conflicts`, currently empty in smoke mode because there is only one source per product;
- `internalSignals.aggregateRating`, kept out of public export.

## Operational endpoints

Admin endpoints are intentionally small:

```bash
curl http://127.0.0.1:4101/health
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/queue
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/moderation
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/moderation/products?status=needs_review
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/manufacturers
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  http://127.0.0.1:4101/admin/api/manufacturers/spawnson
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
  -X PATCH http://127.0.0.1:4101/admin/api/manufacturers/spawnson \
  -H "content-type: application/json" \
  -d '{"trustStatus":"untrusted","notes":"No stable manufacturer source found"}'
```

Manufacturer records are created automatically from parsed product brands. Use
`trustStatus=untrusted` for brands that should not be trusted as authoritative
manufacturer sources during moderation.

When two sources produce the same canonical name, ingestion links their source
records to one canonical product and queues `same_product` for review. Raw specs
remain source-scoped; differing normalized values appear as source conflicts.
This is an exact canonical-name match only, so variants are never merged by a
fuzzy or image-based heuristic.

The same manufacturer records can be edited in the ingestion admin UI at
`http://127.0.0.1:4101/admin`.

Product cards can be approved or rejected from the `Products` view in that UI.

## Publishing an approved export

The public export includes only approved canonical products. It excludes prices, sellers, availability, review texts, review authors, and aggregate rating.

For the current MVP, moderation is represented in PostgreSQL. Approving a product means setting `products.moderation_status = 'approved'` and approving selected cached image artifacts in `product_images` once image storage policy is accepted. Products without approved image artifacts can still be exported with an empty `images` array.

Generate the latest export:

```bash
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml run --rm ingestion-worker \
  npm run ingestion:export-catalog
```

The job writes to the shared export volume:

- `manifest.json`
- `artifacts/<exportId>/catalog.json`

These land directly on the shared export volume; nothing serves them over HTTP.

## Importing into the Next.js/application

The import reads the export artifacts directly from the shared export volume - no network call and no token. Point it at the mounted export directory and run:

```bash
DASHCAM_CATALOG_EXPORT_DIR=/path/to/exports npm run catalog:import
```

The import command:

- reads `manifest.json` from the export directory;
- validates `schemaVersion`, `exportId`, `createdAt`, `sha256`, and `itemCount`;
- reads the catalog artifact from the same directory;
- verifies the artifact SHA-256;
- validates the payload through the shared Zod schema;
- atomically updates `data/videoreg.json`.

The Next.js application never connects to the ingestion PostgreSQL database; its only catalog contact is the export artifact on the shared volume.

## Backups

Back up both Docker volumes:

- `postgres-data` - canonical ingestion database.
- `ingestion-data` - raw snapshots.
- `data/catalog-exports` locally, or `catalog_exports` in Swarm - approved
  catalog artifacts shared with web.

Losing `ingestion-data` removes audit/debug snapshots even if the database remains available.

## Updating

Pull the new repository state and rebuild:

```bash
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
```

Migrations run automatically before `ingestion-worker` and `ingestion-admin` start. They are tracked in the `schema_migrations` table.