229 lines
9.2 KiB
Markdown
229 lines
9.2 KiB
Markdown
# Dashcam catalog ingestion
|
|
|
|
This stack runs in the same cloud as the Next.js application - one Swarm/Portainer environment, not a separate server. The ingestion side owns PostgreSQL, crawler state, raw snapshots, moderation state, and export artifacts. The `ingestion-worker` writes the approved catalog to a shared export volume, and the Next.js build/app reads those artifacts directly from that volume - no network call and no token. Integrity is guaranteed by the manifest `sha256` and a shared Zod schema validated at read/build time.
|
|
|
|
> Note: co-locating ingestion and the web app on one shared export volume supersedes the earlier separate-server design (decision id `videoreg-nnv.3`).
|
|
|
|
## Services
|
|
|
|
`docker-compose.yml` starts:
|
|
|
|
- `postgres` - source of truth for ingestion state.
|
|
- `ingestion-worker` - HTTP-only crawler/parser/normalizer; also writes export artifacts to the shared export volume.
|
|
- `ingestion-admin` - minimal operational endpoints and operator UI, internal/localhost only.
|
|
|
|
The regular crawl path does not use a browser, stealth mode, proxy rotation, captcha bypass, or aggressive parallel loading.
|
|
|
|
For Portainer-based deployment, deploy the same combined Swarm stack described
|
|
in [`deploy/portainer`](../deploy/portainer/README.md).
|
|
|
|
CI/CD builds and deploys both runtime images through `.gitea/workflows/master.yml`.
|
|
|
|
## Server prerequisites
|
|
|
|
- Docker with Compose v2.
|
|
- Outbound HTTPS access to `market.yandex.ru` and, when enabled, `www.mvideo.ru`.
|
|
- Persistent Docker volumes for `postgres-data` and `ingestion-data` plus the
|
|
shared catalog export directory.
|
|
|
|
Do not expose Postgres or the admin server publicly. Keep them bound to `127.0.0.1` and use a private network for operator access.
|
|
|
|
## First deploy in the application environment
|
|
|
|
Deploy this stack in the same environment as the Next.js app. From the repository root:
|
|
|
|
```bash
|
|
cp ingestion/.env.example ingestion/.env
|
|
```
|
|
|
|
Edit `ingestion/.env`:
|
|
|
|
```bash
|
|
POSTGRES_PASSWORD=<strong-postgres-password>
|
|
INGESTION_ADMIN_USERNAME=admin
|
|
INGESTION_ADMIN_PASSWORD=<strong-admin-password>
|
|
```
|
|
|
|
The `ingestion-worker` writes the validated `manifest.json` and
|
|
`artifacts/<exportId>/catalog.json` to `INGESTION_EXPORT_DIR` (`/app/exports`
|
|
inside local Compose). Local Compose maps it to `data/catalog-exports`; the web
|
|
app reads that directory directly.
|
|
|
|
Start the stack:
|
|
|
|
```bash
|
|
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
|
|
```
|
|
|
|
Check status:
|
|
|
|
```bash
|
|
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml ps
|
|
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml logs -f ingestion-worker
|
|
curl http://127.0.0.1:4101/health
|
|
```
|
|
|
|
Open the ingestion admin UI:
|
|
|
|
```text
|
|
http://127.0.0.1:4101/admin
|
|
```
|
|
|
|
The UI is served by `ingestion-admin` itself and requires Basic Auth using
|
|
`INGESTION_ADMIN_USERNAME` and `INGESTION_ADMIN_PASSWORD`. It is intentionally
|
|
separate from the Next.js application and uses admin API endpoints under
|
|
`/admin/api`.
|
|
|
|
For a single public host/standard HTTPS port, configure Traefik to route by path:
|
|
|
|
```text
|
|
Host(`example.com`) && PathPrefix(`/admin`) -> ingestion-admin:4101
|
|
Host(`example.com`) && PathPrefix(`/`) -> Next.js:3000
|
|
```
|
|
|
|
Set a higher priority on the `/admin` router than the `/` router. Do not strip
|
|
the `/admin` prefix; the admin app serves HTML/assets at `/admin/*` and its API
|
|
at `/admin/api/*`.
|
|
|
|
## Local smoke run without PostgreSQL
|
|
|
|
For parser/debug checks you can run local HTTP-only smoke commands. They do not
|
|
write to PostgreSQL and do not write snapshots.
|
|
|
|
```bash
|
|
npm run --silent ingestion:smoke-yandex
|
|
npm run --silent ingestion:smoke-mvideo
|
|
```
|
|
|
|
Useful overrides:
|
|
|
|
```bash
|
|
INGESTION_SMOKE_LIMIT=5 \
|
|
INGESTION_SMOKE_DELAY_MS=5000 \
|
|
YANDEX_MARKET_DASHCAM_CATEGORY_URL="https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269" \
|
|
npm run --silent ingestion:smoke-yandex > /tmp/yandex-market-smoke.json
|
|
|
|
INGESTION_SMOKE_LIMIT=5 \
|
|
INGESTION_SMOKE_DELAY_MS=5000 \
|
|
MVIDEO_DASHCAM_CATEGORY_URL="https://www.mvideo.ru/product-list-page?q=видеорегистраторы" \
|
|
npm run --silent ingestion:smoke-mvideo > /tmp/mvideo-smoke.json
|
|
```
|
|
|
|
Configured category seed env vars:
|
|
|
|
- `YANDEX_MARKET_DASHCAM_CATEGORY_URL` - active source, used by the worker and smoke run.
|
|
- `MVIDEO_DASHCAM_CATEGORY_URL` - enables MVideo discovery and product parsing in
|
|
the worker. Leave it empty to disable that source.
|
|
|
|
If a source returns a CAPTCHA, 403, or 429, the worker records the reason and
|
|
pauses that source in PostgreSQL. It does not switch user agents, use a browser
|
|
fallback, rotate proxies, or otherwise attempt to bypass the challenge. The
|
|
queue and moderation UI keep the paused state visible to the operator.
|
|
|
|
Progress goes to stderr, so stdout remains valid JSON. Each product includes:
|
|
|
|
- `sources[]` with source name, URL, source product id, fetched time, title, and `stableContentHash`;
|
|
- source card `title`;
|
|
- `brand`, source `model`, `canonicalModel`, and `canonicalName`;
|
|
- `technicalSpecs.normalized` for canonical keys used by matching/export;
|
|
- `technicalSpecs.raw` for the fuller source characteristics with section groups, original labels, source values, and normalized key/value when known;
|
|
- `moderation.conflicts`, currently empty in smoke mode because there is only one source per product;
|
|
- `internalSignals.aggregateRating`, kept out of public export.
|
|
|
|
## Operational endpoints
|
|
|
|
Admin endpoints are intentionally small:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:4101/health
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
http://127.0.0.1:4101/admin/api/queue
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
http://127.0.0.1:4101/admin/api/moderation
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
http://127.0.0.1:4101/admin/api/moderation/products?status=needs_review
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
http://127.0.0.1:4101/admin/api/manufacturers
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
http://127.0.0.1:4101/admin/api/manufacturers/spawnson
|
|
curl -u "$INGESTION_ADMIN_USERNAME:$INGESTION_ADMIN_PASSWORD" \
|
|
-X PATCH http://127.0.0.1:4101/admin/api/manufacturers/spawnson \
|
|
-H "content-type: application/json" \
|
|
-d '{"trustStatus":"untrusted","notes":"No stable manufacturer source found"}'
|
|
```
|
|
|
|
Manufacturer records are created automatically from parsed product brands. Use
|
|
`trustStatus=untrusted` for brands that should not be trusted as authoritative
|
|
manufacturer sources during moderation.
|
|
|
|
When two sources produce the same canonical name, ingestion links their source
|
|
records to one canonical product and queues `same_product` for review. Raw specs
|
|
remain source-scoped; differing normalized values appear as source conflicts.
|
|
This is an exact canonical-name match only, so variants are never merged by a
|
|
fuzzy or image-based heuristic.
|
|
|
|
The same manufacturer records can be edited in the ingestion admin UI at
|
|
`http://127.0.0.1:4101/admin`.
|
|
|
|
Product cards can be approved or rejected from the `Products` view in that UI.
|
|
|
|
## Publishing an approved export
|
|
|
|
The public export includes only approved canonical products. It excludes prices, sellers, availability, review texts, review authors, and aggregate rating.
|
|
|
|
For the current MVP, moderation is represented in PostgreSQL. Approving a product means setting `products.moderation_status = 'approved'` and approving selected cached image artifacts in `product_images` once image storage policy is accepted. Products without approved image artifacts can still be exported with an empty `images` array.
|
|
|
|
Generate the latest export:
|
|
|
|
```bash
|
|
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml run --rm ingestion-worker \
|
|
npm run ingestion:export-catalog
|
|
```
|
|
|
|
The job writes to the shared export volume:
|
|
|
|
- `manifest.json`
|
|
- `artifacts/<exportId>/catalog.json`
|
|
|
|
These land directly on the shared export volume; nothing serves them over HTTP.
|
|
|
|
## Importing into the Next.js/application
|
|
|
|
The import reads the export artifacts directly from the shared export volume - no network call and no token. Point it at the mounted export directory and run:
|
|
|
|
```bash
|
|
DASHCAM_CATALOG_EXPORT_DIR=/path/to/exports npm run catalog:import
|
|
```
|
|
|
|
The import command:
|
|
|
|
- reads `manifest.json` from the export directory;
|
|
- validates `schemaVersion`, `exportId`, `createdAt`, `sha256`, and `itemCount`;
|
|
- reads the catalog artifact from the same directory;
|
|
- verifies the artifact SHA-256;
|
|
- validates the payload through the shared Zod schema;
|
|
- atomically updates `data/videoreg.json`.
|
|
|
|
The Next.js application never connects to the ingestion PostgreSQL database; its only catalog contact is the export artifact on the shared volume.
|
|
|
|
## Backups
|
|
|
|
Back up both Docker volumes:
|
|
|
|
- `postgres-data` - canonical ingestion database.
|
|
- `ingestion-data` - raw snapshots.
|
|
- `data/catalog-exports` locally, or `catalog_exports` in Swarm - approved
|
|
catalog artifacts shared with web.
|
|
|
|
Losing `ingestion-data` removes audit/debug snapshots even if the database remains available.
|
|
|
|
## Updating
|
|
|
|
Pull the new repository state and rebuild:
|
|
|
|
```bash
|
|
docker compose --env-file ingestion/.env -f ingestion/docker-compose.yml up -d --build
|
|
```
|
|
|
|
Migrations run automatically before `ingestion-worker` and `ingestion-admin` start. They are tracked in the `schema_migrations` table.
|