videoreg.ru/README.md

videoreg.ru

Small video registry using TypeScript, a Next.js web frontend, App Router, FSD-oriented folders, and a primitive file-backed JSON database.

The Next.js app lives under web-front/. Ingestion code lives under ingestion/.

Local development

npm install
DASHCAM_CATALOG_EXPORT_DIR="$PWD/data/catalog-exports" npm run dev

The app runs on http://localhost:3000.

Data storage

By default records are stored in data/videoreg.json. Override the path with:

VIDEOREG_DB_FILE=/absolute/path/videoreg.json npm run dev

Reads are intentionally cached aggressively:

• process memory cache for the primitive JSON database;
• Next.js cache tag for dashboard/API reads;
• long HTTP cache headers on the read API.

Writes call revalidateTag and update the process cache.

Dashcam catalog ingestion

The catalog ingestion services run in the same cloud as the Next.js app (one Swarm/Portainer environment), not on a separate server:

docker compose -f ingestion/docker-compose.yml up --build

It starts PostgreSQL, an HTTP-only ingestion worker, and a minimal admin server on http://localhost:4101. The worker writes the validated manifest.json + catalog.json to a shared export volume; there is no HTTP export server and no export token — the app reads the artifacts directly from that volume.

The default Yandex Market seed category is:

https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269

Seed env vars are source-specific: YANDEX_MARKET_DASHCAM_CATEGORY_URL is active today. MVIDEO_DASHCAM_CATEGORY_URL enables the MVideo discovery/parser path when configured.

Crawler policy is intentionally conservative: no browser fallback in the regular path, no stealth/proxy behavior, PostgreSQL-backed queue and daily budgets, raw snapshots stored on the ingestion volume, and source pause on captcha/403/429.

To import an approved export into the app read-model cache, point the import at the shared export volume:

DASHCAM_CATALOG_EXPORT_DIR=/path/to/ingestion/data/exports \
npm run catalog:import

The import command reads manifest.json + catalog.json from that directory, validates them with Zod, verifies the artifact sha256, checks exportId/createdAt/itemCount, and atomically updates data/videoreg.json. No HTTP call and no token are involved.

The ingestion admin server exposes its own operator UI at http://127.0.0.1:4101/admin. It is served by ingestion-admin, requires Basic Auth, and is not part of the Next.js app. In production Traefik should route /admin to ingestion-admin and / to Next.js, without stripping the /admin prefix.

For the production setup where ingestion is co-located with the app and shares the export volume, see ingestion/README.md. For Portainer stacks, see deploy/portainer/README.md. For the local end-to-end demonstration, see DEMO.md.

Deployment

Gitea Actions runs:

• .gitea/workflows/pr.yml: quality gate for pull requests into master;
• .gitea/workflows/master.yml: quality gate, build/push of web and ingestion images, and direct co-located Swarm deploy from master.

The production workflow uses the runner's existing Gitea configuration names:

• variables: REGISTRY, IMAGE_REPO, DOMAIN (videoreg.ru), STACK_NAME, TRAEFIK_NETWORK, TRAEFIK_ENTRYPOINT, and TRAEFIK_CERTRESOLVER;
• secrets: REGISTRY_USERNAME, REGISTRY_PASSWORD, SWARM_MANAGER_HOST, SWARM_SSH_USER, SWARM_SSH_PORT, INGESTION_POSTGRES_PASSWORD, and INGESTION_ADMIN_PASSWORD.

Images are published as ${IMAGE_REPO}-web and ${IMAGE_REPO}-ingestion, then the Swarm manager pulls their immutable digests. Both / and /admin receive the configured TLS certificate through Traefik.

The Swarm stack defaults to one replica on a manager node because the primitive database is a mounted JSON file. Use a real database before raising replicas above one or spreading the service across nodes.