104 lines
4.0 KiB
Markdown
104 lines
4.0 KiB
Markdown
## videoreg.ru
|
|
|
|
Small video registry using TypeScript, a Next.js web frontend, App Router, FSD-oriented folders, and a primitive file-backed JSON database.
|
|
|
|
The Next.js app lives under `web-front/`. Ingestion code lives under `ingestion/`.
|
|
|
|
### Local development
|
|
|
|
```bash
|
|
npm install
|
|
DASHCAM_CATALOG_EXPORT_DIR="$PWD/data/catalog-exports" npm run dev
|
|
```
|
|
|
|
The app runs on [http://localhost:3000](http://localhost:3000).
|
|
|
|
### Data storage
|
|
|
|
By default records are stored in `data/videoreg.json`. Override the path with:
|
|
|
|
```bash
|
|
VIDEOREG_DB_FILE=/absolute/path/videoreg.json npm run dev
|
|
```
|
|
|
|
Reads are intentionally cached aggressively:
|
|
|
|
- process memory cache for the primitive JSON database;
|
|
- Next.js cache tag for dashboard/API reads;
|
|
- long HTTP cache headers on the read API.
|
|
|
|
Writes call `revalidateTag` and update the process cache.
|
|
|
|
### Dashcam catalog ingestion
|
|
|
|
The catalog ingestion services run in the **same cloud** as the Next.js app
|
|
(one Swarm/Portainer environment), not on a separate server:
|
|
|
|
```bash
|
|
docker compose -f ingestion/docker-compose.yml up --build
|
|
```
|
|
|
|
It starts PostgreSQL, an HTTP-only ingestion worker, and a minimal admin server on
|
|
`http://localhost:4101`. The worker writes the validated `manifest.json` +
|
|
`catalog.json` to a shared export volume; there is no HTTP export server and no
|
|
export token — the app reads the artifacts directly from that volume.
|
|
|
|
The default Yandex Market seed category is:
|
|
|
|
```text
|
|
https://market.yandex.ru/catalog--avtomobilnye-videoregistratory/82798275/list?hid=82798269
|
|
```
|
|
|
|
Seed env vars are source-specific: `YANDEX_MARKET_DASHCAM_CATEGORY_URL` is active
|
|
today. `MVIDEO_DASHCAM_CATEGORY_URL` enables the MVideo discovery/parser path
|
|
when configured.
|
|
|
|
Crawler policy is intentionally conservative: no browser fallback in the regular
|
|
path, no stealth/proxy behavior, PostgreSQL-backed queue and daily budgets, raw
|
|
snapshots stored on the ingestion volume, and source pause on captcha/403/429.
|
|
|
|
To import an approved export into the app read-model cache, point the import at
|
|
the shared export volume:
|
|
|
|
```bash
|
|
DASHCAM_CATALOG_EXPORT_DIR=/path/to/ingestion/data/exports \
|
|
npm run catalog:import
|
|
```
|
|
|
|
The import command reads `manifest.json` + `catalog.json` from that directory,
|
|
validates them with Zod, verifies the artifact `sha256`, checks
|
|
`exportId`/`createdAt`/`itemCount`, and atomically updates `data/videoreg.json`.
|
|
No HTTP call and no token are involved.
|
|
|
|
The ingestion admin server exposes its own operator UI at
|
|
`http://127.0.0.1:4101/admin`. It is served by `ingestion-admin`, requires Basic
|
|
Auth, and is not part of the Next.js app. In production Traefik should route
|
|
`/admin` to `ingestion-admin` and `/` to Next.js, without stripping the `/admin`
|
|
prefix.
|
|
|
|
For the production setup where ingestion is co-located with the app and shares
|
|
the export volume, see [`ingestion/README.md`](ingestion/README.md).
|
|
For Portainer stacks, see [`deploy/portainer/README.md`](deploy/portainer/README.md).
|
|
For the local end-to-end demonstration, see [`DEMO.md`](DEMO.md).
|
|
|
|
### Deployment
|
|
|
|
Gitea Actions runs:
|
|
|
|
- `.gitea/workflows/pr.yml`: quality gate for pull requests into `master`;
|
|
- `.gitea/workflows/master.yml`: quality gate, build/push of web and ingestion images, and direct co-located Swarm deploy from `master`.
|
|
|
|
The production workflow uses the runner's existing Gitea configuration names:
|
|
|
|
- variables: `REGISTRY`, `IMAGE_REPO`, `DOMAIN` (`videoreg.ru`), `STACK_NAME`,
|
|
`TRAEFIK_NETWORK`, `TRAEFIK_ENTRYPOINT`, and `TRAEFIK_CERTRESOLVER`;
|
|
- secrets: `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `SWARM_MANAGER_HOST`,
|
|
`SWARM_SSH_USER`, `SWARM_SSH_PORT`, `INGESTION_POSTGRES_PASSWORD`, and
|
|
`INGESTION_ADMIN_PASSWORD`.
|
|
|
|
Images are published as `${IMAGE_REPO}-web` and `${IMAGE_REPO}-ingestion`, then
|
|
the Swarm manager pulls their immutable digests. Both `/` and `/admin` receive
|
|
the configured TLS certificate through Traefik.
|
|
|
|
The Swarm stack defaults to one replica on a manager node because the primitive database is a mounted JSON file. Use a real database before raising replicas above one or spreading the service across nodes.
|