feat: image variants cache panel

This commit is contained in:
fsylvain committed 2026-10-08 21:13:32 +02:00
1 parent 951fef18ba
commit d768668014
53 files changed
+5809 -46

No files matched your search

+28 -1
View File
@@ -62,7 +62,7 @@ never changes the response or the change itself.
| Category | Actions |
| --- | --- |
| `articles` | `article.create`, `article.edit`, `article.publish`, `article.unpublish`, `article.delete` |
| `media` | `media.upload`, `media.rename`, `media.delete`, `media.merge`, `media.purge`, `media.variant_create`, `media.variant_update`, `media.variant_delete`, `media.variant_enable`, `media.variant_disable`, `media.variant_purge`, `media.variant_generate_all`, `media.variant_purge_image` |
| `media` | `media.upload`, `media.rename`, `media.delete`, `media.merge`, `media.purge`, `media.variant_create`, `media.variant_update`, `media.variant_delete`, `media.variant_enable`, `media.variant_disable`, `media.variant_purge`, `media.variant_generate_all`, `media.variant_purge_image`, `media.variant_purge_all`, `media.variant_purge_unused` |
| `users` | `user.register`, `user.verify`, `user.update_profile`, `invite.create`, `invite.cancel`, `invite.renew`, `invite.accept` |
| `system` | `system.media_purge` (scheduled), `system.update_checked`, `system.update_downloaded`, `system.restart_requested`, `system.activity_cleanup`, `system.activity_retention_changed` |
@@ -77,6 +77,10 @@ name) records nothing. Params: `media.merge` and `media.purge` and `system.media
`unsupported_format` or `failed`) and is `failed` only when nothing was generated and something failed;
`media.variant_purge_image` carries `purged_files` and `purged_bytes`; both have an `image` target
and are recorded only when files were generated or removed;
`media.variant_purge_all` and `media.variant_purge_unused` (no target) carry `purged_files` and
`purged_bytes`, the latter also `older_than_days`; a partial purge adds `failed_files` and stays
`success`, and a purge that removed nothing while some files failed is `failed`; nothing is recorded
when no file was removed or failed;
`system.update_checked` carries `update_available`; `system.restart_requested` carries `kind` and,
for an update, `tag` (it has no target). `system.update_downloaded` is recorded when the job reaches
`ready` or `failed`, with the origin captured at request time; a refusal at the start is recorded
@@ -110,6 +114,8 @@ this." Behaviour, caching and the public `?v=<name>` contract are in [image-vari
| `PATCH /v1/image-variants/{name}` | `{ "enabled": true }` or `{ "enabled": false }`. Keeps the generated files. |
| `DELETE /v1/image-variants/{name}` | 204. Removes the definition and its generated files. |
| `POST /v1/image-variants/{name}/purge` | `{ "purged_files", "purged_bytes" }`. Keeps the definition. |
| `GET /v1/image-variants/cache` | Cache statistics, see below. |
| `POST /v1/image-variants/cache/purge` | Purge generated files by scope, see below. |
A variant is `{ name, width, height, fit, format, quality, generate_on_upload, enabled,
generated_files, cache_size, created_at, updated_at }`. `width` and `height` are 1 to 8000 or `null`
@@ -122,6 +128,27 @@ Errors are plain text: 400 for a body that is not valid JSON, 404 "Variant not f
named “thumb” already exists. Choose another name." and 422 with one sentence for the first invalid
field (name, width and height, fit, format, quality).
### Cache statistics and purge by scope
`GET /v1/image-variants/cache` answers `{ generated_files, disk_used, variants_defined,
variants_enabled, variants_disabled, rejected_requests, rejected_window_days, unused: { files, bytes,
days }, by_variant: [{ name, enabled, files, bytes }], top_images: [{ id, filename, url, files, bytes }] }`.
`by_variant` lists every defined variant by bytes descending then name; `top_images` lists the ten
images using the most variant bytes (ties by id, images that no longer exist are skipped). `unused`
counts the files of enabled variants whose reference time (the last public request, or the creation
time when never requested) is older than 30 days. `rejected_requests` counts the public requests
refused with 400 over today and the six previous UTC days. Pending request tracking is flushed before
the read. Arrays are never `null`.
`POST /v1/image-variants/cache/purge` takes `{ "scope": "all" | "unused" | "variant" | "image",
"variant": "<name>", "image_id": <id> }` (`variant` for the scope `variant`, `image_id` for `image`; a
target given for another scope is ignored) and answers `{ scope, purged_files, purged_bytes,
failed_files }`. A purge that fails on some files still answers 200 with `failed_files` above 0; the
failed rows are kept so the same request can be sent again. Errors are plain text: 400 for a body that
is not valid JSON, 422 "Scope must be all, unused, variant or image." or "A variant name or an image
id is required for this scope.", 404 "Variant not found" or "Image not found", 500 when the rows
cannot be listed.
### Variants of one image
Owners and admins only, session only: a request with an API key is 401. All ids are image ids.
+2
View File
@@ -75,6 +75,8 @@ Producers: `ArticleChanges`, `TextChange` and `BodyChange` in `internal/domain/a
Generated bytes are stored only through `image.Storage`, so they follow `STORAGE_DRIVER`, and are never rows of `images`. `ServeVariantUseCase` checks, in order: unsupported parameters, the variant (it must exist and be enabled), the source blob, SVG and animated GIF (served as they are), the size cap, the `If-None-Match` ETag, then the cached file, and generates under a `singleflight` keyed by variant, settings and source. A row whose fingerprint no longer matches, or whose blob vanished, is replaced. `UpdateVariantUseCase` purges a variant's files when a rendering setting changes; `DeleteVariantUseCase` and `PurgeVariantUseCase` purge a blob and then its row, so a failed blob delete keeps the row and a retry can finish; `mediaRemover` purges the files of a source before it deletes the source (delete and merge).
The cache screen adds three ports in `internal/domain/image`: `VariantCacheRepository` (unused totals, top sources by bytes and cursor-paged row listing, implemented by the same persistence type as `VariantFileRepository`), `VariantUsageRepository` (one batched write of request times and per-day rejection counts, and the rejection total of a window) and `VariantUsageTracker` (in-memory hits and rejections). `internal/infrastructure/usage.Tracker` implements the tracker; `GetVariantCacheStatsUseCase` and `PurgeVariantCacheUseCase` use the ports only. See [image-variants.md](image-variants.md).
### Image metadata backfill
At boot, `BackfillImageMetadataUseCase` (through `image.Repository.ListMissingMetadata` and `UpdateMetadata`) fills `filename`, `size`, `content_type`, `width`, `height` and `sha256` for image rows that lack them, reading the bytes through `image.Storage` (the files in `UPLOAD_DIR` with the local adapter, the bucket objects with the S3 adapter). It is idempotent, and a row whose file is missing or unreadable is left as it is without stopping the boot. The old `article_id` column and the `article_images` table are kept and never written.
+25
View File
@@ -56,6 +56,31 @@ file never takes a slot. The pixel cap defaults to 50000000 pixels.
- `generate_on_upload` is stored and returned only. Generating at upload time is not part of this
release.
### Request tracking and the Cache screen
`/settings/media/cache` shows what the cache holds and purges it by scope (all, unused, one variant,
one image). It reads `GET /v1/image-variants/cache` and writes `POST /v1/image-variants/cache/purge`
([api.md](api.md)).
Two things are tracked for it, without a database write per request. `usage.Tracker`
(`internal/infrastructure/usage`) keeps in memory, behind the `image.VariantUsageTracker` port, the
last time each (variant, source) file was requested and a counter of rejected requests per UTC day.
`ServeVariantUseCase` records a hit when a cached file is served, when a file is generated by the
request and on a 304; it counts a rejection when the request answers 400 (unknown or disabled
variant, unsupported parameter). A missing source (404), a request without `?v=` and the admin
generation routes record nothing. The tracker flushes one batch through `image.VariantUsageRepository`
every 30 seconds, as soon as 500 distinct files are pending, before `GET /v1/image-variants/cache`
reads, and on graceful shutdown; only `api/dependecyInjection.go` builds it and starts the loop. A
failed flush keeps the pending hits and counts, so nothing is lost or counted twice. The stored
`last_requested_at` of a file never moves back, a late update for a purged file changes nothing, and
day rows older than 7 days are deleted by the flush.
A file is unused when its reference time (`last_requested_at`, or `created_at` when never requested)
is older than 30 days and its variant is enabled. A purge reads the rows in batches of 500 and keeps
the rows whose blob cannot be deleted. Each scope records its own activity action
(`media.variant_purge_all`, `media.variant_purge_unused`, `media.variant_purge`,
`media.variant_purge_image`).
## Geometry
Images are never enlarged.