feat: image variants cache panel
This commit is contained in:
1 parent
951fef18ba
commit
d768668014
53 files changed
+5809
-46
No files matched your search
+28
-1
@@ -62,7 +62,7 @@ never changes the response or the change itself.
|
||||
| Category | Actions |
|
||||
| --- | --- |
|
||||
| `articles` | `article.create`, `article.edit`, `article.publish`, `article.unpublish`, `article.delete` |
|
||||
| `media` | `media.upload`, `media.rename`, `media.delete`, `media.merge`, `media.purge`, `media.variant_create`, `media.variant_update`, `media.variant_delete`, `media.variant_enable`, `media.variant_disable`, `media.variant_purge`, `media.variant_generate_all`, `media.variant_purge_image` |
|
||||
| `media` | `media.upload`, `media.rename`, `media.delete`, `media.merge`, `media.purge`, `media.variant_create`, `media.variant_update`, `media.variant_delete`, `media.variant_enable`, `media.variant_disable`, `media.variant_purge`, `media.variant_generate_all`, `media.variant_purge_image`, `media.variant_purge_all`, `media.variant_purge_unused` |
|
||||
| `users` | `user.register`, `user.verify`, `user.update_profile`, `invite.create`, `invite.cancel`, `invite.renew`, `invite.accept` |
|
||||
| `system` | `system.media_purge` (scheduled), `system.update_checked`, `system.update_downloaded`, `system.restart_requested`, `system.activity_cleanup`, `system.activity_retention_changed` |
|
||||
|
||||
@@ -77,6 +77,10 @@ name) records nothing. Params: `media.merge` and `media.purge` and `system.media
|
||||
`unsupported_format` or `failed`) and is `failed` only when nothing was generated and something failed;
|
||||
`media.variant_purge_image` carries `purged_files` and `purged_bytes`; both have an `image` target
|
||||
and are recorded only when files were generated or removed;
|
||||
`media.variant_purge_all` and `media.variant_purge_unused` (no target) carry `purged_files` and
|
||||
`purged_bytes`, the latter also `older_than_days`; a partial purge adds `failed_files` and stays
|
||||
`success`, and a purge that removed nothing while some files failed is `failed`; nothing is recorded
|
||||
when no file was removed or failed;
|
||||
`system.update_checked` carries `update_available`; `system.restart_requested` carries `kind` and,
|
||||
for an update, `tag` (it has no target). `system.update_downloaded` is recorded when the job reaches
|
||||
`ready` or `failed`, with the origin captured at request time; a refusal at the start is recorded
|
||||
@@ -110,6 +114,8 @@ this." Behaviour, caching and the public `?v=<name>` contract are in [image-vari
|
||||
| `PATCH /v1/image-variants/{name}` | `{ "enabled": true }` or `{ "enabled": false }`. Keeps the generated files. |
|
||||
| `DELETE /v1/image-variants/{name}` | 204. Removes the definition and its generated files. |
|
||||
| `POST /v1/image-variants/{name}/purge` | `{ "purged_files", "purged_bytes" }`. Keeps the definition. |
|
||||
| `GET /v1/image-variants/cache` | Cache statistics, see below. |
|
||||
| `POST /v1/image-variants/cache/purge` | Purge generated files by scope, see below. |
|
||||
|
||||
A variant is `{ name, width, height, fit, format, quality, generate_on_upload, enabled,
|
||||
generated_files, cache_size, created_at, updated_at }`. `width` and `height` are 1 to 8000 or `null`
|
||||
@@ -122,6 +128,27 @@ Errors are plain text: 400 for a body that is not valid JSON, 404 "Variant not f
|
||||
named “thumb” already exists. Choose another name." and 422 with one sentence for the first invalid
|
||||
field (name, width and height, fit, format, quality).
|
||||
|
||||
### Cache statistics and purge by scope
|
||||
|
||||
`GET /v1/image-variants/cache` answers `{ generated_files, disk_used, variants_defined,
|
||||
variants_enabled, variants_disabled, rejected_requests, rejected_window_days, unused: { files, bytes,
|
||||
days }, by_variant: [{ name, enabled, files, bytes }], top_images: [{ id, filename, url, files, bytes }] }`.
|
||||
`by_variant` lists every defined variant by bytes descending then name; `top_images` lists the ten
|
||||
images using the most variant bytes (ties by id, images that no longer exist are skipped). `unused`
|
||||
counts the files of enabled variants whose reference time (the last public request, or the creation
|
||||
time when never requested) is older than 30 days. `rejected_requests` counts the public requests
|
||||
refused with 400 over today and the six previous UTC days. Pending request tracking is flushed before
|
||||
the read. Arrays are never `null`.
|
||||
|
||||
`POST /v1/image-variants/cache/purge` takes `{ "scope": "all" | "unused" | "variant" | "image",
|
||||
"variant": "<name>", "image_id": <id> }` (`variant` for the scope `variant`, `image_id` for `image`; a
|
||||
target given for another scope is ignored) and answers `{ scope, purged_files, purged_bytes,
|
||||
failed_files }`. A purge that fails on some files still answers 200 with `failed_files` above 0; the
|
||||
failed rows are kept so the same request can be sent again. Errors are plain text: 400 for a body that
|
||||
is not valid JSON, 422 "Scope must be all, unused, variant or image." or "A variant name or an image
|
||||
id is required for this scope.", 404 "Variant not found" or "Image not found", 500 when the rows
|
||||
cannot be listed.
|
||||
|
||||
### Variants of one image
|
||||
Owners and admins only, session only: a request with an API key is 401. All ids are image ids.
|
||||
|
||||
|
||||
@@ -75,6 +75,8 @@ Producers: `ArticleChanges`, `TextChange` and `BodyChange` in `internal/domain/a
|
||||
|
||||
Generated bytes are stored only through `image.Storage`, so they follow `STORAGE_DRIVER`, and are never rows of `images`. `ServeVariantUseCase` checks, in order: unsupported parameters, the variant (it must exist and be enabled), the source blob, SVG and animated GIF (served as they are), the size cap, the `If-None-Match` ETag, then the cached file, and generates under a `singleflight` keyed by variant, settings and source. A row whose fingerprint no longer matches, or whose blob vanished, is replaced. `UpdateVariantUseCase` purges a variant's files when a rendering setting changes; `DeleteVariantUseCase` and `PurgeVariantUseCase` purge a blob and then its row, so a failed blob delete keeps the row and a retry can finish; `mediaRemover` purges the files of a source before it deletes the source (delete and merge).
|
||||
|
||||
The cache screen adds three ports in `internal/domain/image`: `VariantCacheRepository` (unused totals, top sources by bytes and cursor-paged row listing, implemented by the same persistence type as `VariantFileRepository`), `VariantUsageRepository` (one batched write of request times and per-day rejection counts, and the rejection total of a window) and `VariantUsageTracker` (in-memory hits and rejections). `internal/infrastructure/usage.Tracker` implements the tracker; `GetVariantCacheStatsUseCase` and `PurgeVariantCacheUseCase` use the ports only. See [image-variants.md](image-variants.md).
|
||||
|
||||
### Image metadata backfill
|
||||
|
||||
At boot, `BackfillImageMetadataUseCase` (through `image.Repository.ListMissingMetadata` and `UpdateMetadata`) fills `filename`, `size`, `content_type`, `width`, `height` and `sha256` for image rows that lack them, reading the bytes through `image.Storage` (the files in `UPLOAD_DIR` with the local adapter, the bucket objects with the S3 adapter). It is idempotent, and a row whose file is missing or unreadable is left as it is without stopping the boot. The old `article_id` column and the `article_images` table are kept and never written.
|
||||
|
||||
@@ -56,6 +56,31 @@ file never takes a slot. The pixel cap defaults to 50000000 pixels.
|
||||
- `generate_on_upload` is stored and returned only. Generating at upload time is not part of this
|
||||
release.
|
||||
|
||||
### Request tracking and the Cache screen
|
||||
|
||||
`/settings/media/cache` shows what the cache holds and purges it by scope (all, unused, one variant,
|
||||
one image). It reads `GET /v1/image-variants/cache` and writes `POST /v1/image-variants/cache/purge`
|
||||
([api.md](api.md)).
|
||||
|
||||
Two things are tracked for it, without a database write per request. `usage.Tracker`
|
||||
(`internal/infrastructure/usage`) keeps in memory, behind the `image.VariantUsageTracker` port, the
|
||||
last time each (variant, source) file was requested and a counter of rejected requests per UTC day.
|
||||
`ServeVariantUseCase` records a hit when a cached file is served, when a file is generated by the
|
||||
request and on a 304; it counts a rejection when the request answers 400 (unknown or disabled
|
||||
variant, unsupported parameter). A missing source (404), a request without `?v=` and the admin
|
||||
generation routes record nothing. The tracker flushes one batch through `image.VariantUsageRepository`
|
||||
every 30 seconds, as soon as 500 distinct files are pending, before `GET /v1/image-variants/cache`
|
||||
reads, and on graceful shutdown; only `api/dependecyInjection.go` builds it and starts the loop. A
|
||||
failed flush keeps the pending hits and counts, so nothing is lost or counted twice. The stored
|
||||
`last_requested_at` of a file never moves back, a late update for a purged file changes nothing, and
|
||||
day rows older than 7 days are deleted by the flush.
|
||||
|
||||
A file is unused when its reference time (`last_requested_at`, or `created_at` when never requested)
|
||||
is older than 30 days and its variant is enabled. A purge reads the rows in batches of 500 and keeps
|
||||
the rows whose blob cannot be deleted. Each scope records its own activity action
|
||||
(`media.variant_purge_all`, `media.variant_purge_unused`, `media.variant_purge`,
|
||||
`media.variant_purge_image`).
|
||||
|
||||
## Geometry
|
||||
|
||||
Images are never enlarged.
|
||||
|
||||
Reference in new issue
Block a user