RenewCMS/docs/architecture.md

Architecture

RenewCMS is a Go HTTP API and a Vue 3 SPA, shipped as a single binary.

Backend layering

Dependencies point inward:

api/  ──▶  internal/infrastructure/useCases  ──▶  internal/domain  ◀──  internal/infrastructure/{persistence,mailer}

Layer	Location	Rules
Domain	internal/domain/<name>/	Entities, Err… sentinel errors, Repository interfaces. Imports only the stdlib and other domain packages, with no GORM tags.
Use cases	internal/infrastructure/useCases/	Application logic. Depends only on domain interfaces, never on persistence, mailer, GORM or net/http.
Adapters	internal/infrastructure/persistence/{models,mappers,repositories}, internal/infrastructure/mailer	Implement a domain Repository (assert with var _ domain.Repository = &X{}). GORM structs stay in models and are converted by mappers.
Database drivers	internal/infrastructure/persistence/database	database.Connect(Config{Driver, Source}) looks the driver up in a registry and returns a Connection (GORM handle, NewBackup, Close). A Driver implements Open and NewBackup, so a new database is one new file plus one registry entry. database.Migrate owns the AutoMigrate model list and the user-role backfill. Only sqlite exists today.
Blob storage	domain/image.Storage, internal/infrastructure/storage	Storage (Put, Open, Delete) is the port for uploaded image bytes. storage.Open(Config{Driver, Location}) picks the adapter; local is the only one. Repositories never touch files: use cases coordinate Storage and image.Repository.
Delivery	api/handlers/<name>/, api/router.go	HTTP handlers receive use cases through NewHandler(...). They never import persistence, mailer or GORM.
Composition root	api/dependecyInjection.go	The only place that picks the database and storage drivers, builds repositories and use cases, and runs database.Migrate and the image metadata backfill at boot.
Entry point	cmd/server/main.go	Env/config, HTTP server, graceful shutdown.

Nothing under internal/ imports api/.

Delivery packages

Package	Role
api/handlers/user	Authenticated user endpoints: profile update (UpdateMe) and the users list with invitations (CreateInvite, ListUsers).
api/handlers/media	The media library under /v1/media (list, upload, rename, delete, merge, usage). Images belong to the library, not to an article: deleting an article never deletes an image, and an image's "used in" is read from article bodies each time.
api/pkg/upload	Multipart file reading and the mapping of upload errors (415, 413, 400) shared by both upload routes.
api/pkg/publicuser	PublicUser, the JSON shape of a user with no password, codes or token version. publicuser.New converts a domain user.User, so handlers never serialise the domain struct directly.

Media purge

internal/domain/purge holds Settings (automatic purge on or off, interval of 1 to 365 days, last_purge_* history, enabled_at), the Repository interface (Get returns the defaults when nothing was saved, Save, and SaveLastRun, which writes only the last_purge_* columns so a settings change made during a run is never overwritten) and the sentinels ErrInvalidPurgeInterval, ErrPurgeConflict (stale fingerprint) and ErrPurgeBusy (a run is already in progress). Settings.NextPurgeAt is max(last_purge_at, enabled_at) plus the interval when automatic purge is on, and null otherwise. The settings live in a single-row purge_settings table (models.PurgeSettings, mappers.PurgeSettings*, persistence.PurgeSettingsRepository), created by AutoMigrate.

The use cases GetPurgeSettings, UpdatePurgeSettings, PreviewPurge and RunPurge depend only on domain interfaces, an injected clock and an injected *log.Logger, so a scheduler can call them. PreviewPurge and RunPurge share one plan (purgePlan.go): a duplicate group (same non-empty sha256) is planned only when a copy is used in an article, the most used copy is kept (ties: oldest created_at, then lowest id) and every other copy is merged into it. The fingerprint is a SHA-256 over the planned groups. RunPurge takes a non-blocking in-process lock (ErrPurgeBusy for a concurrent call), refuses a non-matching fingerprint with ErrPurgeConflict, merges group by group with MergeMedia (a failing group is counted and skipped), unions the UpdatedArticleIDs of the fully merged groups, and records last_purge_* after every run.

Automatic purge scheduler

PurgeSchedulerUseCase.Tick(now) returns the purge.RunResult of the run, whether it ran and an error. It reads the purge settings on every call and runs RunPurge.Run(nil) (no fingerprint) when Settings.NextPurgeAt() is set and now is not before it. It never writes settings: only RunPurge records last_purge_*, through SaveLastRun, which moves the anchor forward even when nothing was purged. The runner in internal/infrastructure/scheduler calls Tick and logs one line with groups_purged, merged, articles_updated and failed_groups when a run happened. It calls Tick once after a startup delay of 1 minute, which catches up a run missed during downtime in a single purge, then once every hour. api/dependecyInjection.go builds it and cmd/server/main.go starts it with the signal context of the server. Changes to the settings apply on the next tick, with no restart.

• Overlap guard: a manual run holds the RunPurge lock, so Tick returns ErrPurgeBusy at once. The runner treats it as a skip, logs nothing and re-evaluates at the next tick.
• Failures: any other error is logged and the loop keeps going. There is no retry inside a tick; the next hourly tick evaluates the schedule again.
• Shutdown: on SIGINT or SIGTERM the runner stops taking ticks and waits, with no time limit, for a Tick in progress. main waits for the runner to return before the database is closed.

middleware.RequireOwnerOrAdmin lets only the roles owner and admin through (403 otherwise, including an empty role). The media router mounts these routes after the verified chain and before the /{id} routes:

Route	Purpose
GET /v1/media/purge-settings	Settings with next_purge_at.
PATCH /v1/media/purge-settings	Partial update of auto_purge_enabled and auto_purge_interval_days.
GET /v1/media/purge-preview	Groups, copies, distinct articles, the first 5 groups, more and the fingerprint.
POST /v1/media/purge	Runs the purge. Optional fingerprint; 409 when stale or when a run is in progress.

Activity log

internal/domain/audit holds the append-only Event (actor, target and failure-reason snapshots, outcome, security flag, source, ip, user agent, params and changes), the closed vocabularies (actor type, category, outcome, source, change kind), the Filter, Summary and Settings types, the Repository interface (Append, List, Summary, DeleteOlderThan: no way to edit or delete one event), the SettingsRepository (Get, SaveRetention, SaveLastCleanup; no row means 90 days) and the sentinels ErrInvalidFilter, ErrInvalidRetention, ErrInvalidEvent and ErrActorNotAllowed. The Recorder port is Record(ctx, event); it returns nothing, so a caller can never fail on it.

internal/infrastructure/auditing.Recorder is the adapter. It fills a missing actor, source, ip and user agent from the audit.Origin carried by the request context, stamps created_at from the injected clock in UTC, bounds long names, drops an event outside the vocabulary, and recovers from any error or panic of the repository by writing one line to the process log (the action and the error only). middleware.AuditContext builds the origin after the credential middleware: a session gives a user actor and source web, an API key an api_key actor and source api_key, no valid credential an anonymous actor; the ip is the peer address, or X-Real-IP when the peer is a trusted proxy; the user agent is cut to 255 characters. Use cases that produce events (authenticate and logout, password reset request and completion, API key create and revoke, retention change, cleanup) take the context and an audit.Recorder; secrets (passwords, raw keys, reset codes, tokens and their hashes) are never put in an event or in its log line.

Persistence is models.AuditEvent (audit_events, indexes on created_at, (actor_type, actor_id, created_at) and (category, created_at); params and changes are JSON text) and models.AuditSettings (one row), both in AutoMigrate. The queries use only portable SQL (LOWER(...) LIKE ... ESCAPE '!', COUNT(DISTINCT ...), CASE sums), so they run on SQLite, MySQL/MariaDB and PostgreSQL. The summary splits the range into seven equal buckets, oldest first.

The use cases ListActivity, GetActivitySummary and ExportActivity turn the raw query into a domain filter (range defaults to 7 days; custom needs two dates, to included), scope a caller who is not owner or admin to its own events, and resolve target.exists at read time. The export reads the log through List in pages of 500 and writes the CSV with encoding/csv. /v1/activity and the settings PATCH sit behind RequireSession, the verifier, the current-token and verified-user checks; summary and export also behind RequireOwnerOrAdmin, the settings PATCH behind RequireOwner.

CleanupActivityUseCase.Tick(now) follows the PurgeScheduler model. With a retention of 0 it never runs. Otherwise it is due once a day, at or after 03:00 in the location of now and when the last cleanup is before today's 03:00. It deletes events older than now minus the retention days (never newer than 24 hours), records the run (activity_last_cleanup_at, activity_last_cleanup_removed) even when nothing was removed, and only then records a system.activity_cleanup event when something was removed. A failing delete returns the error and leaves the last-cleanup fields as they were, so the next tick retries. scheduler.Runner now wraps a task with a label: NewRunner keeps the purge behaviour and log line, and NewCleanupRunner ticks the cleanup. UseCases.RunSchedulers runs both runners and cmd/server/main.go starts it with the server's signal context.

Producers: ArticleChanges, TextChange and BodyChange in internal/domain/audit build the change lists (the body change holds only a signed rune count delta), and NewTarget and ReleaseTarget build the target snapshots (types article, image, user, invite, api_key, release). The use cases create, edit, publish, unpublish and delete article, upload, rename, delete, merge and purge media, register, verify, update profile, create, cancel, renew and accept invite, check, download and restart take the request context first and an audit.Recorder last, and record exactly one event after the change or the refusal (the action catalogue is in docs/api.md). Events are never written by handlers, middleware or GORM hooks, and nothing is recorded for reads or for a target that does not exist. useCases/auditFailures.go holds the fixed table of failure sentences keyed by domain Err... sentinels (matched with errors.Is, each at most 255 characters; the restart action overrides ErrJobInProgress) with the generic fallback "Something went wrong on our side. Please try again."; raw error text is never stored. The manual purge runs the groups through the non-recording MergeMediaUseCase.merge, so only one media.purge event is written; RunPurgeUseCase.Execute is the non-recording run that PurgeSchedulerUseCase.Tick(now) uses to record one system.media_purge event with the system actor and context.Background(). StartUpdateDownloadUseCase hands context.WithoutCancel(ctx) to its job goroutine so the system.update_downloaded event, written when the job becomes ready or failed, keeps the origin of the request. middleware.AuditContext is mounted on the article, user, media and system routers (after the API key middleware where keys are accepted), so API key calls are attributed to the key.

Sites

internal/domain/site holds the Site entity (name, immutable key, normalised base URL, optional default social image, index or noindex robots, optional locale, the default flag and the computed article_count), NormalizeBaseURL, SeedBaseURL, NormalizeLocale, IsValidKey, the Patch type (with Nullable for the two nullable fields), the Err... sentinels (ErrSiteNotFound, ErrSiteKeyTaken, ErrSiteHasArticles, ErrDefaultSite, ErrKeyImmutable, ErrInvalidSiteURL, ErrInvalidLocale, ErrUseSetDefault, ErrInvalidReassign) and the Repository interface. persistence.SiteRepository implements it over models.Site (sites, unique site_key); the single default is kept by the transactional operations Create (when flagged) and SetDefault, not by an index, and DeleteMovingArticles moves the articles and deletes the site in one transaction. Articles carry a site_id (indexed, no foreign key, so the SQL stays portable).

The use cases ListSites, CreateSite, UpdateSite, SetDefaultSite and DeleteSite live in internal/infrastructure/useCases; the article, media and purge use cases take the site repository as a required constructor argument, wired by the composition root (new articles take the default site, UpdateArticle moves an article and records site with site names, the media use cases count, clear and repoint default social images). api/handlers/site serves /v1/site; reads accept a session or an API key, writes are session only behind RequireOwner. database.Migrate(db, publicURL) registers the model, seeds one default site from PUBLIC_URL when the table is empty and backfills articles.site_id.

Image variants

internal/domain/image holds the Variant entity and its rules (VariantSettings.Normalized, the settings Fingerprint, VariantETag, ETagMatches, IsAnimatedGIF), the sentinels (ErrVariantNotFound, ErrVariantExists, ErrVariantInvalid, ErrVariantFormatUnavailable, ErrUnknownVariant, ErrUnsupportedParameter, ErrSourceTooLarge) and three ports: VariantRepository (definitions), VariantFileRepository (the index of generated files: variant name, source stored name, cached stored name, content type, size and the settings fingerprint, needed because Storage.Put chooses the stored name itself) and VariantRenderer (decode a source, encode a variant, report the formats it can encode). internal/infrastructure/imaging is the renderer adapter, free of GORM and HTTP; the persistence adapters are VariantRepository and VariantFileRepository over models.ImageVariant and models.ImageVariantFile.

Generated bytes are stored only through image.Storage, so they follow STORAGE_DRIVER, and are never rows of images. ServeVariantUseCase checks, in order: unsupported parameters, the variant (it must exist and be enabled), the source blob, SVG and animated GIF (served as they are), the size cap, the If-None-Match ETag, then the cached file, and generates under a singleflight keyed by variant, settings and source. A row whose fingerprint no longer matches, or whose blob vanished, is replaced. UpdateVariantUseCase purges a variant's files when a rendering setting changes; DeleteVariantUseCase and PurgeVariantUseCase purge a blob and then its row, so a failed blob delete keeps the row and a retry can finish; mediaRemover purges the files of a source before it deletes the source (delete and merge).

The cache screen adds three ports in internal/domain/image: VariantCacheRepository (unused totals, top sources by bytes and cursor-paged row listing, implemented by the same persistence type as VariantFileRepository), VariantUsageRepository (one batched write of request times and per-day rejection counts, and the rejection total of a window) and VariantUsageTracker (in-memory hits and rejections). internal/infrastructure/usage.Tracker implements the tracker; GetVariantCacheStatsUseCase and PurgeVariantCacheUseCase use the ports only. See image-variants.md.

Image metadata backfill

At boot, BackfillImageMetadataUseCase (through image.Repository.ListMissingMetadata and UpdateMetadata) fills filename, size, content_type, width, height and sha256 for image rows that lack them, reading the bytes through image.Storage (the files in UPLOAD_DIR with the local adapter, the bucket objects with the S3 adapter). It is idempotent, and a row whose file is missing or unreadable is left as it is without stopping the boot. The old article_id column and the article_images table are kept and never written.

Frontend ↔ backend

web/ (Vue, Vite)
  │  task build:web  (vue-tsc + vite build)
  ▼
internal/infrastructure/assets/dist/        gitignored build output
  │  //go:embed all:dist   (assets_embed.go, default build)
  ▼
bin/renewcms  ──  /v1/*  API
               ──  /healthz
               ──  /*    SPA (api/handlers/frontend), index.html fallback

• Production: a single origin. The SPA calls the API with relative URLs, so the JWT cookie works without CORS. Files under assets/ are content-hashed by Vite and cached for a year. index.html is never cached.
• Development: built with -tags nofrontend (assets_stub.go, assets.Dist == nil). Vite serves the SPA on :5173 and proxies /v1 to the API on :8080. See development.md.

Storage and database seams

Uploaded images are addressed by a stored name (<uuid>.<ext>). CreateImageUseCase calls Storage.Put (which generates the name), then Repository.Create with Path = /static/uploadedImages/<name>, and deletes the blob again if the row cannot be created. DeleteMediaUseCase and MergeMediaUseCase remove the blob first and the row second (mediaRemover), so a failed blob delete keeps the row and the delete can be retried. GET /static/uploadedImages/<name> is served by OpenMediaUseCase through api/handlers/staticimages, which uses http.ServeContent when the blob is seekable and streams it otherwise; ErrBlobNotFound maps to 404 and any other error to 500.

SQL in repositories must run on SQLite, MySQL/MariaDB and PostgreSQL: substring searches use LIKE ... ESCAPE '!' through containsPattern instead of instr. Settings.DataDir() is where update state, staging and backups live; it is DATA_DIR when set and the directory of DB_FILE otherwise, so it does not depend on where the database is.

Three drivers are registered in database.drivers: sqlite (default), mysql (MySQL 8 and MariaDB 11, through gorm.io/driver/mysql) and postgres (PostgreSQL 16 and 17, alias postgresql, through gorm.io/driver/postgres on pgx v5); all are pure Go so CGO_ENABLED=0 and the six release targets still build. DB_DRIVER picks one in databaseConfig (composition root); an unknown name fails in database.Connect with unsupported database driver "<name>". database.MySQLDSN and database.PostgresDSN build the DSN from separate variables (the PostgreSQL one as an escaped URL with sslmode, timezone=UTC and a connect timeout), and the driver redacts the password from every error and names only host and port. Connect applies the optional PoolSettings to any driver. What keeps the repositories identical on both engines, all without touching the domain, use cases or handlers:

• Indexed and unique strings carry size: tags in the models (a MySQL index needs a bounded varchar). SQLite maps them to the same text column, so an existing SQLite file migrates with no schema or row change (test/sqlite_schema_unchanged_test.go builds a file from the old model definitions and compares sqlite_master and every row).
• Migrate creates MySQL tables with utf8mb4_bin (gorm:table_options), so Bob and bob are distinct like in SQLite. The DSN also sets collation=utf8mb4_bin.
• The DSN sets clientFoundRows=true: an update that matches a row but changes nothing still reports one affected row, as in SQLite. InviteRepository.Renew, deleteOrNotFound and PurgeSettingsRepository.SaveLastRun depend on that.
• The Go driver sends a zero time.Time as 0000-00-00, which MySQL's default NO_ZERO_DATE rejects, so each session sets sql_mode to the server default minus NO_ZERO_DATE. Reading 0000-00-00 gives the zero time back.
• UserRepository.Create returns the filled model instead of re-reading it, because MySQL has no RETURNING.

On PostgreSQL the MySQL-only table options are not applied, pgx returns timestamptz values in the local zone so a query callback in the driver converts them to UTC, and only the public schema is used.

Backups are a domain/update.DatabaseBackup per driver. SQLite uses VACUUM INTO. MySQL uses DumpBackup, a pure Go logical dump (SHOW CREATE TABLE plus every row, strings escaped and binary values as hex, NULL and zero dates kept) written inside one repeatable-read transaction to DATA_DIR/backups/renewcms-<time>.sql.partial and renamed once complete. Any failure removes the partial file and returns an error, so the update download fails instead of claiming a backup. PostgreSQL reuses DumpBackup with its own dumpDialect (double-quoted identifiers, doubled single quotes, \x bytea literals): CREATE TABLE is rebuilt from pg_catalog (serial columns, defaults, primary key, unique and check constraints), followed by the indexes, the rows and a setval per id sequence, all inside one repeatable-read read-only transaction. All drivers keep the newest 3 files (pruneBackups).

task test runs everything that needs no server. The repository contract tests (test/repository_contract_test.go) run on SQLite always, and on MySQL when RENEWCMS_TEST_MYSQL_DSN points at a server (admin DSN; each test creates and drops its own database). task test:mysql starts mysql:8.4 and then mariadb:11 in Docker and runs the MySQL tests against each. The same contract tests run on PostgreSQL when RENEWCMS_TEST_POSTGRES_DSN points at a server (admin DSN, for example postgres://postgres:pw@127.0.0.1:5432/postgres?sslmode=disable), and task test:postgres starts postgres:16 then postgres:17 in Docker and runs the Postgres tests against each.

Two storage adapters implement image.Storage: local (default, files in UPLOAD_DIR) and s3 (storage.S3Storage, aws-sdk-go-v2 over plain HTTPS, pure Go so CGO_ENABLED=0 still holds). STORAGE_DRIVER picks one in imageStorageConfig (composition root, from config.StorageSettings); storage.Open fails with unsupported storage driver "<name>" for anything else, and openImageStorage panics with the error. The S3 adapter works as follows, with no change to the domain, use cases, repositories or the staticimages handler:

• Open calls GetObject and returns a Blob whose Content is an io.ReadSeeker (s3Reader): Size and ModTime come from ContentLength and LastModified, the first bytes are read from that response and every Seek to another offset makes the next Read issue a ranged GetObject. http.ServeContent therefore answers Range (206, 416) and If-Modified-Since (304) with no ETag. A 404 maps to image.ErrBlobNotFound; any other failure is a plain error (500 in the handler).
• Put does one PutObject of <S3_KEY_PREFIX><uuid>.<ext> with the upload's Content-Type. Delete treats 404 as success. Names that are not plain file names are never requested, as in the local adapter.
• Each call has a 30 second timeout to get a response (the body then streams without a deadline) and the SDK default retries. Request checksums are only sent when required, so older S3-compatible servers work.
• storage.Open checks the bucket with HeadBucket. The failure text names the bucket and the reason (404, 403, other status or transport error) and never the credentials.
• At boot, RunBackfillReportingReads wraps the storage in a counter for the BackfillImageMetadataUseCase run and logs how many images it read; the use case itself is unchanged.
• Update backups (DATA_DIR) never include uploads, whatever the driver.

task test:s3 starts SeaweedFS in Docker and runs TestS3StorageIntegration against it (skipped without Docker, and not part of task test). Everything else uses an in-process fake S3 server listening on 127.0.0.1.

Adding a database driver: implement database.Driver (Open, and NewBackup returning a domain/update.DatabaseBackup), register it in drivers, then select it in databaseConfig in the composition root (and add the env vars to main.go, .env.example and configuration.md). Adding a storage backend: implement image.Storage, add a case to storage.Open, and select it in openImageStorage. No use case, handler or repository changes.