feat: optional MySQL/MariaDB database driver for deployments that cannot use local SQLite

2026-10-06 04:08:33 +02:00
1 parent 81e299f09a
commit d752a61e52
@@ -80,6 +80,18 @@ bin/renewcms  ──  /v1/*  API

 Uploaded images are addressed by a stored name (`<uuid>.<ext>`). `CreateImageUseCase` calls `Storage.Put` (which generates the name), then `Repository.Create` with `Path = /static/uploadedImages/<name>`, and deletes the blob again if the row cannot be created. `DeleteMediaUseCase` and `MergeMediaUseCase` remove the blob first and the row second (`mediaRemover`), so a failed blob delete keeps the row and the delete can be retried. `GET /static/uploadedImages/<name>` is served by `OpenMediaUseCase` through `api/handlers/staticimages`, which uses `http.ServeContent` when the blob is seekable and streams it otherwise; `ErrBlobNotFound` maps to 404 and any other error to 500.

-SQL in repositories must run on SQLite, MySQL/MariaDB and PostgreSQL: substring searches use `LIKE ... ESCAPE '!'` through `containsPattern` instead of `instr`. `Settings.DataDir()` is where update state, staging and backups live; it is derived from `DB_FILE` today and does not depend on where the database is.
+SQL in repositories must run on SQLite, MySQL/MariaDB and PostgreSQL: substring searches use `LIKE ... ESCAPE '!'` through `containsPattern` instead of `instr`. `Settings.DataDir()` is where update state, staging and backups live; it is `DATA_DIR` when set and the directory of `DB_FILE` otherwise, so it does not depend on where the database is.

-Adding a database driver: implement `database.Driver` (`Open`, and `NewBackup` returning a `domain/update.DatabaseBackup`), register it in `drivers`, then select it in `openDatabase` in the composition root (and add the env vars to `main.go`, `.env.example` and `configuration.md`). Adding a storage backend: implement `image.Storage`, add a case to `storage.Open`, and select it in `openImageStorage`. No use case, handler or repository changes.
+Two drivers are registered in `database.drivers`: `sqlite` (default) and `mysql` (MySQL 8 and MariaDB 11, through `gorm.io/driver/mysql`, pure Go so `CGO_ENABLED=0` and the six release targets still build). `DB_DRIVER` picks one in `databaseConfig` (composition root); an unknown name fails in `database.Connect` with `unsupported database driver "<name>"`. `database.MySQLDSN` builds the DSN from separate variables, and the driver redacts the password from every error and names only host and port. `Connect` applies the optional `PoolSettings` to any driver. What keeps the repositories identical on both engines, all without touching the domain, use cases or handlers:
+
+- Indexed and unique strings carry `size:` tags in the models (a MySQL index needs a bounded `varchar`). SQLite maps them to the same `text` column, so an existing SQLite file migrates with no schema or row change (`test/sqlite_schema_unchanged_test.go` builds a file from the old model definitions and compares `sqlite_master` and every row).
+- `Migrate` creates MySQL tables with `utf8mb4_bin` (`gorm:table_options`), so `Bob` and `bob` are distinct like in SQLite. The DSN also sets `collation=utf8mb4_bin`.
+- The DSN sets `clientFoundRows=true`: an update that matches a row but changes nothing still reports one affected row, as in SQLite. `InviteRepository.Renew`, `deleteOrNotFound` and `PurgeSettingsRepository.SaveLastRun` depend on that.
+- The Go driver sends a zero `time.Time` as `0000-00-00`, which MySQL's default `NO_ZERO_DATE` rejects, so each session sets `sql_mode` to the server default minus `NO_ZERO_DATE`. Reading `0000-00-00` gives the zero time back.
+- `UserRepository.Create` returns the filled model instead of re-reading it, because MySQL has no `RETURNING`.
+
+Backups are a `domain/update.DatabaseBackup` per driver. SQLite uses `VACUUM INTO`. MySQL uses `DumpBackup`, a pure Go logical dump (`SHOW CREATE TABLE` plus every row, strings escaped and binary values as hex, NULL and zero dates kept) written inside one repeatable-read transaction to `DATA_DIR/backups/renewcms-<time>.sql.partial` and renamed once complete. Any failure removes the partial file and returns an error, so the update download fails instead of claiming a backup. Both keep the newest 3 files (`pruneBackups`).
+
+`task test` runs everything that needs no server. The repository contract tests (`test/repository_contract_test.go`) run on SQLite always, and on MySQL when `RENEWCMS_TEST_MYSQL_DSN` points at a server (admin DSN; each test creates and drops its own database). `task test:mysql` starts `mysql:8.4` and then `mariadb:11` in Docker and runs the `MySQL` tests against each.
+
+Adding a database driver: implement `database.Driver` (`Open`, and `NewBackup` returning a `domain/update.DatabaseBackup`), register it in `drivers`, then select it in `databaseConfig` in the composition root (and add the env vars to `main.go`, `.env.example` and `configuration.md`). Adding a storage backend: implement `image.Storage`, add a case to `storage.Open`, and select it in `openImageStorage`. No use case, handler or repository changes.
@@ -24,17 +24,41 @@ Copy [`.env.example`](../.env.example) to `.env` to get started. When a required
 |------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `ENVIRONMENT`          | `production`         | `production` or `development`. In `production` the auth cookie is `Secure`, so it is only sent over HTTPS. `development` turns on HTTP request logging.                                                                           |
 | `PORT`                 | `8080`               | Port the HTTP server listens on (all interfaces).                                                                                                                                                                               |
-| `DB_FILE`              | `./data/renewcms.db` | Path to the SQLite database file. Missing parent directories are created. The schema is migrated automatically at startup.                                                                                                      |
+| `DB_DRIVER`            | `sqlite`             | `sqlite` or `mysql` (MySQL and MariaDB). Any other value stops the server with `unsupported database driver "<value>"`. See [Database](#database).                                                                             |
+| `DB_FILE`              | `./data/renewcms.db` | Path to the SQLite database file. Missing parent directories are created. The schema is migrated automatically at startup. Ignored when `DB_DRIVER=mysql`.                                                                     |
+| `DATA_DIR`             | directory of `DB_FILE` | Directory for update state, staged updates and database backups (`backups/`). Set it with `DB_DRIVER=mysql`, where there is no database file to sit beside (the default still follows `DB_FILE`). With SQLite, leave it unset to keep everything beside the database. |
 | `UPLOAD_DIR`           | `./data/uploads`     | Directory where uploaded images are stored.                                                                                                                                                                                     |
 | `CORS_ALLOWED_ORIGINS` | *(empty)*            | Semicolon-separated list of origins allowed to call the API. When empty, no CORS headers are sent. That is the right setting whenever the SPA is served by RenewCMS itself (production) or through the Vite proxy (development). |
 | `AUTH_RATE_LIMIT_MAX_ATTEMPTS` | `5`           | Maximum number of requests a single client IP may send to `POST /v1/auth/login`, `POST /v1/auth/password-reset` or the invite-accept pair (`GET /v1/auth/invite` and `POST /v1/auth/invite/accept`, sharing one counter) within the window, tracked independently per route.                                                          |
 | `AUTH_RATE_LIMIT_WINDOW_SECONDS` | `60`        | Length, in seconds, of the fixed rate-limit window for the routes above.                                                                                                                                                    |
 | `TRUSTED_PROXIES`      | *(empty)*            | Semicolon-separated list of exact IP addresses (no CIDR) of reverse proxies allowed to supply the client IP via `X-Real-IP` for the rate limiting of the routes above. When a request's connection address is not in this list, any `X-Real-IP` or `X-Forwarded-For` header it sends is ignored and the connection address is used instead. See [deployment.md](deployment.md).      |
 | `RENEWCMS_CONTAINER` | `false` | Set to `true` (case-insensitive; any other value means `false`) when the app runs in a container. Settings then tells the owner to update by pulling a new image instead of offering browser updates, and `can_apply` is always `false`. The Dockerfile and `docker-compose.yml` set it to `true`. Update checks still work. |
-| `UPDATE_FEED_URL`      | `https://git.floriansylvain.fr` | Base URL of the Gitea server that publishes RenewCMS releases. The Settings page asks `{UPDATE_FEED_URL}/api/v1/repos/fsylvain/RenewCMS/releases` when the owner checks for updates. The last result is kept in `update-status.json`, next to `DB_FILE`. |
+| `UPDATE_FEED_URL`      | `https://git.floriansylvain.fr` | Base URL of the Gitea server that publishes RenewCMS releases. The Settings page asks `{UPDATE_FEED_URL}/api/v1/repos/fsylvain/RenewCMS/releases` when the owner checks for updates. The last result is kept in `update-status.json`, in the data directory. |

 Automatic media purge has no environment variable. Its schedule (on or off, interval, `enabled_at`, `last_purge_at`) lives in the purge settings row of the database, is changed from the Settings page or `PATCH /v1/media/purge-settings`, and is read again at every hourly tick.

+## Database
+
+SQLite is the default and needs nothing but `DB_FILE`. To run against MySQL 8 or MariaDB 11 instead, set `DB_DRIVER=mysql` and the connection variables below. They are only read (and the four required ones only checked) when `DB_DRIVER=mysql`. A required variable that is not set stops the server with `Environment variable DB_HOST is not set` (or `DB_USER`, `DB_PASSWORD`, `DB_NAME`).
+
+| Name                           | Default  | Description                                                                                                                      |
+|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------|
+| `DB_HOST`                      | required | Server host name or IP address.                                                                                                  |
+| `DB_PORT`                      | `3306`   | Server port.                                                                                                                     |
+| `DB_USER`                      | required | Database user. It needs full rights on `DB_NAME`, because the schema is created and migrated at startup.                        |
+| `DB_PASSWORD`                  | required | Database password. It is never logged and never appears in an error message.                                                     |
+| `DB_NAME`                      | required | Name of an existing database. The server does not create it.                                                                     |
+| `DB_TLS`                       | `false`  | `false`, `true`, `skip-verify` or `preferred`. Any other value stops the server with an error naming `DB_TLS` and these values. `true` verifies the server certificate; `skip-verify` encrypts without verifying it. |
+| `DB_MAX_OPEN_CONNS`            | `10`     | Maximum open connections in the pool. `0` means unlimited.                                                                       |
+| `DB_MAX_IDLE_CONNS`            | `5`      | Maximum idle connections kept in the pool.                                                                                       |
+| `DB_CONN_MAX_LIFETIME_SECONDS` | `300`    | Seconds after which a connection is closed and replaced. `0` keeps connections forever. Keep it below the server's `wait_timeout`. |
+
+A non numeric (or negative) pool value stops the server with an error naming the variable. There is no DSN variable: the connection string is built from the variables above, with `parseTime=true`, `loc=UTC`, `charset=utf8mb4`, `collation=utf8mb4_bin` and `clientFoundRows=true`. Connecting uses a 5 second timeout, so a wrong host, port, password or database name stops startup quickly with `Unable to open the database`, naming the host and port but never the password.
+
+Why those settings: tables are created with the `utf8mb4_bin` collation so that usernames and emails compare case-sensitively exactly like in SQLite (`Bob` and `bob` are different users). The collation of `DB_NAME` itself does not matter, but tables that already exist keep the collation they have. `clientFoundRows` makes an update that does not change a row still count as matched, which the invite and purge-settings code relies on. Each session also drops the server's `NO_ZERO_DATE` SQL mode, because the Go driver sends a zero time as `0000-00-00`. The other strict modes stay on.
+
+With MySQL the data directory (`DATA_DIR`) holds update state and backups only. The pre-update backup is a `renewcms-<time>.sql` logical dump (schema and rows, written by RenewCMS itself, no `mysqldump` needed) that keeps the newest 3. If the dump fails, the update download fails with it. The dump holds your data, such as password hashes, but not the connection password.
+
 Relative paths are resolved against the process's working directory. In production, use absolute paths (see [deployment.md](deployment.md)).

 For example, `WorkingDirectory=/var/lib/renewcms` with `DB_FILE=./data/renewcms.db` puts the database in `/var/lib/renewcms/data`. Browser-update `--data-dir` must name that directory, because the app reads helper status and update records beside the database. It is not necessarily `WorkingDirectory` or `ReadWritePaths`. Do not change an existing `DB_FILE` merely to match the helper: that could open a different, empty database. See [browser-update setup](deployment.md#browser-updates).
@@ -42,7 +66,7 @@ For example, `WorkingDirectory=/var/lib/renewcms` with `DB_FILE=./data/renewcms.
 ## Adding a variable

 1. Read it in the code where it is needed (infrastructure or `api/`, never `internal/domain`).
-2. Register it in [`cmd/server/main.go`](../cmd/server/main.go): add it to `requiredEnvVars` if it is required, or to `envDefaults` if it has a default.
+2. Register it in [`cmd/server/main.go`](../cmd/server/main.go): add it to `requiredEnvVars` if it is required (or to `DatabaseRequiredEnvVars` in `internal/infrastructure/config` if only one database driver needs it), or to `envDefaults` if it has a default.
 3. Add it to [`.env.example`](../.env.example) and to the tables above.

 Never hardcode a secret, and never commit `.env`.
@@ -8,7 +8,7 @@ RenewCMS ships as **one self-contained executable**. It serves:
 | `/healthz` | Liveness check, returns `200 ok`        |
 | `/*`       | The Vue SPA (embedded), with fallback to `index.html` for client-side routes |

-Runtime state includes the SQLite database (`DB_FILE`), uploads (`UPLOAD_DIR`), and update records, staged binaries and backups beside the database. Running a release binary needs no Go, Bun or Node installation. Browser updates additionally require Linux with systemd and the helper setup below.
+Runtime state includes the SQLite database (`DB_FILE`, or a MySQL/MariaDB server, see [Using MySQL or MariaDB](#using-mysql-or-mariadb)), uploads (`UPLOAD_DIR`), and update records, staged binaries and backups beside the database. Running a release binary needs no Go, Bun or Node installation. Browser updates additionally require Linux with systemd and the helper setup below.

 To run it in a container instead, see [docker.md](docker.md). The published image `git.floriansylvain.fr/fsylvain/renewcms` is upgraded with `docker compose pull && docker compose up -d`. Tags are `vX.Y.Z`, `X.Y.Z`, `X.Y` and `latest` for a stable release, and `vX.Y.Z-<pre>`, `X.Y.Z-<pre>` and the moving `<pre-name>` (`beta`, `rc`) for a pre-release, which never moves `latest`. Maintainers publish with `task docker:publish` after `docker login git.floriansylvain.fr`; there is no CI. See [Maintainer: publishing](docker.md#maintainer-publishing).

@@ -276,8 +276,37 @@ RenewCMS sends `Content-Security-Policy` (including `frame-ancestors 'none'`), `

 The CSP allows Google Fonts (`fonts.googleapis.com`, `fonts.gstatic.com`) and `https:` images, because the SPA loads its fonts from Google and articles may embed remote images. `style-src` keeps `'unsafe-inline'` for the article editor; `script-src` is `'self'` only.

+## Using MySQL or MariaDB
+
+SQLite is the default. To use an existing MySQL 8 or MariaDB 11 server instead, create an empty database and a user with full rights on it, then set in `/etc/renewcms/renewcms.env`:
+
+```sh
+DB_DRIVER=mysql
+DB_HOST=db.example.com
+DB_USER=renewcms
+DB_PASSWORD=<generated password>
+DB_NAME=renewcms
+DB_TLS=true
+DATA_DIR=/var/lib/renewcms
+```
+
+`DB_FILE` is then ignored. `DATA_DIR` is where update state, staged updates and backups live, so give it the same value as the helper's `--data-dir` (see [browser updates](#browser-updates)). Use `DB_TLS=true` (or `preferred`) whenever the server is not on the same host. Every variable is described in [configuration.md](configuration.md#database). The schema is created and migrated at startup, so the first start on an empty database needs nothing else.
+
+### Moving an existing SQLite site to MySQL
+
+There is no migration tool. The manual procedure, with the site stopped:
+
+1. Take a backup of the data directory (database file and uploads).
+2. Start RenewCMS once on the empty MySQL database so it creates the schema, then stop it.
+3. Export the rows from the SQLite file with `sqlite3 renewcms.db .dump`, keep only the `INSERT INTO` statements for `articles`, `images`, `users`, `invites`, `api_keys` and `purge_settings`, and import them into MySQL. Quoting and date formats differ between the two engines, so check a sample first: SQLite writes times such as `2026-01-02 03:04:05.123456789+00:00`, and the `+00:00` suffix must be removed for MariaDB and older MySQL versions. An unset time (`0001-01-01 00:00:00`) is valid as it is in MySQL and is read back as unset.
+4. Copy `uploads/` to the new `UPLOAD_DIR`, set `DB_DRIVER=mysql` and the connection variables, and start the service.
+
+Keep the SQLite file until you have checked the site.
+
 ## Backups

+With MySQL or MariaDB, back the database up with your server's own tools (for example `mysqldump --single-transaction`), plus the uploads directory. RenewCMS also writes a logical dump to `DATA_DIR/backups` before each update download. The rest of this section is about SQLite.
+
 Everything worth backing up is under the data directory. Take a consistent SQLite copy while the service is running:

 ```sh
@@ -55,6 +55,7 @@ If your editor reports `pattern all:dist: no matching files found`, either run `
 ```sh
 task --list          # everything available
 task test            # Go tests; extra args pass through: task test -- -run TestCreateArticle -v
+task test:mysql      # MySQL tests against real servers in Docker (mysql:8.4, then mariadb:11)
 task cover           # coverage of the use cases → tmp/coverage.out
 task mutate          # gremlins mutation testing → tmp/gremlins.json
 task test:web        # SPA component tests (Vitest)
@@ -62,7 +62,7 @@ The container receives SIGTERM on stop, and RenewCMS shuts down gracefully withi

 ## Data volume

-State lives on the named volume `renewcms-data` (Compose prefixes it with the project name, usually `renewcms_renewcms-data`; check with `docker volume ls`). It holds the SQLite database, uploads, and update records.
+State lives on the named volume `renewcms-data` (Compose prefixes it with the project name, usually `renewcms_renewcms-data`; check with `docker volume ls`). It holds the SQLite database (unless you use MySQL), uploads, and update records.

 - `docker compose down` removes the container and keeps the volume. The data survives and `docker compose up -d` brings it back.
 - **`docker compose down -v` deletes the volume and all data.** The next start is an empty instance.
@@ -86,6 +86,22 @@ docker run --rm -v VOLUME:/data -v "$PWD":/backup alpine sh -c "tar xzf /backup/
 docker compose up -d
 ```

+## MySQL or MariaDB
+
+SQLite on the `/data` volume is the default. [`docker-compose.mysql.yml`](../docker-compose.mysql.yml) runs RenewCMS next to a MySQL 8.4 container instead:
+
+```sh
+export MYSQL_ROOT_PASSWORD=<generated>
+export DB_PASSWORD=<generated>
+docker compose -f docker-compose.mysql.yml up -d
+```
+
+It starts the `mysql` service with a named volume `mysql-data` and a healthcheck, and starts `renewcms` only once the database is healthy (`depends_on` with `service_healthy`). RenewCMS gets `DB_DRIVER=mysql`, `DB_HOST=mysql`, `DB_NAME`, `DB_USER` and `DB_PASSWORD` from the compose file and the rest from `docker.env`. `DATA_DIR=/data` keeps update state and the `.sql` pre-update backups on the `renewcms-data` volume, which still holds the uploads. Replace the placeholders; never commit real passwords.
+
+To use MariaDB, change the `mysql` service image to `mariadb:11`, its `MYSQL_*` variables to `MARIADB_*` and its healthcheck to `healthcheck.sh --connect --innodb_initialized`. To use a server outside Compose, drop the `mysql` service and point `DB_HOST` at it. For a server on another machine set `DB_TLS=true`.
+
+Back the database up with `docker compose -f docker-compose.mysql.yml exec mysql mysqldump ...` (a `tar` copy of the volume is not consistent while it runs), and the `renewcms-data` volume as described above. Moving an existing SQLite volume is a manual procedure, see [deployment.md](deployment.md#moving-an-existing-sqlite-site-to-mysql).
+
 ## Reverse proxy and TLS

 With `ENVIRONMENT=production` the site must be served over HTTPS, so put a reverse proxy in front and set `PUBLIC_URL` to the public `https://` address. Bind the published port to the host only when the proxy runs on the same host, by changing `ports` to `"127.0.0.1:8080:8080"`.