feat: PostgreSQL driver support
This commit is contained in:
fsylvain committed
1 parent
eb9bfabed5
commit
dc5d12348c
24 files changed
+1585
-75
No files matched your search
@@ -82,7 +82,7 @@ Uploaded images are addressed by a stored name (`<uuid>.<ext>`). `CreateImageUse
|
||||
|
||||
SQL in repositories must run on SQLite, MySQL/MariaDB and PostgreSQL: substring searches use `LIKE ... ESCAPE '!'` through `containsPattern` instead of `instr`. `Settings.DataDir()` is where update state, staging and backups live; it is `DATA_DIR` when set and the directory of `DB_FILE` otherwise, so it does not depend on where the database is.
|
||||
|
||||
Two drivers are registered in `database.drivers`: `sqlite` (default) and `mysql` (MySQL 8 and MariaDB 11, through `gorm.io/driver/mysql`, pure Go so `CGO_ENABLED=0` and the six release targets still build). `DB_DRIVER` picks one in `databaseConfig` (composition root); an unknown name fails in `database.Connect` with `unsupported database driver "<name>"`. `database.MySQLDSN` builds the DSN from separate variables, and the driver redacts the password from every error and names only host and port. `Connect` applies the optional `PoolSettings` to any driver. What keeps the repositories identical on both engines, all without touching the domain, use cases or handlers:
|
||||
Three drivers are registered in `database.drivers`: `sqlite` (default), `mysql` (MySQL 8 and MariaDB 11, through `gorm.io/driver/mysql`) and `postgres` (PostgreSQL 16 and 17, alias `postgresql`, through `gorm.io/driver/postgres` on pgx v5); all are pure Go so `CGO_ENABLED=0` and the six release targets still build. `DB_DRIVER` picks one in `databaseConfig` (composition root); an unknown name fails in `database.Connect` with `unsupported database driver "<name>"`. `database.MySQLDSN` and `database.PostgresDSN` build the DSN from separate variables (the PostgreSQL one as an escaped URL with `sslmode`, `timezone=UTC` and a connect timeout), and the driver redacts the password from every error and names only host and port. `Connect` applies the optional `PoolSettings` to any driver. What keeps the repositories identical on both engines, all without touching the domain, use cases or handlers:
|
||||
|
||||
- Indexed and unique strings carry `size:` tags in the models (a MySQL index needs a bounded `varchar`). SQLite maps them to the same `text` column, so an existing SQLite file migrates with no schema or row change (`test/sqlite_schema_unchanged_test.go` builds a file from the old model definitions and compares `sqlite_master` and every row).
|
||||
- `Migrate` creates MySQL tables with `utf8mb4_bin` (`gorm:table_options`), so `Bob` and `bob` are distinct like in SQLite. The DSN also sets `collation=utf8mb4_bin`.
|
||||
@@ -90,9 +90,11 @@ Two drivers are registered in `database.drivers`: `sqlite` (default) and `mysql`
|
||||
- The Go driver sends a zero `time.Time` as `0000-00-00`, which MySQL's default `NO_ZERO_DATE` rejects, so each session sets `sql_mode` to the server default minus `NO_ZERO_DATE`. Reading `0000-00-00` gives the zero time back.
|
||||
- `UserRepository.Create` returns the filled model instead of re-reading it, because MySQL has no `RETURNING`.
|
||||
|
||||
Backups are a `domain/update.DatabaseBackup` per driver. SQLite uses `VACUUM INTO`. MySQL uses `DumpBackup`, a pure Go logical dump (`SHOW CREATE TABLE` plus every row, strings escaped and binary values as hex, NULL and zero dates kept) written inside one repeatable-read transaction to `DATA_DIR/backups/renewcms-<time>.sql.partial` and renamed once complete. Any failure removes the partial file and returns an error, so the update download fails instead of claiming a backup. Both keep the newest 3 files (`pruneBackups`).
|
||||
On PostgreSQL the MySQL-only table options are not applied, pgx returns `timestamptz` values in the local zone so a query callback in the driver converts them to UTC, and only the `public` schema is used.
|
||||
|
||||
`task test` runs everything that needs no server. The repository contract tests (`test/repository_contract_test.go`) run on SQLite always, and on MySQL when `RENEWCMS_TEST_MYSQL_DSN` points at a server (admin DSN; each test creates and drops its own database). `task test:mysql` starts `mysql:8.4` and then `mariadb:11` in Docker and runs the `MySQL` tests against each.
|
||||
Backups are a `domain/update.DatabaseBackup` per driver. SQLite uses `VACUUM INTO`. MySQL uses `DumpBackup`, a pure Go logical dump (`SHOW CREATE TABLE` plus every row, strings escaped and binary values as hex, NULL and zero dates kept) written inside one repeatable-read transaction to `DATA_DIR/backups/renewcms-<time>.sql.partial` and renamed once complete. Any failure removes the partial file and returns an error, so the update download fails instead of claiming a backup. PostgreSQL reuses `DumpBackup` with its own `dumpDialect` (double-quoted identifiers, doubled single quotes, `\x` bytea literals): `CREATE TABLE` is rebuilt from `pg_catalog` (serial columns, defaults, primary key, unique and check constraints), followed by the indexes, the rows and a `setval` per `id` sequence, all inside one repeatable-read read-only transaction. All drivers keep the newest 3 files (`pruneBackups`).
|
||||
|
||||
`task test` runs everything that needs no server. The repository contract tests (`test/repository_contract_test.go`) run on SQLite always, and on MySQL when `RENEWCMS_TEST_MYSQL_DSN` points at a server (admin DSN; each test creates and drops its own database). `task test:mysql` starts `mysql:8.4` and then `mariadb:11` in Docker and runs the `MySQL` tests against each. The same contract tests run on PostgreSQL when `RENEWCMS_TEST_POSTGRES_DSN` points at a server (admin DSN, for example `postgres://postgres:pw@127.0.0.1:5432/postgres?sslmode=disable`), and `task test:postgres` starts `postgres:16` then `postgres:17` in Docker and runs the `Postgres` tests against each.
|
||||
|
||||
Two storage adapters implement `image.Storage`: `local` (default, files in `UPLOAD_DIR`) and `s3` (`storage.S3Storage`, `aws-sdk-go-v2` over plain HTTPS, pure Go so `CGO_ENABLED=0` still holds). `STORAGE_DRIVER` picks one in `imageStorageConfig` (composition root, from `config.StorageSettings`); `storage.Open` fails with `unsupported storage driver "<name>"` for anything else, and `openImageStorage` panics with the error. The S3 adapter works as follows, with no change to the domain, use cases, repositories or the `staticimages` handler:
|
||||
|
||||
|
||||
+12
-5
@@ -24,9 +24,9 @@ Copy [`.env.example`](../.env.example) to `.env` to get started. When a required
|
||||
|------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `ENVIRONMENT` | `production` | `production` or `development`. In `production` the auth cookie is `Secure`, so it is only sent over HTTPS. `development` turns on HTTP request logging. |
|
||||
| `PORT` | `8080` | Port the HTTP server listens on (all interfaces). |
|
||||
| `DB_DRIVER` | `sqlite` | `sqlite` or `mysql` (MySQL and MariaDB). Any other value stops the server with `unsupported database driver "<value>"`. See [Database](#database). |
|
||||
| `DB_FILE` | `./data/renewcms.db` | Path to the SQLite database file. Missing parent directories are created. The schema is migrated automatically at startup. Ignored when `DB_DRIVER=mysql`. |
|
||||
| `DATA_DIR` | directory of `DB_FILE` | Directory for update state, staged updates and database backups (`backups/`). Set it with `DB_DRIVER=mysql`, where there is no database file to sit beside (the default still follows `DB_FILE`). With SQLite, leave it unset to keep everything beside the database. |
|
||||
| `DB_DRIVER` | `sqlite` | `sqlite`, `mysql` (MySQL and MariaDB) or `postgres` (alias `postgresql`). Any other value stops the server with `unsupported database driver "<value>"`. See [Database](#database). |
|
||||
| `DB_FILE` | `./data/renewcms.db` | Path to the SQLite database file. Missing parent directories are created. The schema is migrated automatically at startup. Ignored when `DB_DRIVER` is `mysql` or `postgres`. |
|
||||
| `DATA_DIR` | directory of `DB_FILE` | Directory for update state, staged updates and database backups (`backups/`). Set it with `DB_DRIVER=mysql` or `postgres`, where there is no database file to sit beside (the default still follows `DB_FILE`). With SQLite, leave it unset to keep everything beside the database. |
|
||||
| `STORAGE_DRIVER` | `local` | `local` (files in `UPLOAD_DIR`) or `s3` (an S3-compatible bucket). Any other value stops the server with `unsupported storage driver "<value>"`. See [Image storage](#image-storage). |
|
||||
| `UPLOAD_DIR` | `./data/uploads` | Directory where uploaded images are stored. Ignored, without error, when `STORAGE_DRIVER=s3`. |
|
||||
| `CORS_ALLOWED_ORIGINS` | *(empty)* | Semicolon-separated list of origins allowed to call the API. When empty, no CORS headers are sent. That is the right setting whenever the SPA is served by RenewCMS itself (production) or through the Vite proxy (development). |
|
||||
@@ -72,22 +72,29 @@ The pre-update backup covers the database only (and `DATA_DIR` state). Neither t
|
||||
|
||||
## Database
|
||||
|
||||
SQLite is the default and needs nothing but `DB_FILE`. To run against MySQL 8 or MariaDB 11 instead, set `DB_DRIVER=mysql` and the connection variables below. They are only read (and the four required ones only checked) when `DB_DRIVER=mysql`. A required variable that is not set stops the server with `Environment variable DB_HOST is not set` (or `DB_USER`, `DB_PASSWORD`, `DB_NAME`).
|
||||
SQLite is the default and needs nothing but `DB_FILE`. To run against MySQL 8 or MariaDB 11 instead, set `DB_DRIVER=mysql` and the connection variables below. They are only read (and the four required ones only checked) when `DB_DRIVER` is `mysql` or `postgres`. A required variable that is not set stops the server with `Environment variable DB_HOST is not set` (or `DB_USER`, `DB_PASSWORD`, `DB_NAME`).
|
||||
|
||||
| Name | Default | Description |
|
||||
|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `DB_HOST` | required | Server host name or IP address. |
|
||||
| `DB_PORT` | `3306` | Server port. |
|
||||
| `DB_PORT` | `3306` (`5432` for PostgreSQL) | Server port. |
|
||||
| `DB_USER` | required | Database user. It needs full rights on `DB_NAME`, because the schema is created and migrated at startup. |
|
||||
| `DB_PASSWORD` | required | Database password. It is never logged and never appears in an error message. |
|
||||
| `DB_NAME` | required | Name of an existing database. The server does not create it. |
|
||||
| `DB_TLS` | `false` | `false`, `true`, `skip-verify` or `preferred`. Any other value stops the server with an error naming `DB_TLS` and these values. `true` verifies the server certificate; `skip-verify` encrypts without verifying it. |
|
||||
| `DB_SSLMODE` | `disable` | PostgreSQL only: `disable`, `require`, `verify-ca` or `verify-full`. Any other value stops the server with an error naming `DB_SSLMODE` and these values. Ignored for SQLite and MySQL (use `DB_TLS` there). |
|
||||
| `DB_MAX_OPEN_CONNS` | `10` | Maximum open connections in the pool. `0` means unlimited. |
|
||||
| `DB_MAX_IDLE_CONNS` | `5` | Maximum idle connections kept in the pool. |
|
||||
| `DB_CONN_MAX_LIFETIME_SECONDS` | `300` | Seconds after which a connection is closed and replaced. `0` keeps connections forever. Keep it below the server's `wait_timeout`. |
|
||||
|
||||
A non numeric (or negative) pool value stops the server with an error naming the variable. There is no DSN variable: the connection string is built from the variables above, with `parseTime=true`, `loc=UTC`, `charset=utf8mb4`, `collation=utf8mb4_bin` and `clientFoundRows=true`. Connecting uses a 5 second timeout, so a wrong host, port, password or database name stops startup quickly with `Unable to open the database`, naming the host and port but never the password.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
Set `DB_DRIVER=postgres` (or `postgresql`) and the same connection variables, with `DB_PORT` defaulting to `5432` and `DB_SSLMODE` instead of `DB_TLS`. PostgreSQL 16 and 17 are tested. The database must already exist and `DB_USER` needs full rights on the `public` schema, which is the only schema used. The connection string is built with escaping from the variables (a password may contain spaces, quotes, `@` or `/`), sets the session time zone to UTC and a 5 second connect timeout, and the pool variables above apply. A failure to connect stops startup with `Unable to open the database`, naming the host and port but never the password. Usernames and emails compare case-sensitively, as on SQLite. Before an update download RenewCMS writes a logical dump to `DATA_DIR/backups/renewcms-<time>.sql` (the newest 3 are kept); replay it into an empty database with `psql -f`. For routine backups prefer `pg_dump`.
|
||||
|
||||
### MySQL settings
|
||||
|
||||
Why those settings: tables are created with the `utf8mb4_bin` collation so that usernames and emails compare case-sensitively exactly like in SQLite (`Bob` and `bob` are different users). The collation of `DB_NAME` itself does not matter, but tables that already exist keep the collation they have. `clientFoundRows` makes an update that does not change a row still count as matched, which the invite and purge-settings code relies on. Each session also drops the server's `NO_ZERO_DATE` SQL mode, because the Go driver sends a zero time as `0000-00-00`. The other strict modes stay on.
|
||||
|
||||
With MySQL the data directory (`DATA_DIR`) holds update state and backups only. The pre-update backup is a `renewcms-<time>.sql` logical dump (schema and rows, written by RenewCMS itself, no `mysqldump` needed) that keeps the newest 3. If the dump fails, the update download fails with it. The dump holds your data, such as password hashes, but not the connection password.
|
||||
|
||||
+18
-2
@@ -8,7 +8,7 @@ RenewCMS ships as **one self-contained executable**. It serves:
|
||||
| `/healthz` | Liveness check, returns `200 ok` |
|
||||
| `/*` | The Vue SPA (embedded), with fallback to `index.html` for client-side routes |
|
||||
|
||||
Runtime state includes the SQLite database (`DB_FILE`, or a MySQL/MariaDB server, see [Using MySQL or MariaDB](#using-mysql-or-mariadb)), uploads (`UPLOAD_DIR`, or an S3-compatible bucket with `STORAGE_DRIVER=s3`, see [configuration.md](configuration.md#image-storage)), and update records, staged binaries and backups beside the database. Running a release binary needs no Go, Bun or Node installation. Browser updates additionally require Linux with systemd and the helper setup below.
|
||||
Runtime state includes the SQLite database (`DB_FILE`, or a MySQL/MariaDB or PostgreSQL server, see [Using MySQL or MariaDB](#using-mysql-or-mariadb) and [Using PostgreSQL](#using-postgresql)), uploads (`UPLOAD_DIR`, or an S3-compatible bucket with `STORAGE_DRIVER=s3`, see [configuration.md](configuration.md#image-storage)), and update records, staged binaries and backups beside the database. Running a release binary needs no Go, Bun or Node installation. Browser updates additionally require Linux with systemd and the helper setup below.
|
||||
|
||||
To run it in a container instead, see [docker.md](docker.md). The published image `git.floriansylvain.fr/fsylvain/renewcms` is upgraded with `docker compose pull && docker compose up -d`. Tags are `vX.Y.Z`, `X.Y.Z`, `X.Y` and `latest` for a stable release, and `vX.Y.Z-<pre>`, `X.Y.Z-<pre>` and the moving `<pre-name>` (`beta`, `rc`) for a pre-release, which never moves `latest`. Maintainers publish with `task docker:publish` after `docker login git.floriansylvain.fr`; there is no CI. See [Maintainer: publishing](docker.md#maintainer-publishing).
|
||||
|
||||
@@ -292,6 +292,22 @@ DATA_DIR=/var/lib/renewcms
|
||||
|
||||
`DB_FILE` is then ignored. `DATA_DIR` is where update state, staged updates and backups live, so give it the same value as the helper's `--data-dir` (see [browser updates](#browser-updates)). Use `DB_TLS=true` (or `preferred`) whenever the server is not on the same host. Every variable is described in [configuration.md](configuration.md#database). The schema is created and migrated at startup, so the first start on an empty database needs nothing else.
|
||||
|
||||
## Using PostgreSQL
|
||||
|
||||
To use an existing PostgreSQL 16 or 17 server, create an empty database and a user with full rights on its `public` schema, then set in `/etc/renewcms/renewcms.env`:
|
||||
|
||||
```sh
|
||||
DB_DRIVER=postgres
|
||||
DB_HOST=db.example.com
|
||||
DB_USER=renewcms
|
||||
DB_PASSWORD=<generated password>
|
||||
DB_NAME=renewcms
|
||||
DB_SSLMODE=require
|
||||
DATA_DIR=/var/lib/renewcms
|
||||
```
|
||||
|
||||
`DB_FILE` is then ignored, and `DB_PORT` defaults to `5432`. `DATA_DIR` is where update state, staged updates and backups live, so give it the same value as the helper's `--data-dir`. Use `DB_SSLMODE=require` (or `verify-full`) whenever the server is not on the same host. Every variable is described in [configuration.md](configuration.md#database). The schema is created and migrated at startup. Moving an existing SQLite site is a manual procedure like the MySQL one below, and RenewCMS writes a logical `.sql` dump to `DATA_DIR/backups` before each update download.
|
||||
|
||||
### Moving an existing SQLite site to MySQL
|
||||
|
||||
There is no migration tool. The manual procedure, with the site stopped:
|
||||
@@ -305,7 +321,7 @@ Keep the SQLite file until you have checked the site.
|
||||
|
||||
## Backups
|
||||
|
||||
With MySQL or MariaDB, back the database up with your server's own tools (for example `mysqldump --single-transaction`), plus the uploads directory. RenewCMS also writes a logical dump to `DATA_DIR/backups` before each update download. The rest of this section is about SQLite.
|
||||
With MySQL, MariaDB or PostgreSQL, back the database up with your server's own tools (for example `mysqldump --single-transaction` or `pg_dump`), plus the uploads directory. RenewCMS also writes a logical dump to `DATA_DIR/backups` before each update download. The rest of this section is about SQLite.
|
||||
|
||||
Everything worth backing up is under the data directory. Take a consistent SQLite copy while the service is running:
|
||||
|
||||
|
||||
@@ -56,6 +56,7 @@ If your editor reports `pattern all:dist: no matching files found`, either run `
|
||||
task --list # everything available
|
||||
task test # Go tests; extra args pass through: task test -- -run TestCreateArticle -v
|
||||
task test:mysql # MySQL tests against real servers in Docker (mysql:8.4, then mariadb:11)
|
||||
task test:postgres # PostgreSQL tests against real servers in Docker (postgres:16, then postgres:17)
|
||||
task cover # coverage of the use cases → tmp/coverage.out
|
||||
task mutate # gremlins mutation testing → tmp/gremlins.json
|
||||
task test:web # SPA component tests (Vitest)
|
||||
|
||||
@@ -102,6 +102,17 @@ To use MariaDB, change the `mysql` service image to `mariadb:11`, its `MYSQL_*`
|
||||
|
||||
Back the database up with `docker compose -f docker-compose.mysql.yml exec mysql mysqldump ...` (a `tar` copy of the volume is not consistent while it runs), and the `renewcms-data` volume as described above. Moving an existing SQLite volume is a manual procedure, see [deployment.md](deployment.md#moving-an-existing-sqlite-site-to-mysql).
|
||||
|
||||
## PostgreSQL
|
||||
|
||||
[`docker-compose.postgres.yml`](../docker-compose.postgres.yml) runs RenewCMS next to a PostgreSQL 17 container:
|
||||
|
||||
```sh
|
||||
export DB_PASSWORD=<generated>
|
||||
docker compose -f docker-compose.postgres.yml up -d
|
||||
```
|
||||
|
||||
It starts the `postgres` service with a named volume `postgres-data` and a `pg_isready` healthcheck, and starts `renewcms` only once the database is healthy (`depends_on` with `service_healthy`). RenewCMS gets `DB_DRIVER=postgres`, `DB_HOST=postgres`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD` and `DB_SSLMODE=disable` (the traffic stays on the Compose network) from the compose file and the rest from `docker.env`. `DATA_DIR=/data` keeps update state and the `.sql` pre-update backups on the `renewcms-data` volume. For a server on another machine drop the `postgres` service, point `DB_HOST` at it and set `DB_SSLMODE=require`. Back the database up with `docker compose -f docker-compose.postgres.yml exec postgres pg_dump -U renewcms renewcms`. Replace the placeholders; never commit real passwords.
|
||||
|
||||
## S3-compatible storage (SeaweedFS)
|
||||
|
||||
[`docker-compose.s3.yml`](../docker-compose.s3.yml) runs RenewCMS with `STORAGE_DRIVER=s3` next to a [SeaweedFS](https://github.com/seaweedfs/seaweedfs) server (Apache 2.0, image `chrislusf/seaweedfs`, override with `SEAWEEDFS_IMAGE`) with its S3 gateway enabled, and a one-shot `seaweedfs-init` step that creates the bucket with `weed shell`:
|
||||
|
||||
Reference in new issue
Block a user