feat: optional S3-compatible storage for uploaded images, instead of the local upload directory
This commit is contained in:
fsylvain committed
1 parent
d752a61e52
commit
eb9bfabed5
23 files changed
+1962
-7
No files matched your search
@@ -0,0 +1,143 @@
|
||||
Feature: Uploaded images can be stored in an S3-compatible bucket
|
||||
|
||||
As an operator whose host disk is ephemeral or who runs several hosts
|
||||
I want to set STORAGE_DRIVER=s3 and bucket settings
|
||||
So that uploads live in the bucket while images are still served by the app
|
||||
at /static/uploadedImages/<name>, with unchanged URLs
|
||||
|
||||
Background:
|
||||
Given STORAGE_DRIVER defaults to "local" and local disk behaves exactly as today
|
||||
And an S3 storage implements image.Storage (Put, Open, Delete) in internal/infrastructure/storage
|
||||
And storage.Open has a case for the driver "s3" and openImageStorage selects it from the settings
|
||||
And no domain, use case, repository or staticimages handler file changes
|
||||
And the bucket is faked by an in-process httptest S3 server listening on 127.0.0.1
|
||||
And Open returns a Blob whose Content is an io.ReadSeeker, so http.ServeContent handles Range and conditional requests
|
||||
And the Blob ModTime and Size come from the S3 object metadata
|
||||
And no ETag is emitted and conditional requests are validated by Last-Modified only
|
||||
|
||||
Scenario Outline: Startup configuration validation
|
||||
Given STORAGE_DRIVER is <driver> and the S3 variables are <vars>
|
||||
When the storage settings are validated
|
||||
Then the result is <result>
|
||||
|
||||
Examples:
|
||||
| driver | vars | result |
|
||||
| (unset) | none | driver local, no S3 variable required |
|
||||
| local | S3_BUCKET set | driver local, S3 variables ignored |
|
||||
| s3 | S3_BUCKET and S3_REGION set | valid, default AWS credential chain |
|
||||
| s3 | S3_BUCKET set, S3_REGION unset | region defaults to us-east-1 |
|
||||
| s3 | S3_REGION set, S3_BUCKET unset | error "Environment variable S3_BUCKET is not set" |
|
||||
| s3 | key id set, secret unset | error naming S3_SECRET_ACCESS_KEY |
|
||||
| s3 | secret set, key id unset | error naming S3_ACCESS_KEY_ID |
|
||||
| s3 | UPLOAD_DIR set | valid, UPLOAD_DIR ignored without error |
|
||||
| gcs | none | error unsupported storage driver "gcs" |
|
||||
|
||||
Scenario: The registry opens the S3 storage after a bucket check
|
||||
Given the fake S3 answers HEAD bucket with 200
|
||||
When storage.Open is called with driver "s3"
|
||||
Then it returns a storage and no error
|
||||
|
||||
Scenario Outline: Startup fails fast with a clear secret-free error
|
||||
Given the fake S3 answers HEAD bucket with <status>
|
||||
When storage.Open is called with driver "s3" and secret access key "TOPSECRET"
|
||||
Then it returns an error mentioning the bucket name
|
||||
And the error text does not contain "TOPSECRET"
|
||||
And openImageStorage panics with that same text
|
||||
|
||||
Examples:
|
||||
| status |
|
||||
| 404 |
|
||||
| 403 |
|
||||
| 500 |
|
||||
|
||||
Scenario: Put stores the object under the prefix with its content type
|
||||
Given S3_KEY_PREFIX is "uploads/"
|
||||
When Put is called with stored name "a1b2.png", content type "image/png" and 3 bytes
|
||||
Then the fake S3 holds key "uploads/a1b2.png" with those bytes and Content-Type "image/png"
|
||||
|
||||
Scenario: Put failure is returned as an error
|
||||
Given the fake S3 answers PUT with 500
|
||||
When Put is called
|
||||
Then Put returns a non-nil error that is not image.ErrBlobNotFound
|
||||
|
||||
Scenario: Open returns content, size and modification time
|
||||
Given the fake S3 holds "uploads/a1b2.png" with 10 bytes and a known Last-Modified
|
||||
When Open("a1b2.png") is called
|
||||
Then the Blob has Size 10 and that ModTime
|
||||
And its Content implements io.ReadSeeker and reads the 10 bytes
|
||||
|
||||
Scenario: Open of a missing key is not found
|
||||
Given the fake S3 answers GET with 404 NoSuchKey
|
||||
When Open("missing.png") is called
|
||||
Then the result is image.ErrBlobNotFound, asserted with errors.Is
|
||||
|
||||
Scenario: Open with a backend failure is not a not-found
|
||||
Given the fake S3 answers GET with 500
|
||||
When Open is called
|
||||
Then the error is non-nil and is not image.ErrBlobNotFound
|
||||
|
||||
Scenario Outline: Delete is idempotent
|
||||
Given the fake S3 <state>
|
||||
When Delete("a1b2.png") is called
|
||||
Then it returns no error
|
||||
|
||||
Examples:
|
||||
| state |
|
||||
| holds the key |
|
||||
| does not hold the key (404) |
|
||||
|
||||
Scenario: A backend failure on Delete is returned
|
||||
Given the fake S3 answers DELETE with 500
|
||||
When Delete is called
|
||||
Then it returns a non-nil error
|
||||
|
||||
Scenario Outline: Serving /static/uploadedImages/<name> from the bucket
|
||||
Given the bucket holds "a1b2.png" with 10 bytes and a known Last-Modified
|
||||
When <method> /static/uploadedImages/a1b2.png is requested <with>
|
||||
Then the status is <status>
|
||||
And the response has <result>
|
||||
|
||||
Examples:
|
||||
| method | with | status | result |
|
||||
| GET | no extra header | 200 | Content-Type image/png, Content-Length 10, Last-Modified, Accept-Ranges bytes, Cache-Control "public, no-cache", X-Content-Type-Options nosniff |
|
||||
| HEAD | no extra header | 200 | the same headers and an empty body |
|
||||
| GET | If-Modified-Since = Last-Modified | 304 | an empty body |
|
||||
| GET | Range bytes=0-3 | 206 | Content-Range "bytes 0-3/10" and Content-Length 4 |
|
||||
| GET | Range bytes=50-60 | 416 | Content-Range "bytes */10" |
|
||||
|
||||
Scenario: SVG from the bucket keeps its security headers
|
||||
Given the bucket holds "logo.svg"
|
||||
When GET /static/uploadedImages/logo.svg is requested
|
||||
Then Content-Type is image/svg+xml
|
||||
And Content-Security-Policy is "default-src 'none'; style-src 'unsafe-inline'; sandbox"
|
||||
|
||||
Scenario: Missing object is 404 and a backend failure is 500
|
||||
Given the bucket does not hold "missing.png"
|
||||
When GET /static/uploadedImages/missing.png is requested
|
||||
Then the status is 404
|
||||
Given the fake S3 answers GET with 500
|
||||
When GET /static/uploadedImages/a1b2.png is requested
|
||||
Then the status is 500 and the plain text body is "Unable to read the file"
|
||||
|
||||
Scenario: Path traversal never reaches another key
|
||||
When GET /static/uploadedImages/../secret.png is requested
|
||||
Then the key requested from the fake S3 is the prefix plus "secret.png"
|
||||
|
||||
Scenario: Handler as-is for a non-seekable blob
|
||||
Given a storage fake returning a Blob whose Content is not an io.ReadSeeker, with Size 10 and a ModTime
|
||||
When GET and HEAD /static/uploadedImages/a1b2.png are requested
|
||||
Then GET returns 200 with Content-Length 10, Last-Modified and the body
|
||||
And HEAD returns 200 with the same headers and an empty body
|
||||
And no Range or ETag behaviour is specified for this path
|
||||
|
||||
Scenario: Boot backfill logs how many images it read from the bucket
|
||||
Given the driver is s3 and 3 stored images lack metadata
|
||||
When InitContainer runs the boot backfill
|
||||
Then a log line reports 3 images read from storage and contains no credential
|
||||
And backfill behaviour and errors are unchanged
|
||||
And no cap is applied and BackfillImageMetadataUseCase is not modified
|
||||
|
||||
Scenario: The local driver is unchanged
|
||||
Given STORAGE_DRIVER is unset and UPLOAD_DIR is a temp directory
|
||||
When an image is uploaded and requested
|
||||
Then it is stored in UPLOAD_DIR and served exactly as before
|
||||
@@ -0,0 +1,26 @@
|
||||
# QA: S3 image storage
|
||||
|
||||
All tests live in test/ (package test). No cloud account, no real SMTP, no real SQLite file. Test names carry "S3Storage".
|
||||
|
||||
Settled decisions: boot backfill gets a log line only, wired in the composition root, with no cap and no use case change. No ETag is emitted; the staticimages handler is unchanged and conditional requests are validated by Last-Modified only.
|
||||
|
||||
## Automated
|
||||
1. Config validation: table test over STORAGE_DRIVER and S3_* lookup functions (the MissingEnvVarError pattern, a `StorageRequiredEnvVars(driver)` helper like `DatabaseRequiredEnvVars`). Covers the validation table, UPLOAD_DIR ignored, unknown driver error.
|
||||
2. Registry: storage.Open with "local" returns local storage, "s3" returns the S3 storage after HEAD bucket, anything else returns the "unsupported storage driver" error.
|
||||
3. Fake S3: an httptest.Server on 127.0.0.1 only (never bind :port) implementing HEAD bucket, PUT, GET with Range, HEAD and DELETE object, with injectable statuses and recorded keys, headers and bodies.
|
||||
4. Adapter tests against the fake: the Put, Open (found, 404, 500) and Delete scenarios, HEAD bucket 404/403/500, prefix handling, ModTime and Size from metadata, Content as io.ReadSeeker (seek then read issues a ranged GET), secret absent from error text.
|
||||
5. Handler tests: staticimages.NewHandler over OpenMediaUseCase over the S3 storage on the fake server, mounted on a fresh chi.NewRouter() at /static/uploadedImages/*. Cover GET, HEAD, 304 via If-Modified-Since, 206, 416, SVG CSP, 404, 500, traversal.
|
||||
6. Handler regression: a storage fake returning a non-seekable Blob. GET and HEAD behave exactly as today.
|
||||
7. Backfill log: the boot log line for the number of images read, captured with a log writer, containing no credential.
|
||||
8. Local regression: all existing image tests pass unchanged.
|
||||
9. Gates: `task test` and `task lint:go` pass. `task mutate` is not part of this slice, but the new storage package becomes a target if the mutation scope is widened. The 404 to ErrBlobNotFound and idempotent-Delete branches need killing tests.
|
||||
|
||||
## Optional integration (task test:s3)
|
||||
Start a SeaweedFS container with Docker, create a bucket, set RENEWCMS_TEST_S3_ENDPOINT and run the adapter tests against it. Skip with a message when Docker is unavailable. Not part of `task test`.
|
||||
|
||||
## Manual
|
||||
- Run docker-compose.s3.yml, upload an image in the SPA, confirm the object is in the bucket and /static/uploadedImages/<name> serves it, including after the container is recreated.
|
||||
- Real AWS S3, R2, B2 or Scaleway smoke test, and an instance-role credential check.
|
||||
- Start with a wrong bucket and confirm a clear, secret-free startup error.
|
||||
- Sync an old UPLOAD_DIR with `aws s3 sync` or `mc mirror` and confirm old images load.
|
||||
- Confirm docs describe: variables (configuration.md, .env.example, docker.env.example, Docker docs), UPLOAD_DIR ignored with s3, manual migration, update backups covering the DB only and not the bucket, and the updated architecture.md "Storage and database seams" section.
|
||||
Reference in new issue
Block a user