fsylvain/RenewCMS
1
1 Fork 0
Code Issues Pull Requests Packages Projects Releases 20 Wiki Activity

feat: media duplicate purge, part 2 of 3: the automatic purge scheduler (Go only)

Browse Source
This commit is contained in:
fsylvain committed 2026-10-05 19:44:15 +02:00
1 parent e82d0ce254
commit 1a55a39425
18 files changed
+1389 -39
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

features/media-purge.feature → features/media_purge.feature
+2 -2
View File
@@ -8,7 +8,7 @@ Feature: Owners and admins can merge duplicate images in use, with persisted pur
Given internal/domain/purge defines the entity Settings with AutoPurgeEnabled (default false),
AutoPurgeIntervalDays (default 30, valid 1 to 365), LastPurgeAt, LastPurgeMerged,
LastPurgeArticlesUpdated and EnabledAt (unset or 0 by default), plus a Repository
interface with Get and Save, and the sentinels ErrInvalidPurgeInterval, ErrPurgeConflict
interface with Get, Save and SaveLastRun, and the sentinels ErrInvalidPurgeInterval, ErrPurgeConflict
(the library changed since the fingerprint) and ErrPurgeBusy (a run is already in progress),
all asserted with errors.Is
And Repository.Get returns the defaults when nothing was ever saved (a single-row table)
@@ -389,7 +389,7 @@ Feature: Owners and admins can merge duplicate images in use, with persisted pur
And no last_purge_* values are saved
Scenario: A failing save of the run result is a logged 500 and the merges stay
Given the purge settings repository's Save fails after the merges succeeded
Given the purge settings repository's SaveLastRun fails after the merges succeeded
When the client sends POST /v1/media/purge with {}
Then the response status is 500 and the body is "Internal server error"
And the merges that already happened stay done
features/media-purge.qa.md → features/media_purge.qa.md
+1 -1
View File
@@ -1,4 +1,4 @@
# QA procedure: media-purge
# QA procedure: media_purge
Package: `test` (black-box). New files `test/media_purge_http_test.go` and `test/media_purge_test.go` (use case level). Fake type names must be unique in the package. Go only: no SPA, no scheduler, no env var.
features/media_purge_scheduler.feature
+286
View File
@@ -0,0 +1,286 @@
Feature: Duplicate images are purged automatically on the configured schedule
As a RenewCMS owner or admin
I want the duplicate purge to run by itself every few days once I turn it on
So that I do not have to remember to purge, and the Settings "Last purge" line stays current
Background:
Given part 1 provides purge.Settings, purge.Repository (Get, Save and SaveLastRun) and RunPurgeUseCase.Run(fingerprint *string)
And a new use case PurgeSchedulerUseCase in internal/infrastructure/useCases depends only on purge.Repository
and a RunPurge port, takes no HTTP types and imports no persistence, mailer, models, gorm, net/http, chi or api
And PurgeSchedulerUseCase.Tick(now time.Time) (result purge.RunResult, ran bool, err error) reads the settings on every call,
where result carries groups purged, copies merged and articles updated and is empty when ran is false
And a run is due when auto_purge_enabled is true, enabled_at is set and now is not before
Settings.NextPurgeAt(), that is max(last_purge_at, enabled_at) + auto_purge_interval_days days
And a scheduled run is RunPurge.Run(nil): no fingerprint, no HTTP, no user-visible message
And a run that finds nothing to purge still records last_purge_at with merged 0 and articles_updated 0,
so the anchor moves forward (this is RunPurge's existing behaviour)
And the scheduler never writes settings itself: only RunPurge records last_purge_*,
through Repository.SaveLastRun, which writes only the last_purge_* fields and never the whole settings row
And there is no env var: the schedule (auto_purge_enabled, auto_purge_interval_days, enabled_at,
last_purge_at) lives in the purge settings row, and is read again on every tick
And a runner in the new package internal/infrastructure/scheduler calls Tick on a ticker every 1 hour,
runs one Tick after a startup delay of 1 minute, and stops when its context is cancelled
And the runner takes its clock, its tick source and its startup delay by injection, so tests use a fake clock
and hand-fed channels, with no real sleeping and no real time.Ticker
And api/dependecyInjection.go builds the runner and cmd/server/main.go only starts it
And the runner uses the *log.Logger already used by RunPurge (log.Default() in the container)
Rule: When a run is due
Background:
Given the fake clock reads 2026-10-05T09:00:00Z
Scenario: Automatic purge off never runs, whatever the dates are
Given auto_purge_enabled is false, interval 1, enabled_at 2020-01-01T00:00:00Z and last_purge_at 2020-01-02T00:00:00Z
When Tick is called with the fake clock's time
Then ran is false and err is nil
And RunPurge was not called and Save was not called
Scenario: The defaults never run
Given nothing was ever saved, so the settings are the defaults
When Tick is called
Then ran is false and RunPurge was not called
Scenario: On, never ran, not due yet
Given auto_purge_enabled is true, interval 30, enabled_at 2026-09-20T09:00:00Z and no last_purge_at
When Tick is called
Then ran is false and RunPurge was not called
Scenario: On, never ran, due since enabled_at plus the interval
Given auto_purge_enabled is true, interval 10, enabled_at 2026-09-25T09:00:00Z and no last_purge_at
When Tick is called
Then RunPurge is called once with no fingerprint and ran is true
Scenario Outline: The anchor is the later of last_purge_at and enabled_at
Given auto_purge_enabled is true and interval 10
And enabled_at is <enabled_at> and last_purge_at is <last_purge_at>
When Tick is called at 2026-10-05T09:00:00Z
Then RunPurge is called <calls> time(s)
Examples:
| enabled_at | last_purge_at | calls | why |
| 2026-09-01T00:00:00Z | 2026-09-20T00:00:00Z | 1 | last purge + 10 days = 2026-09-30, due |
| 2026-09-01T00:00:00Z | 2026-09-28T00:00:00Z | 0 | last purge + 10 days = 2026-10-08, not due |
| 2026-09-28T00:00:00Z | 2026-09-01T00:00:00Z | 0 | enabled_at is later, 2026-10-08, not due |
| 2026-09-20T00:00:00Z | 2026-09-01T00:00:00Z | 1 | enabled_at is later, 2026-09-30, due |
| 2026-09-25T09:00:00Z | none | 1 | exactly now is due (2026-10-05T09:00:00Z) |
| 2026-09-25T09:00:01Z | none | 0 | one second before due |
Scenario: Due exactly at the boundary, not before
Given the next purge is 2026-10-05T09:00:00Z
When Tick is called at 2026-10-05T08:59:59Z
Then RunPurge is not called
When Tick is called at 2026-10-05T09:00:00Z
Then RunPurge is called once
Scenario: A run advances the anchor, so the next tick is not due
Given a due run with RunPurge backed by the fake settings repository
When Tick is called twice with the same time
Then RunPurge is called once
And last_purge_at equals the fake clock's time
Scenario: A due run that finds nothing to purge still moves the clock forward
Given a due schedule and a library with no planned group
When Tick is called
Then the stored last_purge_at is the clock's time, last_purge_merged is 0 and last_purge_articles_updated is 0
And a second Tick at the same time does not run
And a Tick after the interval has elapsed runs again
Scenario: A run keeps auto_purge_enabled, the interval and enabled_at
Given a due schedule
When Tick runs
Then auto_purge_enabled, auto_purge_interval_days and enabled_at are unchanged in the stored settings
Scenario: Enabled with no enabled_at is not due
Given auto_purge_enabled is true and enabled_at is unset
When Tick is called
Then RunPurge is not called and the scheduler does not stamp enabled_at itself
Rule: Settings changes take effect on the next tick, with no restart
Background:
Given the fake clock reads 2026-10-05T09:00:00Z
Scenario: Shortening the interval makes a run due on the next tick
Given auto_purge_enabled is true since 2026-09-25T09:00:00Z with interval 30 and no purge yet
And Tick did not run
When UpdatePurgeSettings sets auto_purge_interval_days to 7
And Tick is called again
Then RunPurge is called once
Scenario: Lengthening the interval postpones the run
Given auto_purge_enabled is true since 2026-09-20T09:00:00Z with interval 10 and no purge yet, so a run is due
When UpdatePurgeSettings sets auto_purge_interval_days to 30
And Tick is called
Then RunPurge is not called
Scenario: Turning it off stops runs
Given a due schedule
When UpdatePurgeSettings sets auto_purge_enabled to false
And Tick is called
Then RunPurge is not called
Scenario: Turning it on resets the anchor to the moment of the switch
Given auto_purge_enabled was turned off, last_purge_at is 2026-09-01T00:00:00Z, the interval is 30 and the clock reads 2026-10-05T09:00:00Z
When UpdatePurgeSettings sets auto_purge_enabled to true
And Tick is called at once
Then RunPurge is not called, because enabled_at is now
When Tick is called at 2026-11-04T09:00:00Z
Then RunPurge is called once
Scenario: The scheduler reads the settings at every tick
When Tick is called three times with different stored settings
Then Repository.Get is called three times and no value is cached between ticks
Rule: Failures never crash the loop and are retried at the next tick
Background:
Given a due schedule and the fake clock reads 2026-10-05T09:00:00Z
Scenario: Reading the settings fails
Given the purge settings repository's next Get fails
When Tick is called
Then ran is false and err is the repository error
And RunPurge was not called
And the next Tick, with Get working again, runs the purge
Scenario: RunPurge returns an error
Given RunPurge fails, for example ListDuplicates fails
When Tick is called
Then err is that error and last_purge_at is unchanged
And RunPurge was called once, not retried inside the same Tick
And the next Tick, with the failure cleared, runs the purge
Scenario: Saving the run result fails after the merges succeeded
Given the purge settings repository's SaveLastRun fails
When Tick is called
Then err is returned and the merges stay done
And the next Tick plans again from current data and converges
Scenario: A run that partially fails still records what it merged
Given group A merges and group B's UpdateBody fails
When Tick is called
Then ran is true and err is nil
And last_purge_at is set and last_purge_merged and last_purge_articles_updated count only group A
And group B is planned again by the next run
Scenario: A settings change made during a run survives
Given a run is in progress and about to record its result
When UpdatePurgeSettings turns auto_purge_enabled off and sets the interval to 5 meanwhile
And the run completes
Then the stored auto_purge_enabled is false and the interval is 5
And last_purge_at, last_purge_merged and last_purge_articles_updated hold the run's values
Scenario: A manual run is in progress
Given a manual POST /v1/media/purge holds the RunPurge guard
When Tick is called with a due schedule
Then the scheduler does not queue or wait: it returns at once with errors.Is(err, purge.ErrPurgeBusy)
And nothing is saved by the scheduler
And the runner treats it as a skip, not a failure, and the next tick re-evaluates
When the manual run finishes and records last_purge_at
Then the next Tick finds the schedule not due and does not run
Scenario: A failing tick does not stop the runner
Given the runner is running with a hand-fed tick channel
And the first Tick returns an error and the second returns ran true
When two ticks are fed
Then the error is written to the injected logger
And the runner is still running and the second Tick was executed
And no Tick is called between the two fed ticks (no tight retry loop)
Scenario: A completed run logs one line with the counts
Given Tick returns ran true with groups purged 3, merged 5, articles updated 2 and failed groups 1
When the runner receives the tick
Then exactly one line is logged: "scheduler: automatic purge completed: groups_purged=3 merged=5 articles_updated=2 failed_groups=1"
And a run that found nothing to purge logs the same line with every count at 0
Scenario: A due Tick returns what the run did
Given a due schedule and one planned group
When Tick is called
Then ran is true and the result holds the run's groups purged, merged and articles updated
And a Tick that is not due returns an empty result
Scenario: A busy skip is not logged as an error
Given Tick returns ErrPurgeBusy
When the runner receives the tick
Then it logs no error line and keeps waiting for the next tick
Rule: The runner starts, ticks and stops with the server
Scenario: One Tick shortly after startup
Given a runner with a hand-fed startup signal standing in for the 1 minute delay
When the startup signal fires
Then Tick is called once with the fake clock's time
And no Tick is called before the signal
Scenario: A due run is caught up at startup after downtime
Given the server was down for 40 days, the interval is 30 and last_purge_at is 40 days old
When the runner's startup Tick fires
Then RunPurge is called once, and only once however many ticks were missed
Scenario: A Tick on each hourly tick
Given the runner is running
When three hourly ticks are fed
Then Tick is called three times, each with the fake clock's time at that tick
Scenario: Cancelling the context stops the runner
Given the runner is idle
When its context is cancelled
Then Run returns promptly
And no further Tick is called even if a tick is fed after cancel
Scenario: Shutdown waits for a run in progress without a time limit
Given Tick is blocked inside RunPurge (fake image repository blocked on a channel)
When the context is cancelled
Then Run does not return yet
When the blocked run is released and completes
Then Run returns after the run has recorded last_purge_at
And the database is closed only after Run has returned
Scenario: Cancelled before the startup signal
Given the context is cancelled before the startup signal
Then Tick is never called and Run returns
Rule: The Settings page shows the result (HTTP observable, no SPA change)
Scenario: A scheduled run updates the Last purge values served by the settings endpoint
Given an owner, automatic purge on with interval 7, enabled_at 2026-09-20T09:00:00Z and a planned group
And the fake clock reads 2026-10-05T09:00:00Z
When Tick runs
And the client sends GET /v1/media/purge-settings with a valid cookie of the owner
Then "last_purge_at" is "2026-10-05T09:00:00Z"
And "last_purge_merged" and "last_purge_articles_updated" hold the run's counts
And "next_purge_at" is "2026-10-12T09:00:00Z"
And "auto_purge_enabled", "auto_purge_interval_days" and "enabled_at" are unchanged
Scenario: A scheduled run with nothing to purge shows zeros, not "Never run"
Given automatic purge on and due and no planned group
When Tick runs and the client sends GET /v1/media/purge-settings
Then "last_purge_at" is "2026-10-05T09:00:00Z", "last_purge_merged" is 0 and "last_purge_articles_updated" is 0
Scenario: A scheduled run answers no HTTP request and returns no message
When Tick runs
Then no HTTP handler is involved and the only observable results are the stored settings, the images, the article bodies and a log line
Rule: Layering and documentation
Scenario: The scheduler stays inside the domain boundary
Then PurgeSchedulerUseCase imports only the stdlib, internal/domain/* and package-local use case types
And internal/infrastructure/scheduler imports no persistence, mailer, models or gorm package
And only api/dependecyInjection.go constructs it and only cmd/server/main.go starts it
Scenario: The docs describe the scheduler and state that there is no env var
Then docs/architecture.md describes the scheduler, the hourly tick, the startup tick, the due rule, the overlap guard and the unbounded shutdown wait
And docs/configuration.md states that automatic purge has no env var and that its schedule lives in the database
And no new Go, TS, docs or config file contains a comment
@manual
Rule: Wiring verified by hand (see the qa.md)
Scenario: The runner starts with the real server and stops on SIGINT or SIGTERM
Given the server runs and automatic purge is on and due
When one minute elapses after startup
Then a purge ran and the Settings "Last purge" line shows the new date on the next visit
When SIGTERM is sent during a run
Then the process exits only after the run finished, and the SQLite file is not corrupted
features/media_purge_scheduler.qa.md
+56
View File
@@ -0,0 +1,56 @@
# QA procedure: media_purge_scheduler
Package: `test` (black-box). New file `test/media_purge_scheduler_test.go`. Go only: no SPA. No env var. Fake type names must be unique in the package. Reuse part 1's `mockPurgeSettingsRepo`, `mockImageRepo`, `mockArticleRepo`, `seedGroup` and `articleUsing` from test/media_purge_test.go and test/media_purge_http_test.go. No real sleeping, no `time.Ticker`, no `time.Sleep`, no live server, no real SQLite file.
## Setup shared by every case
1. `validator.InitValidator()` once (only needed by the HTTP cases).
2. Fakes:
- `fakeClock` (mutable, `Now()` and `Set(t)`), starting at 2026-10-05T09:00:00Z. Pass `clock.Now` as the `now func() time.Time` of RunPurge and UpdatePurgeSettings so last_purge_at and enabled_at come from it.
- `mockPurgeSettingsRepo` with `Get`, `Save` and `SaveLastRun` (writes only the last_purge_* fields), failure switches for each, a write counter shared by Save and SaveLastRun, and a hook called before SaveLastRun writes.
- A `fakePurgeRunPort` wrapping the real `RunPurgeUseCase`. It counts calls, records the fingerprint argument (must be nil), and can be armed to return an error. A second variant uses the real RunPurge directly for the integration cases.
- Hand-fed `tickCh` and `startCh` channels passed to the runner in `internal/infrastructure/scheduler` in place of time.NewTicker and the startup delay. Use a `bytes.Buffer` logger.
3. Wire the real `PurgeSchedulerUseCase` over the settings fake, the run port and the clock.
## Tick decisions (table-driven)
4. Off never runs: off with ancient dates; the defaults; assert ran false, err nil, zero RunPurge calls, zero Save calls.
5. Not due, due, boundary: enabled_at + interval one second after now gives no run; exactly now gives a run; assert the fingerprint argument is nil.
6. Anchor table: the six rows of the outline. Cross-check each row against `Settings.NextPurgeAt()` to prove the scheduler reuses it.
7. Enabled with unset enabled_at never runs and the scheduler never stamps it.
8. Repeat the same Tick twice with the real RunPurge: one run (anchor moved), and `last_purge_at` equals the clock. Advance the clock past the interval: a third Tick runs.
9. Empty library: due Tick records `last_purge_at` = clock, merged 0, articles_updated 0, and leaves auto_purge_enabled, interval and enabled_at untouched.
10. Settings changes: shorten the interval (via `UpdatePurgeSettings` on the same fake) and a run becomes due; lengthen it and a due run is postponed; toggle off and no run; toggle off then on gives enabled_at = clock, no run now, a run once the clock passes +30 days. Assert `Get` is called once per Tick.
## Failures
11. `failGet`: Tick returns the error, RunPurge count 0, the next Tick with the fake healed runs.
12. RunPurge error (`ListDuplicates` failure): error returned, `last_purge_at` unchanged, one RunPurge call per Tick, recovery on the next Tick.
13. `SaveLastRun` failure after merges: error returned, merged images still gone, next Tick converges.
14. Partial failure (UpdateBody failing for group B): ran true, err nil, `last_purge_at` set, counters count group A only.
15. Busy: block a manual `RunPurge.Run(nil)` inside the image fake (started channel). Call Tick with a due schedule and assert it returns promptly (use a timeout) with `errors.Is(err, purge.ErrPurgeBusy)`, with no `Save` from the scheduler. Release; the next Tick is not due because the manual run recorded `last_purge_at`. Run with `-race`.
## Runner
16. Feed a startup signal: exactly one Tick, with the clock's time; no Tick before the signal.
17. Feed three ticks: three Ticks. Feed none: no Tick. Assert no Tick between fed ticks.
18. Error then success from Tick: the error text is in the logger buffer, the loop continues, the second Tick executes, and `Run` has not returned.
19. Busy from Tick: no error line in the logger; the loop continues.
19b. Completed run: Tick returns ran true with a known `purge.RunResult`; the logger buffer holds exactly one line with groups_purged, merged, articles_updated and failed_groups. An empty result logs zeros.
19c. Tick result: a due Tick returns the run result; a Tick that is not due returns the empty result.
19d. Lost update: block the settings fake just before `SaveLastRun` writes, PATCH auto_purge_enabled off and the interval to 5 through `UpdatePurgeSettings`, release; the stored auto_purge_* survive and last_purge_* are the run's. The persistence repository test checks that `SaveLastRun` leaves the other columns untouched and creates the default row when none exists.
20. Cancel the context while idle: `Run` returns within a timeout; then feed a tick and assert no Tick happens. Cancel before the startup signal: no Tick at all.
21. Shutdown waits without a bound: block RunPurge in the image fake, cancel the context, assert `Run` has not returned (select with a short timeout), release, assert `Run` returns and `last_purge_at` was recorded. `-race`.
22. The runner and the use case spawn no goroutine that outlives `Run` (verify with a done channel closed on return).
## HTTP-observable check (existing settings endpoint)
23. Mount the part 1 media router (same setup as media_purge.qa.md) over the same fakes. Seed on, interval 7, enabled_at 2026-09-20T09:00:00Z, one planned group. Tick, then GET `/v1/media/purge-settings` with the owner cookie. Assert `last_purge_at` 2026-10-05T09:00:00Z, counts equal the run, `next_purge_at` 2026-10-12T09:00:00Z, and the other fields unchanged. Repeat with an empty library: zeros and a non-null `last_purge_at`.
## Layering guards
24. The scheduler use case imports no `net/http`, chi, gorm, `persistence`, `mailer` or `api/...`. Same grep-style test as media_purge.qa.md. `internal/infrastructure/scheduler` imports no `persistence`, `mailer`, `models` or `gorm`, and no `api/...`.
25. Grep the new and edited files (Go, docs) for comments: none; only tool directives.
## Docs
26. `docs/architecture.md` has a scheduler paragraph (hourly tick, startup tick, due rule from `NextPurgeAt`, overlap guard from RunPurge, unbounded shutdown wait, failures retried at the next tick). `docs/configuration.md` states that there is no env var and that the schedule lives in the database. Read both files and confirm.
## Manual (not automated)
27. Real boot with `task dev:api` or the built binary: turn automatic purge on, set interval 1 and backdate `enabled_at` in the SQLite file (or wait); about a minute after start, a purge runs, `GET /v1/media/purge-settings` shows the new `last_purge_at`, and Settings "Last purge" is updated on the next visit.
28. Send SIGINT or SIGTERM during a long run: the process exits only after the run ended, with no database error on close.
29. A scheduled run and a manual "Purge now" at the same moment: one wins, the other answers 409 busy (HTTP) or is skipped (scheduler).
30. Docker: the same behaviour in the container, with state on /data.