fsylvain/RenewCMS
1
1 Fork 0
Code Issues Pull Requests Packages Projects Releases 19 Wiki Activity

feat: setup logs retention by max file size

Browse Source
This commit is contained in:
fsylvain committed 2026-10-07 22:02:13 +02:00
1 parent 70c8492f4f
commit f914eae3c5
44 files changed
+2316 -161
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

features/activity-log-backend.feature
+9 -2
View File
@@ -30,7 +30,7 @@ Feature: The back-office keeps an append-only activity log that owners can read
| change.kind | text, body |
Scenario: The repository exposes no way to edit or delete one event
Then the audit Repository interface has exactly the methods Append, List, Summary and DeleteOlderThan
Then the audit Repository interface has exactly the methods Append, List, Summary, DeleteOlderThan, DeleteOldest and SizeBytes
And the persistence implementation exposes no other exported method than those and its constructor
And the HTTP surface has no POST, PUT, PATCH or DELETE under /v1/activity
@@ -464,7 +464,7 @@ Feature: The back-office keeps an append-only activity log that owners can read
| 2026-10-06T03:00:00 | 2026-10-06T04:00:00 | false |
| 2026-10-06T03:00:00 | 2026-10-07T03:00:00 | true |
Scenario: Forever never runs
Scenario: Forever with no size limit never runs
Given the retention is 0
When Tick runs at 2026-10-06T03:00:00
Then ran is false and DeleteOlderThan is not called
@@ -559,3 +559,10 @@ Feature: The back-office keeps an append-only activity log that owners can read
When the owner sends GET /v1/activity
Then the three events are listed newest first with their documented items
And GET /v1/activity?security=true lists only the failed login
Scenario: The size limit extension amends the retention rules
Given features/activity_home_card_and_retention.feature, rule "Retention size limit"
Then the settings default also has activity_retention_max_gb 0 and activity_log_size_bytes 0
And PATCH /v1/settings also accepts activity_retention_max_gb validated against audit.SizeOptions()
And the last 24 hours are never removed by DeleteOldest either
And a retention change is recorded with the max_gb_from and max_gb_to params when the size changes
features/activity_home_card_and_retention.feature
+269
View File
@@ -347,3 +347,272 @@ Feature: Recent activity card on Home and Retention card on Settings
Scenario: Existing behaviour is unchanged
Then every scenario of home.feature, settings_page.feature (except its exact-body statement, superseded above) and the Media cleanup scenarios still holds with GET /v1/activity?size=5 mocked and the settings mock widened with the four activity_* fields
Rule: Retention size limit, an optional maximum size as an alternative to the age limit
Background:
Given this rule supersedes the statements above where they conflict: the settings body, the next-cleanup rule, the PATCH validation, the card copy and the shortening rule
And this extension changes Go code in the audit settings, the audit repositories, the settings and cleanup use cases and the settings handler
And GET /v1/settings and the PATCH response carry exactly {"version", "activity_retention_days", "activity_retention_max_gb", "activity_log_size_bytes", "activity_last_cleanup_at", "activity_last_cleanup_removed", "activity_next_cleanup_at"}, snake_case
And activity_next_cleanup_at is null only when activity_retention_days is 0 and activity_retention_max_gb is 0, otherwise it is the next 03:00 (audit.CleanupHour) in the server zone strictly after now, so Forever with a size limit has a schedule
And the PATCH body is a partial object with activity_retention_days in {0, 30, 90, 365} and/or activity_retention_max_gb in {0, 1, 5, 10, 25, 50}, absent keys are left unchanged, and an empty object, a null, a wrong type, any other value or malformed JSON answers 400 with apperr.BodyErrorMessage and saves nothing
And a non-owner PATCH answers 403 "Only the owner can do this."
And validation goes through validate tags and validator.Validate.Struct
And the size options are one list in the audit domain, audit.SizeOptions(), mirroring RetentionOptions()
And activity_log_size_bytes is the payload size of the activity events table, measured by audit.Repository.SizeBytes: the sum of the OCTET_LENGTH of the text columns action, actor_name, target_type, target_id, target_name, failure_reason, ip, user_agent, params and changes plus a fixed per-row overhead constant defined in the domain
And it is not the database file size and excludes indexes, free pages and other tables, and repositories never touch files
And the audit Repository gains SizeBytes() (int64, error) and DeleteOldest(limit int, before time.Time) (int, error); DeleteOldest removes the oldest events by created_at then id and only those older than before
And the cleanup runs the age step when days is greater than 0, then removes the oldest events in batches through DeleteOldest, calling SizeBytes between batches, until the log is at or under the limit or a batch removes 0
And the cutoff of every removal is the 24 hour guard (audit.KeepRecentHours); when the limit cannot be met because of the guard, the cleanup stops without error, records the run and tries again at the next 03:00
And Tick returns early only when days is 0 and max_gb is 0
And the removed count is the total of the age and size removals, in the last-cleanup fields and in the system.activity_cleanup params {"removed": n}
And system.activity_retention_changed is recorded once per PATCH that changes something, with params {"from","to"} when the age changes and "max_gb_from" and "max_gb_to" when the size changes, and no event when nothing changes
And lib/activitySentence.ts has an English and a French sentence for a size change
And GET /v1/settings computes the size on every call and a SizeBytes failure answers 500, like any use case error; every verified user receives the number, only the owner sees the card
And the SPA shows the size in GB (1 GB is 1024^3 bytes) with one decimal, or "Less than 0.1 GB" below 0.1 GB
And the header description is "How long activity is kept, and how much log data to keep. Whichever limit is reached first removes the oldest events. Only owners can change it."
And the append-only alert reads "Events can't be edited or deleted from the app. Only the retention limits remove them."
And below the age select an "or" divider is followed by the label "Keep at most" over a USelectMenu 160 px wide (full width on mobile) with the items "No size limit", "1 GB", "5 GB", "10 GB", "25 GB", "50 GB" (values 0, 1, 5, 10, 25, 50)
And the age helper is "Older events are removed automatically." without a size limit, "Events older than this are removed." with one, "Nothing is removed automatically. The log grows over time." for Forever without a size limit and "No age limit. Only the size limit applies." for Forever with one
And the size helper is "Optional. Logs can grow without a size cap." without a limit and "When logs pass 5 GB, the oldest events are removed. Currently 3.2 GB." with one, the figure being activity_log_size_bytes and never a hard-coded value
And the size rank is 1 < 5 < 10 < 25 < 50 < No size limit and a lower rank than the saved one is "smaller"
And choosing a smaller size, including a size when there was none, sends nothing and shows the warning "Events over {n} GB will be removed" with "This happens at the next cleanup on {date}. It can't be undone.", "Apply {n} GB" and "Cancel"; choosing a larger size or "No size limit" saves at once
And each select sends only its own key, both share one status area, one error alert and one pending warning, and a warning concerns one control at a time: a longer or larger choice on either control drops it and another shortening replaces it
And the visual states are "Default, 90 days or 5 GB", "Age only (no size limit)", "Size only (forever)", the shortening warning with a 5 GB limit, Forever and Save error
Scenario Outline: Settings API, GET reports the size limit and the measured size
Given the saved retention is <days> days with a size limit of <gb> GB and the activity table holds <bytes> bytes of payload
When a verified user sends GET /v1/settings at the server time 2026-10-06T10:00:00+02:00
Then the status is 200 and the body has exactly the seven keys
And "activity_retention_max_gb" is <gb> and "activity_log_size_bytes" is <bytes>
And "activity_next_cleanup_at" is <next>
Examples:
| days | gb | bytes | next |
| 90 | 0 | 0 | "2026-10-07T03:00:00+02:00" |
| 90 | 5 | 3435973837 | "2026-10-07T03:00:00+02:00" |
| 0 | 5 | 3435973837 | "2026-10-07T03:00:00+02:00" |
| 0 | 0 | 1024 | null |
Scenario: Settings API, a fresh install defaults to no size limit
Given no settings row exists
When a verified user sends GET /v1/settings
Then activity_retention_days is 90, activity_retention_max_gb is 0 and activity_log_size_bytes is 0
Scenario Outline: Settings API, PATCH saves a size limit, an age, or both
Given the saved retention is 90 days and no size limit and the user is the owner
When the owner sends PATCH /v1/settings with <body>
Then the status is 200 and the body has activity_retention_days <days>, activity_retention_max_gb <gb> and activity_next_cleanup_at <next>
And the repository holds <days> and <gb>
Examples:
| body | days | gb | next |
| {"activity_retention_max_gb": 5} | 90 | 5 | the next 03:00 server time |
| {"activity_retention_days": 0, "activity_retention_max_gb": 5} | 0 | 5 | the next 03:00 server time |
| {"activity_retention_days": 30} | 30 | 0 | the next 03:00 server time |
| {"activity_retention_days": 0} | 0 | 0 | null |
Scenario: Settings API, leaving the size limit while Forever clears the schedule
Given the saved retention is 0 days with a size limit of 5 GB
When the owner sends PATCH /v1/settings with {"activity_retention_max_gb": 0}
Then activity_next_cleanup_at is null
Scenario Outline: Settings API, PATCH rejects invalid size input
Given the saved retention is 90 days with a size limit of 5 GB
When the owner sends PATCH /v1/settings with <body>
Then the status is 400 with the text of apperr.BodyErrorMessage and nothing is saved
Examples:
| body |
| {"activity_retention_max_gb": 2} |
| {"activity_retention_max_gb": -1} |
| {"activity_retention_max_gb": 100} |
| {"activity_retention_max_gb": 5.5} |
| {"activity_retention_max_gb": "5"} |
| {"activity_retention_max_gb": null} |
| {"activity_retention_days": 30, "activity_retention_max_gb": 2} |
| {} |
Scenario: Settings API, a PATCH changing nothing is a no-op
Given the saved retention is 90 days and 5 GB
When the owner sends PATCH /v1/settings with {"activity_retention_days": 90, "activity_retention_max_gb": 5}
Then the status is 200, nothing is saved and no audit event is recorded
Scenario: Settings API, a non-owner cannot set the size limit
When an admin sends PATCH /v1/settings with {"activity_retention_max_gb": 5}
Then the status is 403 with "Only the owner can do this." and nothing is saved
Scenario: Settings API, a failing size measurement answers 500
Given SizeBytes fails
When a verified user sends GET /v1/settings
Then the status is 500 with the generic internal error text
Scenario Outline: Settings API, the change is audited
Given the saved retention is 90 days and no size limit
When the owner sends PATCH /v1/settings with <body>
Then one event "system.activity_retention_changed" exists with params <params>
Examples:
| body | params |
| {"activity_retention_days": 30} | {"from": 90, "to": 30} |
| {"activity_retention_max_gb": 5} | {"max_gb_from": 0, "max_gb_to": 5} |
| {"activity_retention_days": 30, "activity_retention_max_gb": 5} | {"from": 90, "to": 30, "max_gb_from": 0, "max_gb_to": 5} |
Scenario Outline: Cleanup, the size limit removes the oldest events
Given the retention is <days> days, the size limit is 5 GB and the log is 7 GB
And the oldest events are older than the guard
When the cleanup runs at 2026-10-06T03:00:00
Then events are removed oldest first, by created_at then id, until SizeBytes is at or under 5 GB
And the removed count is the total of the age removals and the size removals
And the last cleanup time is the clock's time
Examples:
| days |
| 90 |
| 0 |
Scenario: Cleanup, age first then size
Given the retention is 30 days, the size limit is 5 GB, 100 events are older than 30 days and the log still exceeds 5 GB after they go
When the cleanup runs
Then the 100 events go first, then the oldest remaining events until the log is at or under 5 GB
Scenario: Cleanup, the age limit alone is enough
Given the log is under the size limit after the age removal
When the cleanup runs
Then no size removal happens
Scenario: Cleanup, Forever with no size limit never runs
Given the retention is 0 days and no size limit
When Tick runs at 2026-10-06T03:00:00
Then ran is false and neither DeleteOlderThan nor DeleteOldest is called
Scenario: Cleanup, the last 24 hours are protected even when the size cannot be met
Given the size limit is 1 GB and the log is 4 GB, all of it from the last 24 hours
When the cleanup runs
Then nothing is removed, no error is returned, the last cleanup records 0 removed and the next tick tries again
Scenario: Cleanup, a failing size removal is retried
Given DeleteOldest fails
When Tick runs
Then the error is returned and logged, the last cleanup fields are unchanged and the next tick runs again
Scenario: Cleanup, a size removal records one system event
Given the age step removed 10 events and the size step removed 40
When the cleanup runs
Then one "system.activity_cleanup" event exists with params {"removed": 50}
Scenario: Persistence measures the table payload portably
Given a real in-memory SQLite database and events with known text lengths, one with a multi-byte name
Then SizeBytes equals the sum of the byte lengths of the text columns plus the fixed per-row overhead
And it is 0 for an empty table
And deleting events lowers it
Scenario: Persistence deletes the oldest events up to a limit and a guard
Given 10 events, 3 of them newer than the guard
When DeleteOldest runs with limit 5
Then the 5 oldest events are removed
When DeleteOldest runs with limit 20
Then only the 2 remaining events older than the guard are removed and the 3 newest stay
Scenario: Settings retention, Default with 90 days or 5 GB
Given the role is "owner" and GET /v1/settings resolves 200 with activity_retention_days 90, activity_retention_max_gb 5 and activity_log_size_bytes 3435973837
When the owner navigates to /settings
Then the age select shows "90 days" with the helper "Events older than this are removed."
And an "or" divider and "Keep at most" show "5 GB" with the helper "When logs pass 5 GB, the oldest events are removed. Currently 3.2 GB."
And the lock alert reads "Events can't be edited or deleted from the app. Only the retention limits remove them."
And "Next cleanup" shows the date from activity_next_cleanup_at
Scenario: Settings retention, Age only (no size limit)
Given activity_retention_max_gb is 0
When the owner navigates to /settings
Then the size select reads "No size limit" with the helper "Optional. Logs can grow without a size cap." and the age helper is "Older events are removed automatically."
Scenario: Settings retention, Size only (forever)
Given activity_retention_days is 0, activity_retention_max_gb is 5 and activity_next_cleanup_at is set
When the owner navigates to /settings
Then the age helper reads "No age limit. Only the size limit applies." and "Next cleanup" shows the date, not "Not scheduled"
Scenario: Settings retention, Forever with no size limit
Given activity_retention_days is 0, activity_retention_max_gb is 0 and activity_next_cleanup_at is null
When the owner navigates to /settings
Then the age helper reads "Nothing is removed automatically. The log grows over time." and "Next cleanup" reads "Not scheduled"
Scenario Outline: Settings retention, the measured size is formatted
Given the size limit is 5 GB and activity_log_size_bytes is <bytes>
When the owner navigates to /settings
Then the size helper ends with "<shown>"
Examples:
| bytes | shown |
| 3435973837 | Currently 3.2 GB. |
| 0 | Currently Less than 0.1 GB. |
| 53687091 | Currently Less than 0.1 GB. |
| 5368709120 | Currently 5.0 GB. |
Scenario Outline: Settings retention, a larger size or no limit saves at once
Given the saved size is <from>
When the owner chooses "<to>"
Then PATCH is sent once with {"activity_retention_max_gb": <value>}, never the age key, and no warning shows
And the shared status reads "Saving…" then "Saved"
Examples:
| from | to | value |
| 5 GB | 10 GB | 10 |
| 5 GB | No size limit | 0 |
| 1 GB | 50 GB | 50 |
Scenario Outline: Settings retention, a smaller size asks first
Given the saved size is <from>
When the owner chooses "<to>"
Then no PATCH is sent and the size select shows "<to>"
And a warning alert shows "Events over <n> GB will be removed" and "This happens at the next cleanup on 7 Oct 2026, 03:00. It can't be undone." with "Apply <n> GB" and "Cancel"
Examples:
| from | to | n |
| 10 GB | 5 GB | 5 |
| No size limit | 5 GB | 5 |
| 50 GB | 1 GB | 1 |
Scenario: Settings retention, Apply and Cancel for a smaller size
Given the warning for 5 GB is shown on a saved 10 GB
When the owner activates "Apply 5 GB" and PATCH resolves 200 with activity_retention_max_gb 5
Then PATCH was sent once with {"activity_retention_max_gb": 5} and "Saved" shows
Given the warning is shown again
When the owner activates "Cancel"
Then the size select reads the saved value and no PATCH was sent
Scenario: Settings retention, only one warning at a time
Given the age warning for 30 days is open
When the owner chooses "5 GB" as a smaller size
Then the age warning is replaced by the size warning and the age select reverts to the saved value
When the owner chooses a longer age or a larger size
Then the warning is dropped and PATCH is sent once for that control
Scenario: Settings retention, switching age to Forever with a size limit
Given the saved period is 90 days and 5 GB
When the owner chooses "Forever"
Then PATCH is sent with {"activity_retention_days": 0} and the age helper becomes "No age limit. Only the size limit applies."
And "Next cleanup" still shows the returned date
Scenario Outline: Settings retention, a size save error reverts only the size select
Given the saved size is 5 GB and the owner chooses "10 GB"
When PATCH /v1/settings <failure>
Then the error alert "Couldn't save the retention period" with "Your previous setting is still in place." and "Try again" shows
And the size select reads "5 GB" again and the age select is unchanged
When the owner activates "Try again" and PATCH resolves 200
Then PATCH was sent a second time with {"activity_retention_max_gb": 10} and "Saved" replaces the alert
Examples:
| failure |
| resolves with a 500 response |
| resolves with a 400 response |
| resolves with a 403 response |
| never reaches the server |
Scenario: Settings retention, offline disables the size select too
Given the owner is on /settings and the browser goes offline
Then both selects, "Apply" and "Cancel" are disabled
Scenario: The French interface covers the size strings
Given the language is French
Then every new string has a French entry in messages.ts, and the Activity sentence for a size change is French
features/activity_home_card_and_retention.qa.md
+48
View File
@@ -72,3 +72,51 @@ Settings (mount SettingsView, GET /v1/settings mocked with the five-key body):
- [ ] Home at 1440 and 390 px, light and dark: 420 px column in the far-right of the full content width, 16 px gap, row clamping, "View all" 44 px touch target; the article column fills the remaining width and still reads well.
- [ ] Keyboard and focus: Home tab order "New article", article rows, "View all"; Settings select, Apply, Cancel in order and focus back on the select after Cancel.
- [ ] Hover, focus rings, the 2 s "Saved" fade and reduced motion.
## Retention size limit extension
Spec: rule "Retention size limit" in activity_home_card_and_retention.feature. Items 14 to 25 above are updated for the widened mock body (activity_retention_max_gb, activity_log_size_bytes) and the new helper and alert copy.
Implementation note for coder (informational, not a QA assertion):
- audit.Settings gains RetentionMaxGB; the settings use case result carries LogSizeBytes from audit.Repository.SizeBytes.
- audit.SizeOptions() is [0, 1, 5, 10, 25, 50]; invalid values return ErrInvalidRetention.
- audit.Repository gains SizeBytes() (int64, error) and DeleteOldest(limit int, before time.Time) (int, error).
- audit.SettingsRepository.SaveRetention takes days and max_gb; the model and mapper gain retention_max_gb.
- The cleanup use case runs the age step when days > 0, then loops DeleteOldest batches with SizeBytes until the log fits or a batch removes 0; the cutoff is the 24 h guard.
- The handler request struct is a partial with pointer fields and validate tags; an empty object is rejected.
- settingsResponse gains activity_retention_max_gb and activity_log_size_bytes. Update the settings_page tests that count keys.
### Go (httptest, fakes)
- G11. GET key set is exactly the seven keys; max_gb and size_bytes carry the fake's values; a fresh fake gives 90, 0, 0.
- G12. GET next-cleanup table: (90, 0), (90, 5), (0, 5) are non-null; (0, 0) is JSON null with the key present.
- G13. PATCH table: size alone, age alone, both, Forever plus size, size 0 with Forever (next becomes null); the fake repository holds the pair.
- G14. PATCH invalid: sizes 2, -1, 100, 5.5, "5", null, mixed with a valid age, {}, not JSON; each 400 with the BodyErrorMessage text and nothing saved. Non-owner 403, no cookie 401.
- G15. No-op PATCH of both saved values: 200, no save, no event.
- G16. Audit params table for the three bodies (age, size, both).
- G17. Cleanup use case with a fake events repository that has a size model: age then size, Forever plus size, Forever with no size (ran=false, no calls), size already met (no DeleteOldest), the guard stops at 24 h with 0 removed and no error, DeleteOldest error propagates and leaves the last cleanup unchanged, total removed in the event params and in SaveLastCleanup, the batch loop terminates when a batch removes 0.
- G18. CleanupDue table unchanged; Tick runs for Forever when a size limit is set.
- G19. Domain: SizeOptions validation; next-cleanup is nil only when both are 0.
- G20. Persistence on in-memory SQLite (extend test/activity_persistence_test.go): SizeBytes with multi-byte text, the empty table, after delete; DeleteOldest with the limit and the guard; the migrated model has retention_max_gb defaulting to 0.
- G21. The interface-shape test is updated to the six methods.
- G22. A SizeBytes error propagates and the handler answers 500 "Internal server error".
### SPA (Vitest), extending web/src/views/__tests__/SettingsRetention.spec.ts
- 26. Default 90 days or 5 GB; Age only; Size only (forever); Forever without a size: selects, helpers, "Currently 3.2 GB.", the "or" divider, the lines.
- 27. Size formatting table (3.2, "Less than 0.1 GB", 5.0); 1024^3 base.
- 28. Larger size or No size limit: one PATCH with only the size key, "Saving…" then "Saved" (fake timers).
- 29. Smaller size, and a size when none was set: no PATCH, warning copy, Apply sends one PATCH, Cancel reverts.
- 30. One warning at a time; a longer or larger choice drops it.
- 31. Save errors (500, 400, 403, network): only the size select reverts; Try again re-sends the same body without re-confirming; 403 refreshes the session and the card disappears.
- 32. Offline disables both selects and Apply and Cancel; 401 is global.
- 33. French parity for the new strings; lib/activitySentence.spec.ts covers the size-change sentence in English and French.
### Visual (task visual:check)
The board draws the card in group "Settings: Retention" with the states 'Age only (no size limit)' and 'Size only (forever)'. In settings.md the states are bullets, not headings, so confirm on the first run whether the tool iterates them. Setups use the board's own sample data (5 GB, 3.2 GB) and the owner /v1/settings mock gets the widened body. Clock-sensitive states stay skip: 'clock-sensitive'. No exception beforehand.
### Manual verification
- [ ] Real SQLite file: activity_log_size_bytes grows as events are added and falls after a forced cleanup.
- [ ] Forced cleanup with a small limit removes the oldest events and leaves the last 24 h.
- [ ] Optional: SizeBytes on real MySQL and PostgreSQL.
- [ ] Real PATCH as owner for each option, as admin 403, as owner with 2 GB 400.
- [ ] Keyboard order: age select, size select, Apply, Cancel; focus returns to the triggering select after Cancel.
- [ ] 1440 and 390 px, light and dark: the "or" divider and the 160 px or full-width size select as drawn in the board.