refac: atomic merge, lost-update race, RunPurge cost, duplicated walks
This commit is contained in:
Florian Sylvain committed
1 parent
0fc3511e2b
commit
074944c99a
45 files changed
+2204
-267
No files matched your search
@@ -0,0 +1,274 @@
|
||||
Feature: Media merge hardening: atomic merge, no lost entry edits, one entry scan per purge
|
||||
|
||||
As a site owner or admin
|
||||
I want a duplicate merge to either fully happen or not happen at all
|
||||
So that a failure halfway never leaves articles, sites, authors, entries or the library half merged
|
||||
|
||||
Background:
|
||||
Given this slice is backend only: no route, request body, response body, status code, error text or audit payload changes
|
||||
And internal/domain gains a unit-of-work port (package transaction): Transactor with Run(ctx, func(Repositories) error) error, and Repositories holding the article, site, author, entry and image repositories bound to the open transaction
|
||||
And internal/domain/transaction imports only the stdlib and other internal/domain/* packages
|
||||
And Run commits when the function returns nil and rolls everything back when it returns an error or panics, and returns that error unchanged
|
||||
And the GORM adapter lives in internal/infrastructure/persistence, asserts var _ transaction.Transactor = &X{}, and is built only in api/dependecyInjection.go
|
||||
And SQL in the adapter stays portable across SQLite, MySQL/MariaDB and PostgreSQL
|
||||
And MergeMedia and RunPurge depend on transaction.Transactor and never on persistence, gorm or models
|
||||
And MergeMedia keeps its order: validate, load copies, plan which articles contain a copy name, then inside one transaction re-read each planned article body, apply the replacement of the copy names by the kept name to that re-read body and UpdateBody it, articles RepointImage, sites RepointImage, authors RepointAvatar, entry repoint and image row delete per copy
|
||||
And restoreBodies, the entry restore of entryMediaScanner and every logCompensation call of the merge are deleted; nothing compensates by hand any more
|
||||
And the stored file, the variant cache files and the variant cache rows of a merged copy are removed only after the transaction committed, through image.Storage and the variant file repository
|
||||
And a failure removing those files after the commit is written to the injected *log.Logger, never rolled back, never changes the response or the audit event
|
||||
And the audit event media.merge and the audit event media.purge are recorded outside the transaction, with the same params as today
|
||||
And entries are repointed from candidates found by one scan (contenttype.Repository.List then entry.Repository.ListPage with Unpaged, filtered in Go), and each candidate is re-read with entry.Repository.Get inside the transaction before it is written
|
||||
And an entry repoint only changes the media values that still point at a merged copy in the re-read data, keeps every other value of the re-read data, and leaves created_by, updated_by, created_at and updated_at untouched
|
||||
And a candidate that no longer exists at re-read time is skipped
|
||||
And RunPurge and PreviewPurge scan entries once per run and once per preview into an index of image id to entries, shared by the usage counts and by every duplicate group; the index is built in the use case and no repository method is added
|
||||
And RunPurge runs one transaction per duplicate group, so a failing group rolls back alone and the other groups still merge
|
||||
And the replacement of a copy name inside an article body matches whole stored file names only, so a name that is a suffix of another name is not rewritten inside it
|
||||
And the media-field walks (field-type loops, the required and type rule constants, the entry.InvalidField wrapper) are deduplicated into one place; this is a pure refactor and has no scenario of its own beyond "behaviour unchanged"
|
||||
And MergeMediaResult, purge.RunResult, the merge response {"articles_updated","merged"} (plus sites_updated and authors_updated where the sites and authors features already report them) and the 409, 404, 400 and 500 answers are unchanged
|
||||
And fixtures: images 7, 8 and 9 share sha256 A with stored names; image 2 and 3 share sha256 B; the type "event" has title (text, required) then thumbnail (media, optional); users: admin 2 (verified)
|
||||
|
||||
Rule: A merge is all-or-nothing
|
||||
|
||||
Scenario Outline: Any failing step leaves the library exactly as it was
|
||||
Given images 7 and 9 share sha256 A and 7 is the kept one
|
||||
And article 1 and article 2 contain the stored name of image 9 in their bodies, article 3 has image 9 as its cover
|
||||
And site 1 uses image 9 as its default social image and author 4 uses image 9 as its avatar
|
||||
And entries 41 and 42 of "event" hold thumbnail 9
|
||||
And <failing step> is armed to fail
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 500 and the body is "Internal server error"
|
||||
And the bodies of articles 1 and 2 are byte-identical to before
|
||||
And article 3 still has cover 9, site 1 still uses 9, author 4 still uses avatar 9
|
||||
And entries 41 and 42 still hold thumbnail 9 with updated_at and updated_by unchanged
|
||||
And images 7 and 9 both still exist
|
||||
And image.Storage.Delete and the variant file repository Delete are never called
|
||||
And one failed media.merge audit event is recorded
|
||||
|
||||
Examples:
|
||||
| failing step |
|
||||
| the UpdateBody of the first article |
|
||||
| the UpdateBody of the second article |
|
||||
| the article repository RepointImage |
|
||||
| the site repository RepointImage |
|
||||
| the author repository RepointAvatar |
|
||||
| the entry repository Get of the second entry |
|
||||
| the entry repository Update of the second entry |
|
||||
| the image repository Delete of image 9 |
|
||||
| the commit of the transaction |
|
||||
|
||||
Scenario: The transaction cannot start
|
||||
Given the transactor fails to begin
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 500
|
||||
And no repository other than the reads of the planning phase was called, and no file was touched
|
||||
|
||||
Scenario: Several copies roll back together
|
||||
Given images 7, 8 and 9 share sha256 A and the image repository Delete fails for image 9 after image 8 was deleted
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [8, 9]}
|
||||
Then the response status is 500
|
||||
And image 8 exists again, image 9 exists and no article, site, author or entry changed
|
||||
|
||||
Scenario: A successful merge still does everything
|
||||
Given the Background fixtures of the first scenario and no armed failure
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 200 and the body is {"articles_updated": 3, "merged": 1} plus the existing sites_updated and authors_updated keys
|
||||
And entries 41 and 42 hold thumbnail 7 with created_by, updated_by and updated_at unchanged
|
||||
And image 9 is gone from the repository and its stored file is removed
|
||||
|
||||
Scenario: A validation or lookup error never opens a transaction
|
||||
Given any of the existing 400, 404 or 409 merge requests
|
||||
Then the transactor Run is not called
|
||||
|
||||
Scenario: A rollback leaves updated_at and updated_by of entries untouched
|
||||
Given the entries 41 and 42 hold thumbnail 9 with known created_at, updated_at, created_by and updated_by
|
||||
And the repoint of entry 42 fails
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 500
|
||||
And entries 41 and 42 are byte-identical to before, including updated_at and updated_by
|
||||
|
||||
Rule: Stored files are removed only after the commit
|
||||
|
||||
Scenario: Files are touched after the commit and never before
|
||||
Given a merge of image 9 into image 7 that succeeds
|
||||
Then image.Storage.Delete("<stored name of 9>") is called after the transaction committed
|
||||
And the variant files of image 9 are removed after the commit
|
||||
And at no time before the commit is a file or a variant cache row of image 9 removed
|
||||
|
||||
Scenario Outline: A file failure after the commit is logged, not rolled back
|
||||
Given a merge of the copies 8 and 9 into image 7 where <file failure>
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [8, 9]}
|
||||
Then the response status is 200 and the body is the same as without the failure
|
||||
And images 8 and 9 are gone from the repository and the articles, sites, authors and entries stay repointed
|
||||
And the removal of the other copy is still attempted
|
||||
And the injected logger holds one line naming the stored name and the error
|
||||
And the media.merge audit event is a success with the usual params
|
||||
|
||||
Examples:
|
||||
| file failure |
|
||||
| image.Storage.Delete fails for image 9 |
|
||||
| the variant file purge of image 9 fails |
|
||||
|
||||
Scenario: A purge group with a file failure after the commit counts as purged
|
||||
Given groups A and B are planned and image.Storage.Delete fails for a merged copy of group A
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then the response status is 200 and the body has groups_purged 2 and failed_groups 0
|
||||
And the merged count includes the copies of group A
|
||||
And the injected logger holds the file failure
|
||||
|
||||
Rule: A concurrent entry edit survives a merge
|
||||
|
||||
Scenario: An edit made after the scan is kept
|
||||
Given entry 41 of "event" is {"title": "Workshop", "thumbnail": 9}
|
||||
And the entry repository changes entry 41 to {"title": "Workshop 2026", "thumbnail": 9}, with a new updated_at and updated_by, right after the scan returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 200
|
||||
And entry 41 is {"title": "Workshop 2026", "thumbnail": 7}
|
||||
And its updated_at and updated_by are the ones of the concurrent edit, not older ones
|
||||
|
||||
Scenario: A concurrent edit that moved the entry off the merged copy is not overwritten
|
||||
Given entry 41 holds thumbnail 9 when scanned
|
||||
And the entry repository changes entry 41 to thumbnail 8 right after the scan returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then entry 41 still holds thumbnail 8
|
||||
And the entry repository Update is not called for entry 41
|
||||
|
||||
Scenario: An entry deleted after the scan is skipped
|
||||
Given entry 41 holds thumbnail 9 when scanned
|
||||
And entry 41 is deleted right after the scan returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 200 and image 9 is gone
|
||||
|
||||
Scenario: Entries are written from the re-read data
|
||||
Given entry 41 holds {"title": "A", "thumbnail": 9, "poster": 9} when scanned
|
||||
When the merge runs
|
||||
Then entry 41 holds thumbnail 7 and poster 7 and the same title
|
||||
And the entry repository Update is called once for entry 41
|
||||
|
||||
Rule: A concurrent article edit survives a merge
|
||||
|
||||
Scenario: A body edited after planning is kept
|
||||
Given article 1 contains the stored name of image 9 and the planning has read its body
|
||||
And the article repository changes article 1 to the same text plus a new paragraph, right after the planning read returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 200
|
||||
And article 1's body holds the new paragraph and the stored name of image 7 instead of image 9
|
||||
|
||||
Scenario: A body edited so that it no longer holds the copy name is not rewritten
|
||||
Given article 1 contains the stored name of image 9 when planned
|
||||
And the article repository changes article 1 to a text without the stored name of image 9 right after the planning read returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then article 1's body is the edited text and the article repository UpdateBody is not called for article 1
|
||||
And article 1 is not counted in "articles_updated"
|
||||
|
||||
Scenario: An article deleted after planning is skipped
|
||||
Given article 1 contains the stored name of image 9 when planned
|
||||
And article 1 is deleted right after the planning read returns it
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then the response status is 200 and image 9 is gone
|
||||
|
||||
Scenario: Only whole stored names are replaced
|
||||
Given image 9 has the stored name "a.jpg" and article 1 contains "ba.jpg" and "a.jpg"
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 7, "merge_ids": [9]}
|
||||
Then article 1 still contains "ba.jpg" and no longer contains the standalone "a.jpg"
|
||||
|
||||
Rule: A cancelled purge stops cleanly
|
||||
|
||||
Scenario: A request cancelled between groups stops the run
|
||||
Given groups A, B and C are planned and the request context is cancelled after group A committed
|
||||
When the purge runs
|
||||
Then group A stays merged and groups B and C are untouched
|
||||
And the run returns the context error and records no last_purge_* values and no media.purge success event
|
||||
And failed_groups is not incremented for B and C
|
||||
|
||||
Scenario: The scheduler passes its context to the run
|
||||
Given the scheduler context is cancelled while a scheduled purge is between groups
|
||||
Then the run stops after the group in progress and records no last_purge_* values
|
||||
|
||||
Rule: A purge scans entries once
|
||||
|
||||
Scenario: A preview over many groups scans once
|
||||
Given 6 planned duplicate groups, 40 entries and 2 content types with a media field
|
||||
When the client sends GET /v1/media/purge-preview
|
||||
Then contenttype.Repository.List is called once
|
||||
And entry.Repository.ListPage is called once per content type with a media field, 2 times in all
|
||||
And the response is the same as before this slice
|
||||
|
||||
Scenario: A run over many groups scans once
|
||||
Given the same 6 groups, 40 entries and 2 content types with a media field
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then contenttype.Repository.List is called once
|
||||
And entry.Repository.ListPage is called 2 times in all, not once per group
|
||||
And the response and the stored last_purge_* values are the same as before this slice
|
||||
|
||||
Scenario: The index is stale after a group but the result is right
|
||||
Given entry 41 holds thumbnail 2 and poster 5, where 2 is a copy of group A and 5 is a copy of group B
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then entry 41 holds the kept image of group A as thumbnail and the kept image of group B as poster
|
||||
And entry.Repository.Update is called twice for entry 41
|
||||
|
||||
Scenario: The usage counts and the candidates come from the same scan
|
||||
Given the duplicate group 7, 9 where only 9 is used, by 1 entry
|
||||
When the purge preview and the purge run
|
||||
Then 9 is the kept image and 7 is the merged copy
|
||||
And there is one scan per call
|
||||
|
||||
Rule: A purge run commits one group at a time
|
||||
|
||||
Scenario Outline: A failing group rolls back alone
|
||||
Given groups A, B and C are planned, in that order
|
||||
And <failure> is armed for group B
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then the response status is 200
|
||||
And the body is {"groups_total": 3, "groups_purged": 2, "merged": <copies of A and C>, "articles_updated": <distinct articles of A and C>, "failed_groups": 1}
|
||||
And all of group B's article bodies, entries, sites, authors and images are back to their original state
|
||||
And groups A and C are merged
|
||||
And the stored last_purge_merged and last_purge_articles_updated equal the response totals
|
||||
|
||||
Examples:
|
||||
| failure |
|
||||
| the UpdateBody of the second article |
|
||||
| the article repository RepointImage |
|
||||
| the entry repository Update |
|
||||
| the image repository Delete of a merged copy |
|
||||
|
||||
Scenario: Every group failing is still a 200
|
||||
Given every group is armed to fail
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then the response status is 200
|
||||
And groups_purged is 0 and failed_groups equals groups_total
|
||||
And no image, body, entry, site or author changed and the stored last_purge_at is set
|
||||
|
||||
Scenario: The run opens one transaction per planned group
|
||||
Given 3 planned groups
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then the transactor Run is called 3 times
|
||||
|
||||
Rule: Behaviour unchanged (the media-field walk refactor)
|
||||
|
||||
Scenario: The shared walk changes nothing
|
||||
Given the media-field rule constants, the InvalidField wrapper and the walks over a content type's media fields live in one place
|
||||
Then every scenario of features/media_field_type.feature, media_library.feature and media_purge.feature passes unchanged, except the two amended below
|
||||
|
||||
Rule: Amendments to earlier features
|
||||
|
||||
Scenario: media_library.feature "A failure deleting a copy after the rewrite is 500 and the bodies are rolled back" is amended
|
||||
Given the image repository's Delete fails for image 3
|
||||
When the client sends POST /v1/media/merge with {"keep_id": 2, "merge_ids": [3]}
|
||||
Then the response status is 500
|
||||
And article 1's and article 2's bodies are back to their original text and image 3 still exists
|
||||
|
||||
Scenario: media_purge.feature "A group that fails after rewriting bodies" is amended
|
||||
Given the image repository's Delete fails for a merged image of group A, and group B is planned
|
||||
When the client sends POST /v1/media/purge with {}
|
||||
Then the response status is 200, "failed_groups" is 1 and "groups_purged" is 1
|
||||
And group A contributes 0 to "merged" and none of its article ids to "articles_updated"
|
||||
And group A's article bodies are back to their original text
|
||||
And a following GET /v1/media/purge-preview plans group A again, and the next run converges
|
||||
|
||||
@manual
|
||||
Rule: Persistence (verified by hand, see the qa.md)
|
||||
|
||||
Scenario: The GORM adapter rolls back on SQLite, MySQL/MariaDB and PostgreSQL
|
||||
Given duplicates with articles, a site, an author and entries pointing at a copy
|
||||
When a merge fails at the image delete on each driver
|
||||
Then no row changed on any of the three drivers
|
||||
Reference in new issue
Block a user