refac: article reads, story delete, no leaks, stored SEO status, rate limit, cleanup (%site%, article repo split, old lang repos, one default per site rule to domain)

docs/performance.md

@@ -2,17 +2,18 @@
 
 ## The `?seo=` article filter
 
-`GET /v1/article/` assesses the SEO status of articles in memory, because the status depends on the body, the site template and the authors. Every request assesses every visible article once, to fill `counts.suggestions`, and `?seo=` reuses that assessment to filter before paging. The cost therefore grows with the number of articles, not with the page size.
+The SEO status of an article depends on its body, its fields, its authors, its site (`default_description`) and the site default language. It is computed by `RefreshArticleSEOUseCase` and stored on the article (`seo_status`, `seo_suggestions`). Every repository write to one of those inputs clears the stored status and raises `seo_version` in the same `UPDATE` (`withStaleSEO`); site-level changes (default description, default language) clear the whole site through `article.SEOIndex.MarkSiteSEOStale`. Before answering, `GET /v1/article/` recomputes the cleared articles only, in batches of 500, reading their bodies once; `StoreSEO` writes a status only while `seo_version` is still the one it read, so a concurrent edit is never overwritten by an older assessment; the site and its languages are read after each batch of stale rows, never cached across batches, so a site-level change that commits during a refresh is either seen or makes the store miss. The server recomputes everything that is cleared at startup (the backfill of existing rows after the upgrade). `?seo=` and `counts.suggestions` are then plain SQL conditions on `seo_status`, so the cost of a list no longer grows with the number of visible bodies.
 
-`TestSeoFilterPerformance` in `test/seo_filter_performance_test.go` (skipped with `-short`) measures 5,000 articles of about 5 KB each in one site on a temporary SQLite file, every article with suggestions, slowest of 3 runs:
+`TestSeoFilterPerformance` in `test/seo_filter_performance_test.go` (skipped with `-short`) measures 5,000 articles of about 5 KB each in one site on a temporary SQLite file, every article with suggestions, slowest of 3 runs after the startup backfill:
 
 | Request | Time |
 | --- | --- |
-| `?seo=suggestions&size=25` | 315 to 338 ms |
-| default page, no `?seo=` | 307 to 329 ms |
+| startup backfill of the 5,000 articles (once) | 270 ms |
+| `?seo=suggestions&size=25` | 24 ms |
+| default page, no `?seo=` | 23 ms |
 
-The target is 300 ms for 5,000 articles. Both requests sit slightly above it on the development machine, so the test logs the target but only fails above 500 ms; no cache or index was added. Re-run it with `go test -tags nofrontend ./test/... -run TestSeoFilterPerformance -v -count=1` after a change to the assessment rules or the article repository.
+The target is 300 ms for 5,000 articles; the test fails above 500 ms. Re-run it with `go test -tags nofrontend ./test/... -run TestSeoFilterPerformance -v -count=1` after a change to the assessment rules or the article repository.
 
 ## The `?seo=` filter of the grouped list
 
-`GET /v1/article/?group=story` uses the same assessment. The versions whose status matches (and, with `q`, whose title matches: the title search runs once per request through `article.Repository.FindTitleMatches`) are handed to the repository as `StoryFilters.VersionIDs`. That set can be larger than any bind-parameter limit (SQLite accepts 32,766 parameters), so it never becomes an `IN (?, …)` list: the repository reads the qualifying `(story_id, id)` pairs and the ordered story rows with id-only projections and keeps the members of the set in memory before paging. `ListByStories` reads the versions of the page without their bodies and only those the caller may read; the representative's body comes from the assessment pass, which still reads every visible body because the status depends on it. `TestArticleStoryListSEOFilterAtScaleSQLite` covers 40,001 versions.
+`GET /v1/article/?group=story` filters on the stored status in SQL (`q.seo_status = ?` among the conditions of the qualifying versions) and counts `suggestions` with `COUNT(DISTINCT story_id)` over the readable versions, so no id list is built for the SEO filter. With `q`, the title search still runs once per request through `article.StoryReader.FindTitleMatches` (accent folding happens in Go) and its matches are handed to the repository as `StoryFilters.VersionIDs`. That set can be larger than any bind-parameter limit (SQLite accepts 32,766 parameters), so it never becomes an `IN (?, …)` list: the repository reads the qualifying `(story_id, id)` pairs and the ordered story rows with id-only projections and keeps the members of the set in memory before paging. `ListByStories` reads the versions of the page without their bodies and only those the caller may read; `ListVersions` reads the bodies of the page's representatives only. `TestArticleStoryListSEOFilterAtScaleSQLite` covers 40,001 versions: the backfill assesses each of them once, and the list requests that follow read no stale version and no page of bodies.