[16.0][FIX] fs_attachment: paginate autovacuum GC loop to bound worker memory

[TecnologiaIG]

Problem

FsFileGC._gc_files_unsafe loads the entire backlog of orphan files into a single Python list (via array_agg(store_fname) grouped by storage) and iterates fs.rm over all of them in one shot. With the Azure Blob backend (adlfs 2026.4.0; same class of issue applies to s3fs and any fsspec-based client) and a large queue, each HEAD+DELETE pair retains response buffers and connection-pool state inside the SDK client — they are only released when the worker exits.

Observed in production

• Odoo.sh instance, Azure Blob backend, ~30 k orphans queued.
• Every @api.autovacuum run hit Odoo's limit_memory_hard and received SIGKILL at around minute 17 of the loop.
• 60 k+ blob requests per run, zero DELETE committed (the process died mid-loop), queue never drained, respawned worker re-ran the same failing loop.
• 14 kills in a single 24 h window, worker lived 16-21 min each round.
• Operationally mitigated with TRUNCATE fs_file_gc (after CSV backup) — at the cost of a small set of orphan blobs leaking on Azure, but acceptable vs. nightly SIGKILL storms.

Fix

Paginate the SELECT + fs.rm loop per storage, in batches of 500, with an explicit gc.collect() between batches to reclaim SDK buffers:

for code in codes:
    fs = self.env["fs.storage"].get_fs_by_code(code)
    while True:
        rows = SELECT ... LIMIT 500
        if not rows: break
        for (store_fname,) in rows:
            fs.rm(...)
            deleted.append(store_fname)
        DELETE FROM fs_file_gc WHERE store_fname = ANY(deleted)
        gc.collect()

• No intermediate commits. The caller _gc_files still holds the SHARE lock on fs_file_gc / ir_attachment and commits at the end — transactional semantics are unchanged.
• Memory is bounded: at most 500 fnames + one SDK response cycle in flight at any time.
• Retries on the next cron tick remain idempotent: any batch whose deletion fails is left in the table and picked up on the next run.

Note on stacking

This PR stacks on top of #596 ([FIX] fs_attachment: bound GC cursor with per-transaction timeouts). Version is bumped to 16.0.2.0.3 to leave room for #596 landing first at 16.0.2.0.2. Happy to rebase as needed.

Test plan

• Validated in production after TRUNCATE: no OOM kills since (~22:00 GT 21-abr).
• Staging reproduction: queue 5 000 synthetic orphan rows in fs_file_gc, run _gc_files(), observe worker RSS — should stay flat around the baseline instead of climbing linearly.
• Concurrent _mark_for_gc during GC: upload's INSERT ... ON CONFLICT takes ROW EXCLUSIVE which conflicts with the caller's SHARE lock → upload waits only for the duration of a batch (~sub-second), not the full queue.

Forward-ports

Code is identical on 17.0 / 18.0 / 19.0 as of today's tip; happy to forward-port once 16.0 is reviewed.

[OCA-git-bot]

Hi @lmignon,
some modules you are maintaining are being modified, check this out!

[lmignon]

Tank you for your proposal. This makes the code more robust since it becomes less sensitive to the amount of data the garbage collector has to process. I'm still not sure about the first part of the change (#596) But this one LGTM. 😏 So let's discuss about #596 so we can move forward.

[TecnologiaIG]

Force-pushed 5cce247a fixing a correctness regression in the original patch: every row the loop attempted is now cleared from fs_file_gc regardless of fs.rm outcome. Previous version only deleted on fs.rm success, which caused an infinite loop (and the 6h CI timeout) whenever a batch failed end-to-end.

Diff vs. the earlier version:

-                deleted = []
-                for (store_fname,) in rows:
+                fnames = [row[0] for row in rows]
+                for store_fname in fnames:
                     try:
                         fs.rm(store_fname.partition("://")[2])
-                        deleted.append(store_fname)
                     except Exception:
                         _logger.debug("Failed to remove file %s", store_fname)
-                if deleted:
-                    self._cr.execute(
-                        "DELETE FROM fs_file_gc WHERE store_fname = ANY(%s)",
-                        (deleted,),
-                    )
+                self._cr.execute(
+                    "DELETE FROM fs_file_gc WHERE store_fname = ANY(%s)",
+                    (fnames,),
+                )

Semantics now match the pre-batching upstream: an fs.rm that fails leaks a blob (debug-logged), same as before, but the DB row is cleared so the loop can terminate.

[lmignon]

@TecnologiaIG This looks like a bug into the related fsspec's client backends... But batching process could be a good idea. Taking into account your PR and #599 I've the feeling that we must define a proper api that will allow to share the same methods between the 2 addons. I'll think about and make a proposal.

[lmignon]

@TecnologiaIG Can you take a look at #610? I added a method to retrieve the files to be garbage collected so it can be reused by for example #599. I also removed the bump on the version number into your commit since it will be done at merge. If it's OK for you, I'll merge this one in the coming hours.