Research record: resolved CI disk incident; residual maintenance deferred #41

Closed
opened 2026-09-21 21:59:39 +00:00 by pyrrha · 1 comment
Member

Weekend disposition (2026-09-26): Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass.


Historical issue (preserved)

Impact

CI was red on every pull request for roughly 2.5 hours (onset between 16:17:58Z and 18:51:38Z on 2026-09-21). No workflow step executed. Every run died about one second in, before the first step, with:

failed to update bridge store for endpoint ... local-kv.db: no space left on device

Runs affected: 79 through 84 on #39, 78 on #38, and run 77. The last green run was 73.

Root cause

buildrunner (100.89.177.21), the host running act-runner.service, had a completely full root filesystem:

  • /dev/sda2 on /: 18G size, 18G used, 0 available, 100%
  • inodes were healthy (19%), so it was bytes, not inodes
  • /var/lib/docker accounted for 14G of the 18G

9.5 GB of that was a single file: the json-file log of container chromium-cdp (chromedp/headless-shell:latest, up 4 weeks). Chromium had been stuck in a tight error loop the whole time:

CommandBufferHelper::AllocateRingBuffer() failed
TransferBuffer::Initialize() failed

There was no /etc/docker/daemon.json, so Docker was on the default json-file driver with no max-size and no max-file. Unbounded growth, four weeks, disk gone.

Docker's own log confirms the mechanism on restart:

msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"

/etc/resolv.conf had been truncated to 0 bytes at 17:34:16 EDT — a dhcpcd renewal that ran while the filesystem was full and could not write its output. It was correctly regenerated at 17:57:24 EDT once space was available, and DNS recovered on its own.

Remediation applied

  1. Truncated the 9.5 GB log file. Free space went from 0 to 8.7G (49% used).
  2. Installed /etc/docker/daemon.json with log-driver: json-file, max-size: 50m, max-file: 3, and restarted Docker. Verified: a newly created container now reports maxsize=50m maxfile=3.
  3. Recreated chromium-cdp with the same configuration (image, entrypoint, 9222:9222, --shm-size 64m, restart=unless-stopped) plus explicit log limits, because an existing container does not inherit daemon-level log options.

Verified by execution, not inference: a throwaway PR at the same commit as #39 produced run 85, success, in 7 seconds. #40 was then closed and its branch deleted. Reopening #39 produced run 86, success at be64404, so its required check is green without moving the head.

Residual risks, not fixed

  • The root filesystem is 18G for a CI runner that also hosts a headless browser. It is currently 49% full. Unless something reclaims the ~2.8GB of idle Docker images and the 787MB volume, this has maybe one more bad week in it.
  • chromium-cdp is still error-looping. Its logs are now capped at 150 MB, so it can no longer fill the disk, but the loop is unresolved. Nobody has confirmed whether this container is still needed. It is also the only memory-hungry process on a host with 3.8 GB of RAM, which is a plausible cause of the GPU ring-buffer failures.
  • docker system prune -a would not have fixed this. docker system df reports only 2.8GB images / 89MB containers / 787MB volumes, so a prune would have freed ~2.5GB, looked effective, and left the disk at ~95%. The log truncation was the actual fix.
  • Failure onset vs. the last green run brackets the full-disk moment to 16:17:58.047Z to 18:51:38Z. Nothing else has been ruled out for that window, but the disk evidence is unambiguous.

Note on reboot behaviour

act-runner.service is a user unit at ~/.config/systemd/user/act-runner.service, enabled, with Linger=yes on pyrrha and Restart=on-failure. It does survive a reboot. A plain systemctl restart docker was sufficient here, so no reboot was performed.

**Weekend disposition (2026-09-26):** Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass. --- ## Historical issue (preserved) ## Impact CI was red on every pull request for roughly 2.5 hours (onset between `16:17:58Z` and `18:51:38Z` on 2026-09-21). No workflow step executed. Every run died about one second in, before the first step, with: ``` failed to update bridge store for endpoint ... local-kv.db: no space left on device ``` Runs affected: 79 through 84 on #39, 78 on #38, and run 77. The last green run was 73. ## Root cause `buildrunner` (`100.89.177.21`), the host running `act-runner.service`, had a **completely full root filesystem**: - `/dev/sda2` on `/`: 18G size, 18G used, **0 available, 100%** - inodes were healthy (19%), so it was bytes, not inodes - `/var/lib/docker` accounted for 14G of the 18G **9.5 GB of that was a single file**: the `json-file` log of container `chromium-cdp` (`chromedp/headless-shell:latest`, up 4 weeks). Chromium had been stuck in a tight error loop the whole time: ``` CommandBufferHelper::AllocateRingBuffer() failed TransferBuffer::Initialize() failed ``` There was no `/etc/docker/daemon.json`, so Docker was on the default `json-file` driver with no `max-size` and no `max-file`. Unbounded growth, four weeks, disk gone. Docker's own log confirms the mechanism on restart: ``` msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers" ``` `/etc/resolv.conf` had been truncated to 0 bytes at `17:34:16 EDT` — a `dhcpcd` renewal that ran while the filesystem was full and could not write its output. It was correctly regenerated at `17:57:24 EDT` once space was available, and DNS recovered on its own. ## Remediation applied 1. Truncated the 9.5 GB log file. Free space went from **0 to 8.7G (49% used)**. 2. Installed `/etc/docker/daemon.json` with `log-driver: json-file`, `max-size: 50m`, `max-file: 3`, and restarted Docker. Verified: a newly created container now reports `maxsize=50m maxfile=3`. 3. Recreated `chromium-cdp` with the same configuration (image, entrypoint, `9222:9222`, `--shm-size 64m`, `restart=unless-stopped`) plus explicit log limits, because an existing container does not inherit daemon-level log options. Verified by execution, not inference: a throwaway PR at the same commit as #39 produced **run 85, success**, in 7 seconds. #40 was then closed and its branch deleted. Reopening #39 produced **run 86, success** at `be64404`, so its required check is green without moving the head. ## Residual risks, not fixed - **The root filesystem is 18G** for a CI runner that also hosts a headless browser. It is currently 49% full. Unless something reclaims the ~2.8GB of idle Docker images and the 787MB volume, this has maybe one more bad week in it. - **`chromium-cdp` is still error-looping.** Its logs are now capped at 150 MB, so it can no longer fill the disk, but the loop is unresolved. Nobody has confirmed whether this container is still needed. It is also the only memory-hungry process on a host with 3.8 GB of RAM, which is a plausible cause of the GPU ring-buffer failures. - **`docker system prune -a` would not have fixed this.** `docker system df` reports only 2.8GB images / 89MB containers / 787MB volumes, so a prune would have freed ~2.5GB, looked effective, and left the disk at ~95%. The log truncation was the actual fix. - Failure onset vs. the last green run brackets the full-disk moment to `16:17:58.047Z` to `18:51:38Z`. Nothing else has been ruled out for that window, but the disk evidence is unambiguous. ## Note on reboot behaviour `act-runner.service` is a **user** unit at `~/.config/systemd/user/act-runner.service`, enabled, with `Linger=yes` on `pyrrha` and `Restart=on-failure`. It does survive a reboot. A plain `systemctl restart docker` was sufficient here, so no reboot was performed.
Author
Member

Weekend research-lead disposition (2026-09-26)

Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass.

## Weekend research-lead disposition (2026-09-26) Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass.
pyrrha changed title from buildrunner: unbounded container log filled the root filesystem and broke CI for ~2.5h to Research record: resolved CI disk incident; residual maintenance deferred 2026-09-26 04:42:23 +00:00
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#41
No description provided.