Research record: resolved CI disk incident; residual maintenance deferred #41
Labels
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany#41
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Weekend disposition (2026-09-26): Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass.
Historical issue (preserved)
Impact
CI was red on every pull request for roughly 2.5 hours (onset between
16:17:58Zand18:51:38Zon 2026-09-21). No workflow step executed. Every run died about one second in, before the first step, with:Runs affected: 79 through 84 on #39, 78 on #38, and run 77. The last green run was 73.
Root cause
buildrunner(100.89.177.21), the host runningact-runner.service, had a completely full root filesystem:/dev/sda2on/: 18G size, 18G used, 0 available, 100%/var/lib/dockeraccounted for 14G of the 18G9.5 GB of that was a single file: the
json-filelog of containerchromium-cdp(chromedp/headless-shell:latest, up 4 weeks). Chromium had been stuck in a tight error loop the whole time:There was no
/etc/docker/daemon.json, so Docker was on the defaultjson-filedriver with nomax-sizeand nomax-file. Unbounded growth, four weeks, disk gone.Docker's own log confirms the mechanism on restart:
/etc/resolv.confhad been truncated to 0 bytes at17:34:16 EDT— adhcpcdrenewal that ran while the filesystem was full and could not write its output. It was correctly regenerated at17:57:24 EDTonce space was available, and DNS recovered on its own.Remediation applied
/etc/docker/daemon.jsonwithlog-driver: json-file,max-size: 50m,max-file: 3, and restarted Docker. Verified: a newly created container now reportsmaxsize=50m maxfile=3.chromium-cdpwith the same configuration (image, entrypoint,9222:9222,--shm-size 64m,restart=unless-stopped) plus explicit log limits, because an existing container does not inherit daemon-level log options.Verified by execution, not inference: a throwaway PR at the same commit as #39 produced run 85, success, in 7 seconds. #40 was then closed and its branch deleted. Reopening #39 produced run 86, success at
be64404, so its required check is green without moving the head.Residual risks, not fixed
chromium-cdpis still error-looping. Its logs are now capped at 150 MB, so it can no longer fill the disk, but the loop is unresolved. Nobody has confirmed whether this container is still needed. It is also the only memory-hungry process on a host with 3.8 GB of RAM, which is a plausible cause of the GPU ring-buffer failures.docker system prune -awould not have fixed this.docker system dfreports only 2.8GB images / 89MB containers / 787MB volumes, so a prune would have freed ~2.5GB, looked effective, and left the disk at ~95%. The log truncation was the actual fix.16:17:58.047Zto18:51:38Z. Nothing else has been ruled out for that window, but the disk evidence is unambiguous.Note on reboot behaviour
act-runner.serviceis a user unit at~/.config/systemd/user/act-runner.service, enabled, withLinger=yesonpyrrhaandRestart=on-failure. It does survive a reboot. A plainsystemctl restart dockerwas sufficient here, so no reboot was performed.Weekend research-lead disposition (2026-09-26)
Closed as a retained incident record, not as a claim that every host issue is fixed. This issue already records log-cap remediation and successful runs 85/86; current PR #66/#67 CI also succeeded. Remaining browser-loop/capacity questions are infrastructure maintenance, not an active research uncertainty. Do not spend the weekend building monitoring or pruning hosts on this ticket. Reopen on measured recurrence; no host was modified in this pass.
buildrunner: unbounded container log filled the root filesystem and broke CI for ~2.5hto Research record: resolved CI disk incident; residual maintenance deferred