Skip to content

RSDK-14449 - syscfg: report ext4 filesystem errors - #300

Merged
Cheuk (cheukt) merged 3 commits into
RSDK-11671-default-error-log-forwardingfrom
RSDK-14449-storage-health
Oct 6, 2026
Merged

Cheuk (cheukt) merged 3 commits into
RSDK-11671-default-error-log-forwardingfrom
RSDK-14449-storage-health

Conversation

@cheukt

@cheukt Cheuk (cheukt) commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Stacked on #299. Every 15 minutes, read /sys/fs/ext4/*/errors_count and log an error under viam-agent.storage when a filesystem's count rises. SD cards have no SMART data, and sysfs still works when journald is stuck.

Of the ticket's other items, journald watchdog failures are covered by #299, and fsck -n is skipped: it gives false positives on a mounted filesystem and adds I/O to a card that's already failing.

Tested on a Pi with a loopback ext4 image and trigger_fs_error.

Screen Shot 2026-09-30 at 17 30 22 PM

🤖 Generated with Claude Code

@cheukt Cheuk (cheukt) added the dev-release Build and publish agent binaries with a dev version label Sep 29, 2026
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

@cheukt
Cheuk (cheukt) added this pull request to stack #301 September 30, 2026 21:29
@cheukt

Cheuk (cheukt) commented Sep 30, 2026 •

Copy link
Copy Markdown
Member Author

this might be a bit noisy so once it's merged I'll update a few viam-dev robots to the latest build and monitor before releasing

func checkExt4Errors(logger logging.Logger, reported map[string]int) {
all, err := readExt4Errors(ext4SysfsDir)
if err != nil {
logger.Debugw("reading ext4 error counters", "error", err)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only debug level here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switched to warn

"go.viam.com/test"
)

func writeExt4Fs(t *testing.T, dir, dev, count, first, last string) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of me is tempted to mount a loop device and inject errors so this test can run against real /sys/fs files but it's probably not worth it.

@viam-overwatch

viam-overwatch Bot commented Oct 5, 2026

Copy link
Copy Markdown

Hey Cheuk (@cheukt) — this PR has been approved and CI has been green for 4+ business days. Ready to merge?

Auto-comment from overwatch. Will not re-nudge for 7 days.

@cheukt
Cheuk (cheukt) force-pushed the RSDK-14449-storage-health branch from b77d00c to 3efb379 Compare October 5, 2026 20:21
Cheuk (cheukt) and others added 2 commits October 5, 2026 16:28
Poll /sys/fs/ext4/*/errors_count every 15 minutes and log an error when a
filesystem's count rises. SD cards expose no health data, so recorded
filesystem corruption is the best early signal of card failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cheukt
Cheuk (cheukt) force-pushed the RSDK-14449-storage-health branch from 3efb379 to c160f53 Compare October 5, 2026 20:28
@cheukt
Cheuk (cheukt) merged commit c1a1e8f into main Oct 6, 2026
11 of 12 checks passed
@cheukt
Cheuk (cheukt) deleted the RSDK-14449-storage-health branch October 6, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-release Build and publish agent binaries with a dev version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants