Ancestors

Toot

Written by Atemu on 2024-12-14 at 02:14

So today I resumed my PC, cloned a git repository with submodules and built a new #NixOS generation at the same time.

But then all the sudden everything went slow and I couldn't even reach #systemd via systemctl anymore (O.o). After a little debugging I saw a bunch of iowait CPU util, so I guess that something in the storage stack must have gone awry. Couldn't read out dmesg because privilege escalation auth wouldn't load of course; reads weren't working.

Okay then, magic sysrq + b and the system booted fine. Nothing in the journal of the previous boot of course because nothing could be written at that time. There was a kernel BUG after resume regarding bluetooth though; not sure that had anything to do with it.

git status on the newly cloned repo was complaining at me that the submodule clones in their respective .git/modules/ dir weren't git repos. Hm, okay, must have been interrupted at an inopportune time i.g. Okay, deleted those and inited the submodules again which worked fine.

Then I tried to build the NixOS generation again but the fact that I had stashed a change in my nixos-config repo before the IO hang has reverted. Uh-huh. Stashed it again and built.

Eval then aborted with something to the tune of "/nix/store/gqx3ikrm5ff27k5yjykdc1kvmsnn61wa-mesa-demos-9.0.0.drv not found". Oh-oh.

So yes, Nix committed in its DB that some drvs were written to the Nix store but the dnode's state must have reverted and the files were, in fact, non-existent. As it turns out, #Nix doesn't like that very much.

I ran a GC which deletes all unreferenced drv files but no dice somewhat surprisingly. Perhaps it looks at existing files rather than the ValidPaths in the Nix store db? Ugh.

So after a bit of experimentation, I landed on this:

sqlite3 /nix/var/nix/db/db.sqlite "SELECT path from ValidPaths where path like '%.drv';" | xargs -P 32 -I {} sh -c '[ -e "{}" ] || nix store delete -
-derivation "{}^*"'

Some of the missing drv files depended on another and would need to be deleted in the order given by their DAG. Perhaps you could reverse the ValidPaths? Running the above command a few times also works sufficiently quickly if it's not too many and it was like a few dozen drv files total for me.

So yeah, fun.

=> More informations about this toot | More toots from Atemu@darmstadt.social

Descendants

Written by Atemu on 2024-12-14 at 02:15

What's also weird is that I set #btrfs' flushoncommit mount option which supposedly should fix cases like this one. Hm.

=> More informations about this toot | More toots from Atemu@darmstadt.social

Written by Arian on 2024-12-14 at 09:13

@Atemu did nix-store --repair not help?

=> More informations about this toot | More toots from arianvp@functional.cafe

Written by Atemu on 2024-12-14 at 10:32

@arianvp

I tried nix store repair IIRC.

=> More informations about this toot | More toots from Atemu@darmstadt.social