samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs: https://logs.nix.samueldr.com/nixos-infra/
cole-h has quit [Ping timeout: 240 seconds]
<lukegb> hmm, the channel update services are marked as failed
<lukegb> https://monitoring.nixos.org/grafana/d/fBW4tL1Wz/scheduled-task-state-channels-website (update-nixos-20.09, update-nixos-20.09-small, update-nixos-unstable)
<gchristensen> hmm
<gchristensen> upgrading its instance type from t2.large to t3.xlarge has caused its mount points to move around
<lukegb> ah
<lukegb> I'm somewhat amused by bastion's used memory graph https://p.lukegb.com/raw/VirtuallyEternalLark.png
<gchristensen> haha, yeah
<gchristensen> Fixing this has been troublesome
<gchristensen> I'll get it shortly
cole-h has joined #nixos-infra
* lukegb starts federating some metrics from monitoring.nixos.org into his own prometheus instance
<gchristensen> sure :)
<gchristensen> cool, whichuns?
<lukegb> muahaha, there shall be no active="1" channel left behind
<gchristensen> lol
<lukegb> oh, current="1"
<supersandro2000> lukegb: can you share your config when you are done?
<supersandro2000> I am still looking into a good way to know when a channel is blocked and letting me know
<supersandro2000> great. Looking at that later
<lukegb> the channels seem to advance faster now :)
<gchristensen> https://channels.nix.gsc.io/ has a timeline
<lukegb> yeah, I mean, the channel update script seems to take less time
<gchristensen> oh probably :)
<gchristensen> the system's swap space was absent for a few hours
<gchristensen> oh nice lukegb
<supersandro2000> Can we use the time since channel update to send notifications when it is above a day from here? https://monitoring.nixos.org/grafana/d/LhIq8iLWk/channel-updates?orgId=1&refresh=10s
<lukegb> hmm
<lukegb> gchristensen: update-nixos-unstable.service is failing (3x so far) :(
<gchristensen> I don't understand
<gchristensen> Apr 20 19:38:40 bastion update-nixos-unstable-start[20271]: error: in linux-rt_5_11 (/nix/store/zqwjsgl3g77h53n6blh0khkz77b9jaaw-linux-5.11.4-rt11): error: [json.exception.parse_error.101] parse error at line 1, column 448843: syntax error while parsing value - invalid literal; last read: '82482736}}}}}}}}}}g'; expected end of input
<lukegb> lmao wat
<gchristensen> I deleted all the scratch cache it has, hopefully that fixes it
<lukegb> thanks 👍
<lukegb> ah, generate-programs-index
<lukegb> you might need to `nix-store --verify-path /nix/store/zqwjsgl3g77h53n6blh0khkz77b9jaaw-linux-5.11.4-rt11` if it's still broken; I wonder how it ended up like that though
<lukegb> the run starting at 19:08 failed too :(
maralorn has quit [Ping timeout: 245 seconds]
maralorn has joined #nixos-infra
<lukegb> if I ask too many questions I'll end up being asked to join the infra team, heh
<gchristensen> it is a danger
<lukegb> (thanks for putting up with me graham :P)
<gchristensen> niksnut: I wonder if an update to Nix has broken the channel updater?
<gchristensen> I haven't looked carefully, but this seems to be fallout from the update earlier
<lukegb> which jq seems to indicate is bad
<sterni> ;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~;5;13~
<sterni> oops, sorry
<supersandro2000> lukegb: I think this is like this since a long time
<supersandro2000> nix-index also fails on 10 or so files each pass
<lukegb> right, but, err, this is part of the channel upload script, so if this fails it won't update the channel
<gchristensen> maybe I didn't look at the log closely enough
<lukegb> this json file should still parse as valid json
<lukegb> oh, true, the exception does get caught
<lukegb> so it should be ignored
<supersandro2000> }}}}}}gular","size":4276,"narOf
<supersandro2000> thats not valid json
<lukegb> ah, the end of the file is repeated
<lukegb> ...iiinteresting
<supersandro2000> pos 448848
<lukegb> but yeah, the exception just gets caught, so this should just be ignored
<niksnut> gchristensen: that file (https://cache.nixos.org/zqwjsgl3g77h53n6blh0khkz77b9jaaw.ls) was generated yesterday
<niksnut> so it's not caused by the update today
<niksnut> btw the compression code recently got changed
<gchristensen> well i meant a change to the server processing the file, not the generator
<gchristensen> oh interesting
<supersandro2000> jaaw.ls also has invalid jq with }}}}}}}}}}gular
supersandro2000 has quit [Remote host closed the connection]
supersandro2000 has joined #nixos-infra
supersandro2000 is now known as Guest17333
supersandro2000 has joined #nixos-infra
Guest17333 has quit [Ping timeout: 260 seconds]