gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
orivej has quit [Ping timeout: 258 seconds]
<cole-h> The other day when we were talking about GitHub Actions and the 3000 minutes stuff... that only applies to *private* repos. "GitHub Actions usage is free for public repositories." from https://help.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions
cole-h has quit [Quit: Goodbye]
orivej has joined #nixos-borg
<LnL> oh! no wonder this spews you so many log messages https://github.com/NixOS/ofborg/pull/42/files
<LnL> tought something was horribly wrong
cole-h has joined #nixos-borg
<LnL> kind of itching to deploy the logging changes
<cole-h> Oh hey, looks like you got the promtail stuff working!
<cole-h> Looks niiiice. LnL++
<{^_^}> LnL's karma got increased to 47
<LnL> yeah, pretty much what I was expecting
<LnL> just some fiddling with the details to get the right labels out
<cole-h> You're a braver person than I. yml gives me the heebie jeebies
<cole-h> LnL: Ooooh I didn't even notice the "build jobs by arch" makeover
<cole-h> LnL: Also, was that alerts toggle intentional? To avoid warning about the single darwin builder, I'd assume.
<LnL> there was nothing waiting before
<LnL> alert toggle?
<cole-h> It's usually on
<LnL> ah! no
<cole-h> (Or maybe I clicked it -- maybe it's client side?)
<LnL> my browser just freaks out if I look at the 30d view if there are alerts
<cole-h> Oh wait, I get it now -- alerts are the things like "stalledbuilder" stuff
<cole-h> For a moment I thought it was whether or not we get *paged* for those :D
<LnL> ah no, that's just visualisation
<cole-h> LnL: If you have w+ to ofborg/infra, I can merge the logging PR and you merge the infra PR and deploy? ;^)
<cole-h> gchristensen: ^ You OK with this?
<LnL> don't have commit bit there
<cole-h> RIP. Then we need to wait on benevolent dictator to get this rolling.
<cole-h> :P
<LnL> FYI we _can_ deploy from a branch if really necessary
<cole-h> Oh, indeed :o
<cole-h> And maybe that commit section would be nice in case of rolling back? But only if we need to roll-back ofborg/infra, since AFAICT nixos/ofborg is always pulled from HEAD
<LnL> yeah, most likely not needed but just so you know
<cole-h> Thanks for enlightening me.
<{^_^}> [ofborg] @LnL7 merged pull request #479 → remove static lifetimes from easylapin → https://git.io/JfOdx
<{^_^}> [ofborg] @LnL7 pushed 2 commits to released: https://git.io/Jfs8H
<LnL> oh hmm right, deploy isn't happing about the missing host
<cole-h> The conservative fix for that would be to just delete the hetzner host from nixops/network.nix -> ofborg/infrastructure#17
<{^_^}> https://github.com/ofborg/infrastructure/pull/17 (by cole-h, 21 seconds ago, open): Drop Hetzner host
<cole-h> As noted in that PR, I didn't touch any of the other files that mention Hetzner -- better to start small than over-extend.
<LnL> I think that works, but I'm not entirely sure how nixops deals with this in it's statefile
<cole-h> True.
<cole-h> I should really familiarize myself with nixops... I think I've said that exact statement many times by now lol
<LnL> in this case it's the none backend, which basically means rather then having nixops setup the resources there's an existing host where it can ssh to and run nixos-rebuild switch
<gchristensen> oh oops I thought you did LnL
<LnL> thought I did what?
<cole-h> Have w+ to infra, maybe
<gchristensen> okay you can now LnL
<gchristensen> yup
<cole-h> lol
<cole-h> gchristensen: Anything else I need to do for infra#17 to be good?
<cole-h> Err ofborg/infrastructure#17
<{^_^}> https://github.com/ofborg/infrastructure/pull/17 (by cole-h, 48 minutes ago, open): Drop Hetzner host
<LnL> ah, no big deal
<LnL> wouldn't mind you take a quick look at it anyway :)
<gchristensen> lgtm :)
<cole-h> gchristensen: if you r+ #480, I'll merge :^)
<{^_^}> https://github.com/NixOS/ofborg/pull/480 (by LnL7, 2 days ago, open): tracing logging
<LnL> oh wow look at this new logging
<LnL> Connection closed on channel 0 by 0:0 => AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'") } => CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
<cole-h> :o
<LnL> also if you look at the evals chart you can see the inprogress runs didn't disppear
<cole-h> omg
<LnL> aarch has been reduced to one sole survivor however
<cole-h> Yeah, just noticed the same
<cole-h> "How many deploys does it take to get to all aarch builders offline?" (in the style of the Tootsie Pop commercial)
<LnL> :D
<cole-h> If you think about it, the builders started decreasing when I started getting involved...
<gchristensen> cole-h is sabotaging ups
<cole-h> Crap, I meant to /query that to LnL...
<cole-h> >:^)
hmpffff has joined #nixos-borg
<LnL> btw how do y'all feel about depending on the `nix` commands for something?
<cole-h> What did you have in mind, and why can't it be achieved using any of the other `nix-*` commands?
<LnL> checking what attributes successfully build for the last channel and only failing the build for those
<cole-h> Sounds cool. What's the command?
<LnL> I think nix path-info is the main thing that doesn't have an equivalkent
<LnL> hmm actually maybe not
<cole-h> Something to keep in mind is that if we do this, I think nixUnstable already blocks `nix` commands unless `nix-command` is allowed -- so in a future update of nix, it might contain that change and break
<cole-h> (Until we allow `nix-command`)
<LnL> yeah I know
<LnL> but on the other hand the new cli does have much better interfaces to integrate with in general
<LnL> it's a bit of an awkward situation
<cole-h> Agreed
<LnL> so how about that deploy? :D
<LnL> gchristensen: might want to kick the aarch box
<gchristensen> rebooting :)
<LnL> thanks
<LnL> does that also update them or is it pinned?
<gchristensen> pinned
<gchristensen> working on updating them soon
<LnL> alright, I think the recent changes will mostly resolve this
<cole-h> gchristensen++ LnL++
<{^_^}> LnL's karma got increased to 48
<{^_^}> gchristensen was put on Santa's "nice" list
<cole-h> lol
<LnL> hehe
<gchristensen> lol
<LnL> hm, I figured it would take a while but is this normal?
<gchristensen> ot booted yet?
<gchristensen> oh you know what
<cole-h> Doesn't look like
<gchristensen> the image it is booting from doesn't exist :)
<cole-h> LOL
<gchristensen> I moved the netboot images to a new server and forgot thi sone
<cole-h> Maybe that's why we lost some ~every deploy? lol
<gchristensen> nah
<LnL> nono
<LnL> it's a single machine with a bunch of builders
<gchristensen> this change happened ~4h ago
<cole-h> O
<cole-h> h
<LnL> with each rabbitmq restart they had a chance to get stuck because of the hearbeat issue
<LnL> at least that's what I suspect
<cole-h> Oh, I see
<LnL> ^ that was yesterday
<gchristensen> okay it is erasing / now
<gchristensen> (almost booted)
<cole-h> wew
<gchristensen> this server was the first time I experimented with erasing / on boot
<gchristensen> should be good to go now LnL
<cole-h> Oh yep
<cole-h> 16 builders :D
<cole-h> Nice timing, considering there are 16 waiting build jobs for aarch
<gchristensen> tea
<gchristensen> neat
<LnL> cole-h: refresh
<cole-h> Ohhh fancy
<cole-h> Renovating the dashboard once again I see
<LnL> always fun to do while staring at dashboards :)
<gchristensen> ooooooooohbaby
<cole-h> gchristensen: Just curious, but what needs to be done to update the aarch builders? Trigger the hydra jobset?
<gchristensen> no, I have a buildkite job that I need to run
<cole-h> Oh, OK
<gchristensen> it is Coming Soon.... working on finishing updating the Hydra builders, with the new buildkite builder -- then that builder will get updated
<cole-h> No rush, was just curious. I looked in the build box repo and saw a script that curls hydra, which is what made me think in that direction.
<gchristensen> yeah
<gchristensen> usually, it automatically builds and deploys every Monday but it has been broken for a few weeks by this migration
<LnL> oh you can make annotations in grafana?
<cole-h> Yeppers
<LnL> neat