<cole-h>
(Or maybe I clicked it -- maybe it's client side?)
<LnL>
my browser just freaks out if I look at the 30d view if there are alerts
<cole-h>
Oh wait, I get it now -- alerts are the things like "stalledbuilder" stuff
<cole-h>
For a moment I thought it was whether or not we get *paged* for those :D
<LnL>
ah no, that's just visualisation
<cole-h>
LnL: If you have w+ to ofborg/infra, I can merge the logging PR and you merge the infra PR and deploy? ;^)
<cole-h>
gchristensen: ^ You OK with this?
<LnL>
don't have commit bit there
<cole-h>
RIP. Then we need to wait on benevolent dictator to get this rolling.
<cole-h>
:P
<LnL>
FYI we _can_ deploy from a branch if really necessary
<cole-h>
Oh, indeed :o
<cole-h>
And maybe that commit section would be nice in case of rolling back? But only if we need to roll-back ofborg/infra, since AFAICT nixos/ofborg is always pulled from HEAD
<LnL>
yeah, most likely not needed but just so you know
<cole-h>
As noted in that PR, I didn't touch any of the other files that mention Hetzner -- better to start small than over-extend.
<LnL>
I think that works, but I'm not entirely sure how nixops deals with this in it's statefile
<cole-h>
True.
<cole-h>
I should really familiarize myself with nixops... I think I've said that exact statement many times by now lol
<LnL>
in this case it's the none backend, which basically means rather then having nixops setup the resources there's an existing host where it can ssh to and run nixos-rebuild switch
<gchristensen>
oh oops I thought you did LnL
<LnL>
thought I did what?
<cole-h>
Have w+ to infra, maybe
<gchristensen>
okay you can now LnL
<gchristensen>
yup
<cole-h>
lol
<cole-h>
gchristensen: Anything else I need to do for infra#17 to be good?
<LnL>
Connection closed on channel 0 by 0:0 => AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'") } => CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
<cole-h>
:o
<LnL>
also if you look at the evals chart you can see the inprogress runs didn't disppear
<cole-h>
omg
<LnL>
aarch has been reduced to one sole survivor however
<cole-h>
Yeah, just noticed the same
<cole-h>
"How many deploys does it take to get to all aarch builders offline?" (in the style of the Tootsie Pop commercial)
<LnL>
:D
<cole-h>
If you think about it, the builders started decreasing when I started getting involved...
<gchristensen>
cole-h is sabotaging ups
<cole-h>
Crap, I meant to /query that to LnL...
<cole-h>
>:^)
hmpffff has joined #nixos-borg
<LnL>
btw how do y'all feel about depending on the `nix` commands for something?
<cole-h>
What did you have in mind, and why can't it be achieved using any of the other `nix-*` commands?
<LnL>
checking what attributes successfully build for the last channel and only failing the build for those
<cole-h>
Sounds cool. What's the command?
<LnL>
I think nix path-info is the main thing that doesn't have an equivalkent
<LnL>
hmm actually maybe not
<cole-h>
Something to keep in mind is that if we do this, I think nixUnstable already blocks `nix` commands unless `nix-command` is allowed -- so in a future update of nix, it might contain that change and break
<cole-h>
(Until we allow `nix-command`)
<LnL>
yeah I know
<LnL>
but on the other hand the new cli does have much better interfaces to integrate with in general
<LnL>
it's a bit of an awkward situation
<cole-h>
Agreed
<LnL>
so how about that deploy? :D
<LnL>
gchristensen: might want to kick the aarch box
<gchristensen>
rebooting :)
<LnL>
thanks
<LnL>
does that also update them or is it pinned?
<gchristensen>
pinned
<gchristensen>
working on updating them soon
<LnL>
alright, I think the recent changes will mostly resolve this
<cole-h>
gchristensen++ LnL++
<{^_^}>
LnL's karma got increased to 48
<{^_^}>
gchristensen was put on Santa's "nice" list
<cole-h>
lol
<LnL>
hehe
<gchristensen>
lol
<LnL>
hm, I figured it would take a while but is this normal?
<gchristensen>
ot booted yet?
<gchristensen>
oh you know what
<cole-h>
Doesn't look like
<gchristensen>
the image it is booting from doesn't exist :)
<cole-h>
LOL
<gchristensen>
I moved the netboot images to a new server and forgot thi sone
<cole-h>
Maybe that's why we lost some ~every deploy? lol
<gchristensen>
nah
<LnL>
nono
<LnL>
it's a single machine with a bunch of builders
<gchristensen>
this change happened ~4h ago
<cole-h>
O
<cole-h>
h
<LnL>
with each rabbitmq restart they had a chance to get stuck because of the hearbeat issue
<cole-h>
Renovating the dashboard once again I see
<LnL>
always fun to do while staring at dashboards :)
<gchristensen>
ooooooooohbaby
<cole-h>
gchristensen: Just curious, but what needs to be done to update the aarch builders? Trigger the hydra jobset?
<gchristensen>
no, I have a buildkite job that I need to run
<cole-h>
Oh, OK
<gchristensen>
it is Coming Soon.... working on finishing updating the Hydra builders, with the new buildkite builder -- then that builder will get updated
<cole-h>
No rush, was just curious. I looked in the build box repo and saw a script that curls hydra, which is what made me think in that direction.
<gchristensen>
yeah
<gchristensen>
usually, it automatically builds and deploys every Monday but it has been broken for a few weeks by this migration