<cole-h>
gchristensen: After I merge the README later today (hopefully), your only job is to change the description of the repo "@GrahamcOfBorg tooling automatio" -> "@ofborg tooling automation" :D (Since IIRC LnL has w+ to infra, we can do that part without you)
<gchristensen>
w0000t!
<gchristensen>
<3 <3 <3
<LnL>
great :)
<cole-h>
"Oh yeah, it's all coming together."
<gchristensen>
also big thanks to andi- for getting the openssl 1.0.2u PR in to ofborg months ago, it made yesterday's late-night fixing of hydra much less miserable
<cole-h>
I was about to say, I can see it (and the "New Build" button) already :D
<cole-h>
"..???" lmao
<gchristensen>
that button is safe to press any time. it builds a new image, erases, and reboots the community box. it runs weekly, and typically if I run it any other time I drop a note in #nixos-aarch64 -- but it isn't required and the expectation is firmly "this box is transient"
<cole-h>
Something I just noticed on the dashboard: only core-0 and aarch64 show up on inodes + disk free graphs
<LnL>
shouldn't really need it, with the builders updated they should be reconnecting properly now
<gchristensen>
yeah
<gchristensen>
but maybe you want to push out a fix
<cole-h>
🤞
<LnL>
yeah, still useful for some specific cases but it's generally better to handle versions properly if it's not too problematic
<cole-h>
+1. I've been thinking about working on a changelog eventually. Might be nice to see some of the problems addressed between releases, even if it's only really useful to those of us intimately involved
<gchristensen>
I know someone else who runs ofborg X_X
<gchristensen>
but don't do any work with that in mind.
<cole-h>
?
<gchristensen>
I know somebody who runs a private ofborg
<LnL>
oh that actually happened?
<gchristensen>
hah, yeah
<gchristensen>
but I pretend that is not true, and I specifically *don't* want that to influence how we develop or work on ofborg
<LnL>
was referring more to the versioned messages, etc.
<gchristensen>
ah
<LnL>
btw, do we need to find another victim to run a darwin builder?
<gchristensen>
oh, on that note, I know of one which we can use
<cole-h>
Suhweet!
<gchristensen>
but have not had the time to set it up :/
<LnL>
ah ok
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-borg
<andi->
How happy are you with the git checkout performance of ofBorg? With broken.sh I am seeing it (as of now) as the primary time waste :/
<gchristensen>
it is slow :)
<andi->
My current idea is `git -C $checkout archive --format=tar $rev | tar ...` in a tempdir
<andi->
that might be faster
<andi->
and to disable the annoying auto gc
<gchristensen>
oh yeah that causes a lot of problems for my builders, eventually they get completely wedged. I've considered forcing a GC every day or something
orivej has quit [Quit: No Ping reply in 180 seconds.]
<LnL>
I've wondered if worktrees would help a bit here
<LnL>
ah it uses git clone --shared, not sure if there's a big difference then
<gchristensen>
all that is not really used anymore, since it always reuses the same direcotry I think
<LnL>
not sure there's actually much point to using git other than for the target branch
<andi->
I actually think worktrees aren't that much of an impreovement over --reference with hardlinks as it is currently used
<LnL>
creates lots of temporary commits that are only referenced/used once
<cole-h>
gchristensen: Now it's your turn! s/@GrahamcOfBorg/@ofborg/ in the description :D
<cole-h>
LnL: All 16 aarch builders came back up. Woohoo! Good work on that stuck-queue issue 🎉
<gchristensen>
done!
<gchristensen>
wooo!
<cole-h>
<3 gchristensen <3 LnL
<{^_^}>
LnL's karma got increased to 54
<{^_^}>
gchristensen's karma got increased to 300.9999999999998
<cole-h>
👀
<gchristensen>
<3 cole-h <3 LnL :)
<{^_^}>
LnL's karma got increased to 55
<{^_^}>
cole-h's karma got increased to 56
<cole-h>
OK, now back to fiddling with my NixOS setup x)
<LnL>
so, guess I'll tackle the evaluator services next?
<gchristensen>
yass
<cole-h>
Also, if it isn't too much to ask (or too dangerous), I'd like to try and get ofborg/infrastructure#20 in for the next deploy, whenever that is
<cole-h>
And here I thought I could get away with s/string/str/
<LnL>
looks like you found a nixops python3 bug
<cole-h>
Also real cool how it tells me nothing :D
<gchristensen>
jikes
<LnL>
also what is the hetzner host doing there?
<gchristensen>
I can't look right now (I'm trying, andi-!)
<andi->
Can't see that page :'(
<gchristensen>
andi-: fears around API keys :(
<gchristensen>
andi-: I'll add you though, do you have a buildkite account already?
<cole-h>
LnL: Good catch
<andi->
No I haven't really felt the need for it
<andi->
they don't do oauth, do they?
<LnL>
"resource ‘builder-0’ is obsolete"
<gchristensen>
I'd love to add you if you're willing :P
<LnL>
yeah it's been gone for like a month
<andi->
nor IPv6 :'(
* andi-
sighs and signs up
<LnL>
oh no all of them
<cole-h>
I guess that PR was dangerous x)
<cole-h>
Gonna try dry-activate once more... maybe (hopefully) just random chance?
<cole-h>
nope
<cole-h>
Wait, different error lol
<cole-h>
"KeyError: 'PACKET_ACCESS_KEY'"
<LnL>
other builds have the same obsolete messages so yeah guess it's packet/api related
<gchristensen>
hmmm
<cole-h>
Which other builds?
<LnL>
gchristensen: make that an intervention yesterday and today I guess :)
<cole-h>
x(
<gchristensen>
hah right
<LnL>
cole-h: you previous deploy
<gchristensen>
adisbladis, LnL, samueldr, andi- (you haven't accepted yet, so I don't know if you did yet or not) please configure 2fa on your buildkite accounts
<LnL>
your*
<cole-h>
LnL: Ah
<cole-h>
One more try, see if I can get a third different error, and then I go get breakfast (at 11am lol)
<cole-h>
Nope, same error this time -- PACKET_ACCESS_KEY. OK, brb like 20 minutes probably
<gchristensen>
hrm
<LnL>
sounds like vault isn't getting the secret anymore or something
orivej has joined #nixos-borg
<gchristensen>
it isn't using a --pure shell is it :X
<cole-h>
lol
<gchristensen>
I was hitting a problem with that for some time yesterday
<cole-h>
I don't think so considering it worked not half an hour ago...
<cole-h>
s/half//
<gchristensen>
heh
<gchristensen>
will look at logs shortly
<cole-h>
`rg -- --pure` doesn't turn up anything
<LnL>
yeah #!nix-shell -i bash ../shell.nix
<LnL>
unless that runs in another shell
<gchristensen>
guh, I need to figure out how to make `journalctl -f -u` tab-completion not take literal minutes
<cole-h>
andi-: Looks like we wouldn't have even made it a day without gchristensen if we had tried that month idea this month... :D
<gchristensen>
:(
<gchristensen>
expanding "🔑 Authenticating with Vault" I see
<{^_^}>
packethost/packet-python#45 (by grahamc, 1 year ago, open): When the API is super down, the API client behaves poorly
<cole-h>
"super down" nice
<cole-h>
We'll see if you actually weren't needed after this API issue is resolved... That 'str' error from the first run still seems strange (though may have been due to the API weirdness since the mentions `ssh_key_id` in the trace)
<cole-h>
Oh
<gchristensen>
it is because their API just returns like "503 Backend Fetch Failed" or something as a bytes thing
<cole-h>
Wait I didn't actually read that issue you linked lol
<gchristensen>
like, it has to be incredibly broken
<cole-h>
Derp
hmpffff has quit [Quit: nchrrrr…]
<gchristensen>
oh, b'Forbidden\n'
<cole-h>
OK, so 99% chance it'll be fine once Packet recovers
<andi->
super down and incredibly broken. Exciting times :)
<LnL>
:D
<gchristensen>
lol
* cole-h
installs an auto-refresh addon just for status.packet.com
<LnL>
wait the status api requires oauth!?
<gchristensen>
it is probably intended for writing :(
<cole-h>
Yeah, I tried GETing it and also got rejected for no auth token :D
<cole-h>
Do I have to make a new build for that to work, or will a retry see this new commit?
<cole-h>
New build it seems.
<cole-h>
Yay :D
<cole-h>
I don't see anything obviously broken, so I think it's safe to celebrate: no more red messages (aside from nixops's color coded fanciness) in deploys!! 🎉 :D
<{^_^}>
<-- just reminded me of this: [2018-05-26 20:24:58] <gchristensen> 25 points gets you a sticker, 100 points gets you a t-shirt, 1000 verified points gets you a free trip to nixcon *restrictions apply, must be verifiable points, given by grateful people, in channels I'm in
<cole-h>
~40 more points and that tee is mine >:)
<gchristensen>
ohno
<LnL>
hehe
<cole-h>
LnL: You better step up your game -- I might get that shirt before you do ;^)
<gchristensen>
LnL gets a shirt as soon as they arrive regardless :)
<gchristensen>
I owe LnL so many beers
<cole-h>
Darn. I better step my game up, so we get the shirt at roughly the same time, then :P
<LnL>
gchristensen: right back at you :)
<andi->
<3 cole-h thanks for all your work on ofborg (been reading all those notification mails about it ;))
<{^_^}>
cole-h's karma got increased to 57
<cole-h>
Thanks for indirectly giving me the title to that README PR :P
<cole-h>
(Before I submitted it I had to grep my logs for the actual phrase)
<gchristensen>
+1
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-borg
* LnL
is slightly confused
<LnL>
ofborg doesn't use the prometheus library?
<gchristensen>
heh
<gchristensen>
blow it all up
<gchristensen>
the metrics collection is full of lies
<LnL>
lol, don't follow
* andi-
notes down "our CI is lying"
<cole-h>
Wait wait you can't do that, you signed an NDA by joining this channel
<cole-h>
:P
<gchristensen>
the metrics collection is really bad and half-done