gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
orivej has quit [Ping timeout: 240 seconds]
cole-h has quit [Quit: Goodbye]
<{^_^}> [ofborg] @ThomasMader opened pull request #475 → config.public.json: Add @ThomasMader to Trusted Users → https://git.io/JfmxG
tilpner has quit [Quit: tilpner]
orivej has joined #nixos-borg
<MichaelRaskin> Are Darwin builders down?
<LnL> yes, intentionally
<LnL> trunk queue is still 15k
cole-h has joined #nixos-borg
<{^_^}> [ofborg] @cole-h opened pull request #476 → config.public.json: add cole-h to trusted_users → https://git.io/JfY4q
<cole-h> ^ Hope this is OK :) I added myself as maintainer of fish yesterday, but don't have access to a darwin system to catch regressions
<gchristensen> cool
<LnL> heh, assumed you where in there already
<cole-h> I only started hanging out with the cool kids within the past ~3 months :P
<LnL> btw, I made a thing to (ab)use github actions for interactive debuggin on darwin
<cole-h> Oh?
<gchristensen> oooh
<cole-h> Good thing I got that free student pro so I have some minutes to spare
<cole-h> Ah, is the tmate the magic part of it?
<LnL> yep :)
<cole-h> LnL++ Cheers. I'll set up a repo that lets me do that within the century :P
<{^_^}> LnL's karma got increased to 40
<LnL> still need to mess with pf but I could also setup an old macbook
<cole-h> What's pf?
<LnL> a bsd packet filter, like iptables
<MichaelRaskin> I wonder if it would be a good idea not to schedule Darwin builds on push
<LnL> howso?
<MichaelRaskin> What is the ETA of the builds to continue?
<LnL> depends on hydra
evanjs has quit [Quit: ZNC 1.7.5 - https://znc.in]
evanjs has joined #nixos-borg
<LnL> there's no autoscaling for darwin so 40k builds take a while
<LnL> linux ran 200 concurrent builds for a about a day
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-borg
<MichaelRaskin> But then I think it makes little sense to auto-schedule builds that will take a few days to be started.
<MichaelRaskin> Right now I pushed, ofborg auto-scheduled a Darwin build, but nothing will happen with that auto-scheduled build for the forseeable future, then later it creates some load when everyone has stopped caring
<LnL> ah yeah, we might want to flush the queue before I start it again
<cole-h> I meant to ask about that yesterday: do we have a way to flush the queue?
<LnL> don't have access to the management interface but there's a purge button there
<gchristensen> if we start it, won't it blow through the queue quickly?
<LnL> yeah it might be ok
<LnL> none of those jobs are filtered yet so a bunch are cantbuild and will be skippeed
<gchristensen> yea
<MichaelRaskin> I think just unrequested creation of queued jobs is a bit confusing in the current situation
<MichaelRaskin> But maybe there is no cheap way to suspend it
<LnL> I think disconnecting the exchange might work for that
<LnL> so I have a patch for carnix and/or a one line change I could do to the generated Cargo.nix
<LnL> what should I do?
<gchristensen> *preferrably* patch Carnix..... but I've been known to check in patches to the Cargo.nix
<gchristensen> (so that it could be regenerated, and then have patches applied automatically)
<cole-h> +1.
<cole-h> A little sad that Carnix's upstream seems to be stalled at the moment
<gchristensen> LnL: could you send me the patch for carnix? maybe I can get pierre to apply it
<LnL> sure, I already opened one before https://nest.pijul.com/pmeunier/carnix/discussions/46
<LnL> still looking into a test for this one
<gchristensen> oh
<gchristensen> then let me know what I should do and ping him about :) and I will
<LnL> aww the ui doesn't work for my second branch :/
<LnL> do you know what the correct way to push a patch do a discussion is?
<cole-h> Anybody know why ofborg sometimes misses PRs? I can look at logs in ~an hour, but the past day or two I've had to kick off evals manually on a handful of PRs.
<cole-h> Example PR that I haven't kicked off yet: #86336
<{^_^}> https://github.com/NixOS/nixpkgs/pull/86336 (by geistesk, 1 hour ago, open): st: 0.8.2 -> 0.8.3
<cole-h> Oh, maybe it's related to credentials again
<cole-h> INFO:ofborg::tasks::evaluate: E: Error(Fault { code: Unauthorized, error: ClientError { message: "Bad credentials", errors: None } }
* cole-h sighs
<LnL> I think nest died again
<LnL> gchristensen: so assuming I can add these #46 and #47
<{^_^}> https://github.com/NixOS/nixpkgs/pull/46 (by mornfall, 7 years ago, closed): build-support/release changes
<{^_^}> https://github.com/NixOS/nixpkgs/pull/47 (by shlevy, 7 years ago, closed): x.org: Update to 7.7
<LnL> {^_^}: not those!
<gchristensen> lol
<gchristensen> I'll reach out
<gchristensen> LnL: since I can't load them, could you give me a very short thing to say about them in the ping?
<gchristensen> I'm going to say like "hey Pierre, my friend Daiderd wrote up a couple patches to carnix to fix some issues we've been having when using it on ofborg. for example «a few words». we had some trouble with the nest, but could you take a look?"
<LnL> first fixes cargo workspaces, second operator precedence of cfg expressions
<cole-h> gchristensen: How often do borg's logs get rotated? There were 2 internal errors ~11 hours ago, but nothing in the logs (didn't even get paged using the Discord thing I hooked up)
<LnL> ^ in case I don't manage to add them before he looks at it
<gchristensen> thanks
<LnL> hrm, how do I get a regular patch out of this...
<LnL> aaaaah
<cole-h> Are you powering up?
<LnL> sorry don't know what happened there
<cole-h> You can't fool me, Saiyan.
<LnL> what a rabbithole
<gchristensen> +1
<cole-h> Carnix, or amqp? If amqp, nice pun.
<LnL> rofl
<cole-h> :D
<LnL> kind of both so I'll take it
infinisil has quit [Quit: Configuring ZNC, sorry for the joins/quits!]
infinisil has joined #nixos-borg
* LnL adds a cargo fmt commit hook
infinisil has quit [Quit: Configuring ZNC, sorry for the joins/quits!]
infinisil has joined #nixos-borg
infinisil has quit [Client Quit]
<LnL> hmm no that doesn't really work
infinisil has joined #nixos-borg
<infinisil> Let's see if this works now: #46 #47
<{^_^}> https://github.com/NixOS/ofborg/issues/46 (by grahamc, 2 years ago, closed): A PR with absolutely no body causes errors in hubcaps
<{^_^}> https://github.com/NixOS/ofborg/issues/47 (by grahamc, 2 years ago, closed): GitHub Comment Coalescing
<infinisil> Nice :)
<gchristensen> nice!
<infinisil> LnL: ^
<cole-h> (btw that 46 and 47 was for Carnix)
<infinisil> Ahh haha
<cole-h> ;)
<infinisil> Well I won't add support for pijul for now at least
<cole-h> Hehe
<gchristensen> okay but if you do {^_^} is in #pijul :)
<infinisil> Ohh
<infinisil> Hm does it have an API?
<gchristensen> not sure, and I wouldn't worry so much
<gchristensen> pijul is undergoing ... significant ... changes right now
<LnL> hmm, beginning to thing that me tying to add that patch is what's bringing nest down
<gchristensen> hah
<LnL> or is it just down for me?
<infinisil> Down for me too
<cole-h> Maybe it can only handle like 3 people ;)
<LnL> could just be the ui, but I don't understand how to push a patch
<LnL> btw I think #474 is mostly good now
<{^_^}> https://github.com/NixOS/ofborg/pull/474 (by LnL7, 1 day ago, open): [WIP] lapin amqp implementation
<LnL> should I just try and run my builder with it for a bit?
<gchristensen> sure!
<gchristensen> sounds great
* cole-h would be really happy if `crate::easyamqp::*` -> `crate::easyamqp::{SomeStruct, AnotherOne, BitesTheDust};`
<infinisil> Hm, how does ofborg evaluate the performance of a PR? Is the outpaths.nix thing used for that?
<infinisil> From the code I think that's the case, which would mean only the performance of the packages is evaluated, nothing for NixOS
<cole-h> LnL: Is that FIXME regarding the consumer trait to be done at a later date?
<infinisil> (which means I might have to look into how to make a good performance benchmark for NixOS itself)
<gchristensen> LnL gogogo!
<LnL> looks like hydra has also built ghc already so it might even succeed
<LnL> cole-h: I'm not sure how to solve that one
<cole-h> OK, sounds good. Just wanted to make sure it wasn't forgotten.
<LnL> or if it should be solved, the wrapper struct is just a bit weird
<LnL> cabal-install on x86_64-darwin — Success
<LnL> does something at least
<gchristensen> nice
<gchristensen> decide between you on the unglobbing and then let's merge & deploy :)
<LnL> yeah I'll do that, just didn't want to deal with it while working
<cole-h> Thanks :) Sorry for the nits
<gchristensen> it is nice having the two of you working on it :)
<cole-h> "working"
<cole-h> If you call what I do working, I'm afraid to ask about what was happening before... :P
<gchristensen> less :)
<gchristensen> LnL: I'm inclined to deploy that PR tonight if you're up for it
<cole-h> gchristensen: Just occurred to me, but one of the most important things to know (after how to deploy) would be: how to roll back if an issue occurs?
<cole-h> Just go back to the previous deploy and restart it or something?
<cole-h> "Rebuild" the button says
<gchristensen> the deploy ALWAYS deploys the current "released" branch
<gchristensen> so if you need to roll back, do a revert PR, merge, and deploy
* cole-h cannae merge
<LnL> yeah, there's no nixops rollback right?
<gchristensen> there is actually
<gchristensen> might be good to look in to that :)
<LnL> no job for it that is
<gchristensen> right
<LnL> yeah I think you do need to enabble history for it, but perhaps there could be a rollback job or something that does that
<LnL> imports done, don't worry about the nitpicks by the way :)
<LnL> as for deploy that's up to you two, it's getting late here so I'm going to bed soon
<gchristensen> cool
<gchristensen> let's do it :)
<gchristensen> as soon as CI passes
<gchristensen> thank you LnL!
<gchristensen> really awesome :D
<LnL> let's hope it actually helps with the heartbeats
<cole-h> <3 LnL
<{^_^}> LnL's karma got increased to 41
<gchristensen> yah
<gchristensen> once this is out, it'll be good to move, say, log collector over
<gchristensen> log collector is actually pretty high bandwidth
<gchristensen> or maybe a lower bandwidth one
<gchristensen> like comment filter
<gchristensen> lots of options
<cole-h> After looking at WOP's NixOS org team stuff, I noticed there's an ofborg team... Should I "Request to join"? 👀
<LnL> I think all the groundwork is done now so moving the other things should be easy
<gchristensen> <3 LnL
<{^_^}> LnL's karma got increased to 42
<LnL> heh there we go, perfect number
<cole-h> :D Now you just have to ++ yourself whenever somebody ++'s you again
<gchristensen> cole-h: sure, you can request
<cole-h> Dang, borg has been going since 2017
<gchristensen> wow :)
<gchristensen> cole-h: want to merge #474?
<{^_^}> https://github.com/NixOS/ofborg/pull/474 (by LnL7, 1 day ago, open): lapin amqp implementation
<LnL> feels like yesterday gchristensen was talking about this crazy idea to make ci sane for nixpkgs
<gchristensen> no kidding
<cole-h> gchristensen: I'll trade you, 474 for 476 :P
<gchristensen> cole-h: okay merge #476 too :)
<{^_^}> https://github.com/NixOS/ofborg/pull/476 (by cole-h, 6 hours ago, open): config.public.json: add cole-h to trusted_users
<cole-h> Do we create a merge commit, squash and merge, or rebase and merge?
* gchristensen shrugs
<cole-h> OK, I'll squash and merge because I like doing that :)
<LnL> I try to keep my git history relatively nice so doesn't really matter
<cole-h> True, it does look really nice... actually, cancel that. I'll straight merge
<{^_^}> [ofborg] @cole-h merged pull request #474 → lapin amqp implementation → https://git.io/Jfmi3
<{^_^}> [ofborg] @cole-h pushed 15 commits to released: https://git.io/JfYXb
<cole-h> :D
<gchristensen> w00t
<{^_^}> [ofborg] @cole-h merged pull request #476 → config.public.json: add cole-h to trusted_users → https://git.io/JfY4q
<{^_^}> [ofborg] @cole-h pushed 2 commits to released: https://git.io/JfYXp
<gchristensen> nice
<cole-h> I kinda want to say "Thanks!" to myself hehe
* cole-h does it
<gchristensen> cole-h: fwiw, don't merge darwin access without consulting LnL and/or me
<cole-h> Absolutely
<cole-h> I won't merge anything without consulting either/or, tbh 👀
<cole-h> (Unless it's an urgent revert or something)
<gchristensen> I'm watching for your deploy whenever you're ready :)
<cole-h> Sir, yes, sir!
<gchristensen> let me know before you do the "actually do it" button
<cole-h> OK. Just did the dry activation button.
* cole-h gets some water.
<gchristensen> LnL: do you see this? cole-h doesn't even know the pain of starting a chef deploy / puppet apply and having to sit around for however long it takes :P
<cole-h> :D
<cole-h> Do you usually get dinner/lunch for those times? :P
<gchristensen> oh no, you're chained to your computer watching everything
<cole-h> God, scrolling past those warnings scared the bejeezus out of me
<gchristensen> because even "dry run" isn't necessarily even safe, and then the not-dry-run takes crazy long
<cole-h> I saw red text and almost flipped out
<LnL> neither do I, just horrible ansible stuff but apparently that's already an improvement?
<gchristensen> same problem though :P
<gchristensen> deploying takes $forever, and is scary
<cole-h> gchristensen: The real reason I wanted to be on the team: I can resolve my own comments! 🥳
<gchristensen> never would I ever do a "dry run" and walk away whil eit ran
<gchristensen> lol
<cole-h> (I'm really mad that GH doesn't let you resolve your OWN freaking comments......)
<cole-h> OK, ready to Just Do It™?
<LnL> yeah, at my previous our integration tests would deploy a vm first which took about 40min
<gchristensen> go for it
<cole-h> Missiles away!
<cole-h> btw ofborg is now a Nix repo on GH :D
<LnL> 🚢🚢🚢
<cole-h> lol
<gchristensen> nice
<cole-h> ofborg RiiN when?
<cole-h> :P
<gchristensen> lol
<gchristensen> okay someone want to trigger a build?
<LnL> hm interesting, what are those 3 other builders?
<gchristensen> not sure
<cole-h> Still need a build, or want to figure that out first?
<LnL> also mine disappeared?
* cole-h opts to hold off.
<gchristensen> go for it cole-h, can only give us more info
<cole-h> Wait a second, this whole "want to merge it?", "want to deploy it?" was just a ploy to put the sole blame on me! :P
<cole-h> Building nixpkgs#82982
<{^_^}> https://github.com/NixOS/nixpkgs/pull/82982 (by cole-h, 5 weeks ago, open): improve robustness of various Rust packages
<gchristensen> LnL: one might be from ikwildrpepper
<gchristensen> but I'm surprised, where are all the aarch64 boxe's
<LnL> it's all arch I think
<gchristensen> ahh
<LnL> each deploy has been dropping a few
<gchristensen> fascinating, heh
<gchristensen> cole-h: now that we've tricked you in to deploying, LnL and I are going on vacation.
<cole-h> 😱
* cole-h /nick LnLchristensen
<LnL> builder 56291 ofborg 5u IPv4 0x2f4eb290010070b9 0t0 TCP 10.0.2.15:49442->core-0.ewr1.nix.ci:5671 (CLOSE_WAIT)
<gchristensen> is that ... erm, bad
<gchristensen> (ruhroh)
<LnL> mine is still building something tho, could be that it will blow up once it tries to do the next thing
<cole-h> So much for going to bed soon, LnL
<gchristensen> we could roll back if you'd rather, LnL, I didn't mean to commit you to staying up
<LnL> let's see what happens after that build first
<gchristensen> okay
<cole-h> Build succeeds on x86_64-linux (on my earlier-linked PR)
<gchristensen> huh I just got a stalledbuilder alert, interesting indeed
<LnL> the aarch ones was an existing problem and where not updated so that's a different thing
<gchristensen> dinner time, back in a bit. do what seems right, y'all
<cole-h> aarch64 build succeeds
<LnL> hmm the darwin queue also disappeared
<LnL> thought I also saw that last time
<LnL> killing the build didn't really help https://gist.github.com/LnL7/cb0d6c3c4e9e987c11cb2c515ecd52ed
<cole-h> Hm, I see that too...
<cole-h> Darwin's back up, it looks like
<cole-h> Somehow it's doing 2 build jobs?
<LnL> interesting
<cole-h> Maybe it's counting nixpkgs-fmt and nixpkgs-fmt.passthru.tests as two separate builds?
<cole-h> nvm. It's at 1 build now and nixpkgs-fmt is still going.
<LnL> that's the real one
<cole-h> Once we get slog going, the builder (or at least arch) should be one of the things logged
<LnL> vdirsyncer is the one that died
<cole-h> How'd you determine that? Access to the machine?
<LnL> yeah, that's the build I killed
<cole-h> Oh, got it. Did it just take that log to update on grafana, or?
<cole-h> We're now at 0 since nixpkgs-fmt succeeded
<LnL> no, I suspect vdirsyncer was sill tied to the previous connection
<cole-h> I'm still curious where that single aarch64 builder went...
<LnL> have a feeling might be similar to what we just was
<LnL> saw*
<cole-h> I do remember that big fall, but I figured it was planned since I didn't hear any hubbub...
<cole-h> Whose machines are they?
<LnL> they run on the aarch community box IIRC
<LnL> anyway, nap time
<cole-h> OK, sleep well. Doesn't seem like any imminent threat..,
<gchristensen> thanks LnL, cole-h :)
<cole-h> Could that giant decrease be related to grahamc/nix-community-aarch64-build-box being red on buildkite?
<LnL> yeah, seems to be mostly working ok if I see something tomorrow morning before you guys are up I can revert if needed
<gchristensen> thanks!
<gchristensen> now go to bed :x
<cole-h> Sounds good. Thanks for your work on that, no matter what happens :P
<gchristensen> +1
<gchristensen> <3 LnL
<{^_^}> LnL's karma got increased to 43
<cole-h> Crap, LnL quick, ++ yourself so you can be back at that nice number!
<gchristensen> lol
<cole-h> gchristensen: Could the drop in arm builders be related to grahamc/nix-community-aarch64-build-box being red on buildkite?
<gchristensen> nah
<gchristensen> that rebuilds an image and erases the server
<gchristensen> that machine didn't get touched at all in the ofborg deply
<cole-h> Oh, OK
<cole-h> Still wondering about where that single arm builder went after the deploy...
<cole-h> Aliens?