gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
<cole-h> I just renamed my ofborg origin to be "upstream" and set `pushurl = nope` just to be super-duper-extra cautious
<cole-h> Also thanks for asking me to trigger a build... my PR finally got merged because of that bump :P
<gchristensen> cool
orivej has quit [Ping timeout: 240 seconds]
<gchristensen> hrm
<gchristensen> seems likely that the new stuff isn't reserving one job at a time
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 260 seconds]
cole-h has quit [Quit: Goodbye]
<LnL> ok yeah, I didn't think about this
<LnL> I'm guessing the consumer stream isn't waiting until the ack since a few futures are involved now
<LnL> hmm no, everything is syncronous
thefloweringash has quit [Quit: killed]
dtz has quit [Quit: killed]
jtojnar has quit [Quit: killed]
jtojnar has joined #nixos-borg
orivej has joined #nixos-borg
dtz has joined #nixos-borg
thefloweringash has joined #nixos-borg
tilpner has joined #nixos-borg
NinjaTrappeur has quit [Quit: WeeChat 2.7]
NinjaTrappeur has joined #nixos-borg
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-borg
cole-h has joined #nixos-borg
<LnL> gchristensen: could you check something in rabbitmq?
<gchristensen> I think so
<gchristensen> what's up?
<LnL> actually might be hard to check while there's nothing in the queue
<gchristensen> LnL: I don't see any connections from you?
<gchristensen> LnL: is your IP 95....?
<LnL> yeah, ok so there should be a build in the darwin queue soon
<LnL> if you get a message from build-inputs-x86_64-darwin, what's the delivery_mode?
<gchristensen> mmmm looking
<gchristensen> your IP is 95.xxx right?
<LnL> hmm no, don't think so why?
<gchristensen> what is your IP?
<LnL> 78.
<gchristensen> hrm
<gchristensen> I don't see you connected
<LnL> yeah, I sopped the builder so you get a chance to look at one of the build jobs :)
<gchristensen> aaah okay one sec
<gchristensen> hmmm no queued darwin jobs
<LnL> yeah, eval is still pending
<gchristensen> ah okay
* gchristensen isn't at the top of his game right now
<gchristensen> ok 1 queued
<cole-h> I think you'd be hard-pressed to find anybody who is ;)
<gchristensen> okay so getting the message via the admin interface doesn't show me delivery mode
<LnL> ^ so that's empty right?
<gchristensen> looks empty ... yeah
<LnL> ok I think that might partially explain the queue issue
<LnL> I delivery
<gchristensen> hrm
<LnL> I don't know the semantics of delivery_mode vs durable queue etc.
<gchristensen> fwiw the queue is durable
<gchristensen> I don't know about delivery-mode either ...
<gchristensen> oh itisn't written to disk though
<LnL> but based on my testing messages not sent with delivery mode 2 (ie persistent) disappear when rabbitmq restarts
<gchristensen> hmm yes that sounds right
<gchristensen> probably need to send messages as persistent
<gchristensen> oh!
<LnL> now question why the hell is rabbitmq restarting during deploy
<LnL> what's that?
<gchristensen> "in memory 2"
<LnL> ah
<gchristensen> LnL: mind connecting your builder?
<LnL> should be coming back up
<gchristensen> I'm not seeing it up
<LnL> vm freaked out a bit
<gchristensen> okay
<gchristensen> one problem LnL is your builder is accepting all the builds
<gchristensen> it has an undefined prefetch count
<gchristensen> but it should be set to 1
<LnL> yeah, looking into that
<gchristensen> cool
<cole-h> If I'm understanding all this genius-speak: setting prefetch count to 1 should make the builder only do 1 at a time again?
<LnL> yeah currently it accepts all jobs in the background and then works on them one by one
<gchristensen> yeah
<gchristensen> if you're seeing 1,000 msg/sec it is more efficient to do it that way
<cole-h> So technically waiting for 1x1, but doesn't advertise to the queue that it's waiting (and therefore accepts all jobs)
<cole-h> 1x1 -> builds to come in one by one
<LnL> about the queue, how come logs don't disappear? or do they also didn't check
<LnL> yeah, they're still there
<cole-h> Been seeing a lot of PRs go unnoticed by borg due to "bad credentials" errors recently. Is there a way to maybe make token generation more aggressive/frequent?
<LnL> or requeue with exponential backoff
<LnL> what happens now?
<cole-h> Logged error and silence o n the actual PR
orivej has quit [Ping timeout: 246 seconds]
<{^_^}> [ofborg] @LnL7 opened pull request #477 → limit prefetch count to one for lapin channels → https://git.io/JfOZ9
<{^_^}> [ofborg] @LnL7 opened pull request #478 → make messages persistent → https://git.io/JfOnX
<LnL> ^ that should work, but I'd like to know why the logs don't seem to need it
<cole-h> LnL: Looks like your fmt hook didn't work :P
<cole-h> Pedantry check is angery
<LnL> yeah no can't really do that
<LnL> would mean you can't commit anything with upstaged changes
<LnL> I have <Leader>f but don't really like having my editor automatically mess with stuff on write
<srk> nnoremap <Leader>q :wqa<CR>
<srk> can recommend nnoremap <Leader><Leader> :wa<CR>
<srk> :)
<srk> and mapped to spacebar :D
<LnL> double space is C-^ for me
<LnL> and q :cclose<CR> :lclose<CR> :pclose<CR>
<srk> hmm, what is C-^?
<LnL> last buffer
<{^_^}> [ofborg] @grahamc merged pull request #477 → limit prefetch count to one for lapin channels → https://git.io/JfOZ9
<{^_^}> [ofborg] @grahamc pushed 2 commits to released: https://git.io/JfOcO
<LnL> gchristensen: before we deploy today I'd like to have something to debug the deploy weirdness with
<gchristensen> ok
<LnL> like not deploy one of the machines so that builder doesn't restart
<gchristensen> I am *extremely* annoyed with an unrelated problem and need to step away from the computer for a bit, sorry
<LnL> doesn't have to be now, but I suspect we'll need to look at the aarch box pretty soon
orivej has joined #nixos-borg
<{^_^}> [ofborg] @grahamc merged pull request #478 → make messages persistent → https://git.io/JfOnX
<{^_^}> [ofborg] @grahamc pushed 2 commits to released: https://git.io/JfOW7
<cole-h> I really wonder why Travis gets curl errors ~frequently
<LnL> unstable connections