gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
orivej has quit [Ping timeout: 260 seconds]
andi- has quit [Ping timeout: 244 seconds]
andi- has joined #nixos-borg
cole-h has quit [Quit: Goodbye]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 264 seconds]
orivej has joined #nixos-borg
orivej_ has joined #nixos-borg
orivej has quit [Ping timeout: 256 seconds]
orivej_ has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 272 seconds]
orivej has joined #nixos-borg
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 258 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 272 seconds]
orivej_ has joined #nixos-borg
<LnL> gchristensen: did you have breakfast yet?
tilpner has quit [Quit: tilpner]
orivej_ has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<gchristensen> LnL: just sat down with it. what's up?
<LnL> what I talked about yesterday happened
<gchristensen> I don't remember what that was :x
<LnL> there's an unexpected converted_to_draft event without a repo in the eval queue, holding up all the other evaluations
<gchristensen> o
<gchristensen> I'm surprised, I thought ofborg was good about discarding events it doesn't know
<gchristensen> okay -- how can I help?
<LnL> yeah, I'm going to look into that
<gchristensen> cool
orivej has quit [Ping timeout: 264 seconds]
<LnL> but for now I think if you get 1 message from the eval inputs queue with "Ack message requeue false" the current stuff should unblock
orivej has joined #nixos-borg
<gchristensen> ok
<gchristensen> we'll need to re-queue #88551 and #88551
<gchristensen> LnL: where did you see the part about this error?
<LnL> what do you mean?
<gchristensen> nvm
<gchristensen> #87274 too
tilpner has joined #nixos-borg
<gchristensen> LnL: it migth be faster to add that variant to the enum
<LnL> multiple ones have a problem?
<gchristensen> no, requesting the queued item isn't getting it for me
<LnL> oh hmm, maybe need to stop the evaluators first so hey don't beat you to it?
<gchristensen> okay
<gchristensen> so, to be clear, I'm seeing one error and I tihnk you're seeing another error
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<gchristensen> LnL: https://monitoring.nix.ci/explore?orgId=1&left=%5B%22now-24h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bjob%3D%5C%22systemd-journal%5C%22,%20unit%3D%5C%22ofborg-evaluation-filter.service%5C%22%7D%20%22%7D,%7B%22mode%22:%22Logs%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D
<LnL> yeah
<gchristensen> so this is in the evaluation filter
<gchristensen> how did it get to the evaluator too
<LnL> oh
<LnL> not sure, but it's all the same pr https://monitoring.nix.ci/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22PromLoki%22,%7B%22expr%22:%22sum(count_over_time(%7Bunit%3D~%5C%22ofborg-.*.service%5C%22,level%3D%5C%22ERROR%5C%22%7D%5B1h%5D))%20by(pr)%22%7D,%7B%22mode%22:%22Metrics%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D
<gchristensen> oh!
<gchristensen> I found a bug LnL :)
<LnL> did I mess something up?
<gchristensen> let's see ...
<gchristensen> I'm not sure
<LnL> err never mind that link, it's too early in the consumer for that to work
<gchristensen> okay, yeah
<gchristensen> it looks like we're accidentally piping all the PR events directly to the filtered work queue :)
<LnL> oh whoops
<LnL> I noticed that they where separate but made a typo :)
<LnL> :/
<gchristensen> no worries :)
<gchristensen> can you send a fixup PR?
<LnL> having queue metrics would have helped here
<LnL> nudge nudge :)
<gchristensen> :)
<gchristensen> can you also add the converted_to_draft variant?
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<{^_^}> [ofborg] @LnL7 opened pull request #494 → fix evaluation-filter queue → https://git.io/Jf2rd
<LnL> I'll look at the rest later today
<{^_^}> [ofborg] @grahamc merged pull request #494 → fix evaluation-filter queue → https://git.io/Jf2rd
<{^_^}> [ofborg] @grahamc pushed 2 commits to released: https://git.io/Jf2rb
<gchristensen> deploying
<gchristensen> I'm not sure if we'll be able to do any work without further deploys, let's see
<gchristensen> I think all the queues are blocked
<gchristensen> oh
<gchristensen> no, this will make a difference
<gchristensen> for sure
<LnL> yeah maybe the filter is probably the one that handles unexpected stuff nicely
<gchristensen> yeah
<gchristensen> we'll see :P
<gchristensen> LnL: you sure you can't add converted_to_draft ?
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-borg
<LnL> not right now :/
<gchristensen> okay
orivej has quit [Ping timeout: 246 seconds]
orivej_ has joined #nixos-borg
orivej_ has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-borg
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #nixos-borg
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
cole-h has joined #nixos-borg
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-borg
<cole-h> <3 gchristensen <3 LnL
<{^_^}> LnL's karma got increased to 57
<{^_^}> gchristensen was put on Santa's "nice" list
<gchristensen> hehe
<cole-h> So, what was the actual fallout of-- Oh, the queue jumping up to ~113 queued. Woof :D
<gchristensen> every github event went right to the evaluators
<cole-h> That eval error graph... x(
<gchristensen> yeah :)
<gchristensen> the evaluators don't use the same message format that github sends, so 100% of those were errors :)
<cole-h> RIP.
<LnL> yeah, I made a boo-boo
<gchristensen> no worries :)
<cole-h> Yeah, jeez LnL, just be perfect 100% of the time like the rest of us >:(
<cole-h> (obvious /s, hopefully)
orivej has quit [Ping timeout: 240 seconds]
orivej_ has joined #nixos-borg
<LnL> lol, developerexcuses.com just gave me this:
<LnL> > I thought you signed off on that?
<{^_^}> error: syntax error, unexpected ')', expecting ID or OR_KW or DOLLAR_CURLY or '"', at (string):313:1
<gchristensen> lol
<cole-h> Oh no
<cole-h> Right back at me haha
<cole-h> "It works, but it's not been tested"
<cole-h> Oh cool, GH breaking the API again -- `thread 'main' panicked at 'Failed to add labels ["ofborg-internal-error"] to issue #88588: Error(Codec(Error("invalid type: map, expected a sequence", line: 1, column: 1))`
<gchristensen> what.
<cole-h> My thoughts exactly
<LnL> ok so, is that enum still relevant?
<cole-h> Which enum?
<LnL> the github event
orivej_ has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-borg
<{^_^}> [ofborg] @LnL7 opened pull request #495 → convert log-message-collector to easylapin → https://git.io/Jf2SK
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<LnL> yeah
<cole-h> Suggestions on `range`'s name? :P
<LnL> only generating the config instead of the name/value would be a bit more flexible but I don't think we need that
<cole-h> How do you mean?
<LnL> packet-spot-eval-foo = mkEvaluator "foo"
<cole-h> Oh, we can do that too.
<LnL> as for range
<LnL> > lib.range 0 3
<{^_^}> [ <CODE> <CODE> <CODE> <CODE> ]
<cole-h> Yeah, but then we have to get `lib` in there
<cole-h> I just lifted the definition from `lists.nix` anyways
<LnL> hmm, isn't this a module?
<cole-h> LnL: Wait, what's the point of the argument to `mkEvaluator`? My function only uses the argument for the name, which would already be `packet-spot-eval-foo` because.. that's what you typed
<cole-h> So it would really be `packet-spot-eval-foo = mkEvaluator;`
<LnL> ah it's not used anywhere else
<cole-h> Nope.
<cole-h> Your call: do we `listToAttrs`, or do we `packet-spot-eval-1 = mkEvaluator; packet-spot-eval-2 = mkEvaluatur; ...`
<LnL> range is fine, don't think we need to mess with individual machines
<cole-h> gchristensen: We're all caught up to the queue. Time to drop the 3 new machines?
<gchristensen> eh, let 'em go :P
<cole-h> I only ask because I was gonna PR the generalization of the `mkEvaluator` stuff
<cole-h> So I could change it to `numMachines 3` instead of `numMachines 6`
<gchristensen> keep it at 6
<cole-h> OK, roger.
<cole-h> For your viewing pleasure: https://github.com/ofborg/infrastructure/pull/25
<{^_^}> ofborg/infrastructure#25 (by cole-h, 3 minutes ago, open): nixops: generalize evaluator setup
* cole-h wonders how hard it would be to get ofborg/infrastructure activity to show up here as well...
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-borg
cole-h has quit [Client Quit]
cole-h has joined #nixos-borg
cole-h has quit [Client Quit]
cole-h has joined #nixos-borg
cole-h has quit [Client Quit]
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<LnL> rg session_from_config
<LnL> ofborg/src/easyamqp.rs:pub fn session_from_config(config: &RabbitMQConfig) -> Result<amqp::Session, amqp::AMQPError> {
<LnL> gchristensen: any reason to keep it at 6?
<gchristensen> any reason to go lower? :)
<LnL> I think it's almost 3x what we need on average
<LnL> urgh the 🦄 is back
<gchristensen> :(
<LnL> was hoping to ship these last two so I can start on the purge
<LnL> but guess not
<LnL> what are the mozilla overlays for btw, don't we test using the target rust/cargo?
<gchristensen> I don't remember :x
cole-h has joined #nixos-borg
<LnL> I addressed to prefetching so both should be good to go if either of you want to deploy later
cole-h has quit [Client Quit]
cole-h has joined #nixos-borg
cole-h has quit [Ping timeout: 260 seconds]
orivej has quit [Ping timeout: 246 seconds]
orivej has joined #nixos-borg
cole-h has joined #nixos-borg
cole-h has quit [Ping timeout: 265 seconds]
hmpffff has joined #nixos-borg
orivej has quit [Read error: Connection reset by peer]
orivej has joined #nixos-borg
hmpffff has quit [Quit: nchrrrr…]
cole-h has joined #nixos-borg
<hexa-> no evals in the last two hours?
orivej has quit [Quit: No Ping reply in 180 seconds.]
<LnL> yeah just noticed
cole-h_ has joined #nixos-borg
<LnL> guess that's the answer to "is the enum still relevant"
orivej has joined #nixos-borg
cole-h has quit [Quit: Goodbye]
<LnL> thanks for the ping :)
<gchristensen> do I need to pop a job off?
<LnL> nah I'll fix it now
cole-h_ has quit [Client Quit]
cole-h has joined #nixos-borg
<LnL> oh, just the entry is enough
<{^_^}> [ofborg] @LnL7 opened pull request #496 → handle converted_to_draft events → https://git.io/Jf2NC
orivej has quit [Ping timeout: 265 seconds]
orivej has joined #nixos-borg
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #nixos-borg
<{^_^}> [ofborg] @LnL7 merged pull request #496 → handle converted_to_draft events → https://git.io/Jf2NC
<{^_^}> [ofborg] @LnL7 pushed 2 commits to released: https://git.io/Jf2Nd
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-borg
orivej has quit [Quit: No Ping reply in 180 seconds.]
<LnL> ok, off to bed now
orivej has joined #nixos-borg
<LnL> gchristensen: if we could spend 15min sometime to go over the rabbitmq exporter setup, I think that could really help give us more visibility
<gchristensen> yeah let's do that
<cole-h> o/ LnL thanks for your hard work
<cole-h> (Sorry I've been in and out)
<gchristensen> what's missing?
cole-h has quit [Quit: Goodbye]
<LnL> for the exporter?
cole-h has joined #nixos-borg
<gchristensen> yea
<LnL> just the credentials
<gchristensen> oh I thought we had solved that already :)
<gchristensen> there are already creds
<LnL> so ?? -> nixops secret
<LnL> oh, so just put them in the unit file?
<gchristensen> yea
orivej has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-borg
<LnL> alright, I can do the rest then