gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
<cole-h> omg so many coredumps
<gchristensen> ??
<cole-h> In the unit selection for Loki
<cole-h> Like actually several hundred
<gchristensen> what host?
<gchristensen> annoying
<cole-h> packet-spot-eval-3
<gchristensen> lol
<cole-h> We got python3.7, opt, cipher-api-test
<cole-h> Ouch
<cole-h> Assuming the evaluator+builder we just lost is packet-spot-eval-3?
<gchristensen> should be the hetzner one
<cole-h> Ah, is this the day of reckoning?
<gchristensen> yea
<gchristensen> rip hetzner
<cole-h> o7
cust0dian has quit [Quit: WeeChat 2.8]
<samueldr> psst, if someone wants to steal this to run offborg off of it as a testing mode thing https://github.com/samueldr/github-to-irc/blob/released/shell.nix#L15-L44
<samueldr> I also have that to push fake events in https://github.com/samueldr/github-to-irc/blob/released/bin/dev-faker
<samueldr> but it could also simply be connected to your own github event handler webhook thingy :)
<cole-h> "offborg" :D
<samueldr> well... in this case it would be an "off"borg
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-borg
cole-h has quit [Client Quit]
Azitrex has joined #nixos-borg
Azitrex has left #nixos-borg [#nixos-borg]
<LnL> samueldr: very nice!
NinjaTrappeur has joined #nixos-borg
cole-h has joined #nixos-borg
<cole-h> LnL: I was just looking at the dashboard and saw the evals thing. I almost panicked, until I went through the history and saw that you did that on purpose lol
<LnL> heh lol
<cole-h> I thought to myself "wait, that shouldn't be negative"
<LnL> seems a bit more readable, you don't like it?
<cole-h> In fact, it should, because of that change x)
<cole-h> No, I do!
<cole-h> It was just different -- I was confused :D
<LnL> what about that?
<cole-h> Oooooh
<cole-h> Pretty colors
<cole-h> I'd be tempted to remove the evaluators graph if we keep the "speedometers"
<LnL> cole-h: FYI for tracing, I only added the last commit the rest was a fixup commit
<cole-h> Oh, talking about the force-push?
<LnL> yeah
<LnL> thought I'd also include the evaluator for good measure
<cole-h> 🙏
<cole-h> Only comment I'd make is: maybe include some context in the clippy FIXME's? Something about how the lint includes post-expansion macros or something
<cole-h> Otherwise, still LGTM 👍
<cole-h> And also 🎉 for todone-ing the "this belongs in the nix module"
<LnL> was before I realised that wouldn't help with the warnings :)
<cole-h> :P
<LnL> so, what's the best next step now?
<LnL> validate the stability for some time / deploys
<LnL> take a look at the next thing to convert
<cole-h> Do we need to do anything special to get loki or whatever to properly format the json logs?
<LnL> or maybe eg. ship the builders to aarch first so see how that deals with the restartss
<LnL> it needs to actually load the json and know about (some of) the fields
<LnL> otherwise you have the current thing with unreadable json messages
<cole-h> Yeah, figured.
<LnL> do you know anything about loki?
<cole-h> How to look at it :P
<cole-h> I was just looking at the loki.yml to see where that might go
<cole-h> But I guess it should go in promtail.yml? Since loki.yml doesn't have a scrap_configs key
<LnL> yeah the format is pretty straightforward, main problem I think is that we need a way to scrape these separately
<LnL> otherwise it would try to parse all logs I think
<LnL> currently it's just all of journald
<cole-h> I wonder if it would work to just add another job below with pipeline stages that have a selector of '{unit="ofborg-evaluator.service"}' (etc)
<gchristensen> (it seems like a lot of good stuff is going on, reallycool!)
<LnL> yeah, ideally there's a way to filter units so we don't have to do some weird other logging mechanism
<LnL> you don't happen to know that right?
<gchristensen> what's this?
<cole-h> We want to take advantage of tracing's json output in Loki
<gchristensen> hmm
<LnL> how to apply specific json filtering, etc. for certain units
<cole-h> Maybe just adding a `pipeline_stages` key to the journal job would work? I wonder if that means it would do nothing for unmatched lines, or print as-is (like it currently does)
hexa- has joined #nixos-borg
<hexa-> gchristensen: so besides the syslog part I have no experience with loki, but I aim to get it back up and running
<gchristensen> hexa-: we've got some questions about handling log lines which have structured (json) data, and how to handle that in loki to get good stuff out of it
<hexa-> hah x(
<hexa-> leads to a deadlink <3
<LnL> yeah, so question is can we just use a match block to handle json for specific units or would that break the other logs
<cole-h> Someday, I'll have bought another machine that can be used as a sort of staging for ofborg, so then we could just hack on this yml until it does what we want, without affecting prod :P
<LnL> yeah that was my plan, I'll look at trying it out on my machine
<cole-h> Oh right, you have this stuff +- set up
<LnL> not really, just a rabbitmq
<cole-h> Then I guess more - than +
<cole-h> x)
<LnL> but I have this thing called nix to help me set it up
<cole-h> Whaaaat
<cole-h> What is this Nix thing you speak of
<cole-h> Looks like a decent example (the one in the docs wasn't doing much for me)
<hexa-> indeed the docs are a bit lacking