<samueldr>
but it could also simply be connected to your own github event handler webhook thingy :)
<cole-h>
"offborg" :D
<samueldr>
well... in this case it would be an "off"borg
cole-h has quit [Quit: Goodbye]
cole-h has joined #nixos-borg
cole-h has quit [Client Quit]
Azitrex has joined #nixos-borg
Azitrex has left #nixos-borg [#nixos-borg]
<LnL>
samueldr: very nice!
NinjaTrappeur has joined #nixos-borg
cole-h has joined #nixos-borg
<cole-h>
LnL: I was just looking at the dashboard and saw the evals thing. I almost panicked, until I went through the history and saw that you did that on purpose lol
<cole-h>
I thought to myself "wait, that shouldn't be negative"
<LnL>
seems a bit more readable, you don't like it?
<cole-h>
In fact, it should, because of that change x)
<cole-h>
No, I do!
<cole-h>
It was just different -- I was confused :D
<LnL>
what about that?
<cole-h>
Oooooh
<cole-h>
Pretty colors
<cole-h>
I'd be tempted to remove the evaluators graph if we keep the "speedometers"
<LnL>
cole-h: FYI for tracing, I only added the last commit the rest was a fixup commit
<cole-h>
Oh, talking about the force-push?
<LnL>
yeah
<LnL>
thought I'd also include the evaluator for good measure
<cole-h>
🙏
<cole-h>
Only comment I'd make is: maybe include some context in the clippy FIXME's? Something about how the lint includes post-expansion macros or something
<cole-h>
Otherwise, still LGTM 👍
<cole-h>
And also 🎉 for todone-ing the "this belongs in the nix module"
<LnL>
was before I realised that wouldn't help with the warnings :)
<cole-h>
:P
<LnL>
so, what's the best next step now?
<LnL>
validate the stability for some time / deploys
<LnL>
take a look at the next thing to convert
<cole-h>
Do we need to do anything special to get loki or whatever to properly format the json logs?
<LnL>
or maybe eg. ship the builders to aarch first so see how that deals with the restartss
<LnL>
it needs to actually load the json and know about (some of) the fields
<LnL>
otherwise you have the current thing with unreadable json messages
<cole-h>
Yeah, figured.
<LnL>
do you know anything about loki?
<cole-h>
How to look at it :P
<cole-h>
I was just looking at the loki.yml to see where that might go
<cole-h>
But I guess it should go in promtail.yml? Since loki.yml doesn't have a scrap_configs key
<LnL>
yeah the format is pretty straightforward, main problem I think is that we need a way to scrape these separately
<LnL>
otherwise it would try to parse all logs I think
<LnL>
currently it's just all of journald
<cole-h>
I wonder if it would work to just add another job below with pipeline stages that have a selector of '{unit="ofborg-evaluator.service"}' (etc)
<gchristensen>
(it seems like a lot of good stuff is going on, reallycool!)
<LnL>
yeah, ideally there's a way to filter units so we don't have to do some weird other logging mechanism
<LnL>
you don't happen to know that right?
<gchristensen>
what's this?
<cole-h>
We want to take advantage of tracing's json output in Loki
<gchristensen>
hmm
<LnL>
how to apply specific json filtering, etc. for certain units
<cole-h>
Maybe just adding a `pipeline_stages` key to the journal job would work? I wonder if that means it would do nothing for unmatched lines, or print as-is (like it currently does)
hexa- has joined #nixos-borg
<hexa->
gchristensen: so besides the syslog part I have no experience with loki, but I aim to get it back up and running
<gchristensen>
hexa-: we've got some questions about handling log lines which have structured (json) data, and how to handle that in loki to get good stuff out of it
<LnL>
yeah, so question is can we just use a match block to handle json for specific units or would that break the other logs
<cole-h>
Someday, I'll have bought another machine that can be used as a sort of staging for ofborg, so then we could just hack on this yml until it does what we want, without affecting prod :P
<LnL>
yeah that was my plan, I'll look at trying it out on my machine
<cole-h>
Oh right, you have this stuff +- set up
<LnL>
not really, just a rabbitmq
<cole-h>
Then I guess more - than +
<cole-h>
x)
<LnL>
but I have this thing called nix to help me set it up