gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg
hmpffff has quit [Quit: nchrrrr…]
* cole-h wishes Loki showed more than 20 rows of context
<gchristensen> heh
<cole-h> 3 new internal-errors and I can't see why
<cole-h> "new"
<gchristensen> ouch
<cole-h> #85347 #85398 #85399
<{^_^}> https://github.com/NixOS/nixpkgs/pull/85347 (by zowoq, 22 hours ago, open): megatools: 1.10.2 -> 1.10.3
<{^_^}> https://github.com/NixOS/nixpkgs/pull/85398 (by seqizz, 5 hours ago, open): g933-utils: init at 0.13
<{^_^}> https://github.com/NixOS/nixpkgs/pull/85399 (by cgevans, 5 hours ago, open): tensorflow-bin_2: fix tensorboard dependency
* cole-h also wishes he could get more than 1k lines of logging
<cole-h> Grrr, these darn "<no info> (0x133742069)" backtraces are starting to annoy me >:(
<gchristensen> yeah those suck a lot
<cole-h> I still don't understand why it has no info
<cole-h> Heh, I can't ping @ofborg. Looks like your deprecation of grahamcofborg will have to wait
<gchristensen> you can it just won't auto-complete the name
<cole-h> Oh really
<gchristensen> yeah
<cole-h> Oh so it does
<cole-h> Very neat
<cole-h> Thanks once again GitHub
<gchristensen> heh
<gchristensen> yae
<gchristensen> lol
<cole-h> Btw those 3 PRs are green and can have that scary purple label removed now ;)
<gchristensen> oh youcan't do it?
<cole-h> I'm w-
<cole-h> Would if I could :P
<gchristensen> cool
<gchristensen> done thanks
<cole-h> Thank *you*
<gchristensen> so actually no need for an envelope status because the check suite covers it https://developer.github.com/v3/checks/suites/
* cole-h wonders if (and hopes) this is already supported in hubcaps...
<gchristensen> ..maybe? but maybe not, this is part of why I've had to keep my own branch
<cole-h> Doesn't look like it. Or at least I don't see the canary Accept header when grepping the code
<gchristensen> hm
<gchristensen> but it supports check runs?
<cole-h> "Sorry, forked repositories are not currently searchable." Why, GitHub, why
<gchristensen> <github> 🖕
<cole-h> Oh wait, it does lmao, I was just grepping for the entire string instead of something shorter
<gchristensen> ah
<cole-h> `MediaType::Preview("antiope")` shows up
<cole-h> Heh
<gchristensen> cool
<cole-h> Great, I don't have to learn GitHub's API (completely) just yet
orivej has joined #nixos-borg
andi- has quit [*.net *.split]
{^_^} has quit [*.net *.split]
{^_^} has joined #nixos-borg
andi- has joined #nixos-borg
sphalerite has joined #nixos-borg
cole-h has quit [Quit: Goodbye]
hmpffff has joined #nixos-borg
hmpffff has quit [Quit: nchrrrr…]
hmpffff has joined #nixos-borg
<{^_^}> [ofborg] @doronbehar opened pull request #462 → config.public.json: add @doronbehar → https://git.io/JfJqd
cole-h has joined #nixos-borg
<cole-h> gchristensen: Purple label that can be removed: #85429
<{^_^}> https://github.com/NixOS/nixpkgs/pull/85429 (by zaninime, 8 hours ago, open): [19.09] nexus: 3.18.1-01 -> 3.22.0-02 (backport)
<cole-h> (and maybe tagged security since it addresses CVEs)
<cole-h> (while you're there)
<cole-h> Just made a regex that searches the logs for added "ofborg-internal-error" labels (+) and disregards lines that already have it (=), for my fellow log gremlins
<cole-h> {unit="ofborg-evaluator.service"} |~ "\\+ \\[[^\\]]*\"ofborg-internal-error\"[^\\]]*\\]"
<gchristensen> nice
<gchristensen> super nice
<cole-h> gchristensen: While I have your attention, could I point you to https://www.youtube.com/watch?v=GdgX46KwKqo ? It describes how to add a PromQL-queryable Loki data source so I can add alerts for the above regex 👀
<gchristensen> oh so yeah I watched that, but I already did it -- that is why loki logs are searchable in the first place
<gchristensen> I'm not sure what is missing for the next bit. I'll make you an admin, so you can debu
<gchristensen> you're an admin
<cole-h> Thanks, I'll take a look
<gchristensen> thanks
<cole-h> Ah, it seems the missing link is that you added the actual Loki data source, not a Prometheus data source /pointing/ to Loki
<gchristensen> oh cool
<cole-h> Added Prometheus data source -> URL set to `<loki URL>/loki` -> all good
<cole-h> Now to figure out the alerts part
<gchristensen> nice :D
<cole-h> What "type" of channel should the alerts go to? Telegram, Discord, Pushover, email are a few of the options (the rest I don't think I have access to or have never heard of)
<cole-h> https://monitoring.nix.ci/alerting/notification/1/edit has a dropdown with all the options
<LnL> gchristensen: if you have time could you give me a quick rundown of the rabbitmq setup?
<LnL> mainly thinking about ack/requeue type stuff
<gchristensen> sure
<gchristensen> for things I definitely want to make sure are completed, I make sure auto-ack = false, and then wait until the job is completed successfully before sending the ack
<gchristensen> if it panicks or sends a nack, it'll retry pretty much immediately
<LnL> where is that configured?
<gchristensen> ah in the bin
<gchristensen> look at like bin/builder.rs
<LnL> oh, that's all in the client?
<gchristensen> yeah
<gchristensen> the clients define all the queues / bindings / exchanges
<LnL> ah so there's nothing to configure on the rabbitmq side except for users
<gchristensen> yep :)
<LnL> and what about the library, I see heartbeats are not supported compared to eg. lapin
<gchristensen> badly, lol.
<gchristensen> LnL: want to integrate lapin?
<gchristensen> I'm ready to adopt our async overlords I guess
<LnL> well I'd like to rather decouple stuff a bit more instead of just hacking this in
<gchristensen> sounds good
<LnL> so if you where thinking of going that direction, might be a good way to try it out?
<gchristensen> I think ofborg would benefit from async in a lot of ways
<gchristensen> and the currenty hacked-in heartbeat support is terrible
<gchristensen> in general, ofborg's code is not great, and moving in a fresh direction would be good
<LnL> oh btw did we figure out what was wrong with the NIX_PATH thing?
<gchristensen> (on the phone)
orivej has quit [Ping timeout: 265 seconds]
WilliButz has quit [Remote host closed the connection]
WilliButz has joined #nixos-borg