<cole-h>
gchristensen: It looks like a few things died ~hr ago. eval filter, comment filter, comment poster, and log collector all have backtraces
<gchristensen>
thanks cole-h
<cole-h>
:)
hmpffff_ has joined #nixos-borg
hmpffff has quit [Ping timeout: 260 seconds]
LnL has quit [Ping timeout: 260 seconds]
LnL has joined #nixos-borg
<cole-h>
gchristensen: comment filter seems dead again, even though it has good logs after a backtrace... Just posted something and nothing showed up in logs
<LnL>
morning
<cole-h>
I guess I never realized you were EU, LnL
<cole-h>
I'm just about to go to sleep :P
<LnL>
yeah
<cole-h>
Well, good night. See you in a handful of hours :)
cole-h has quit [Quit: Goodbye]
orivej_ has quit [Ping timeout: 260 seconds]
hmpffff has joined #nixos-borg
hmpffff_ has quit [Ping timeout: 240 seconds]
hmpffff_ has joined #nixos-borg
hmpffff has quit [Ping timeout: 260 seconds]
orivej has joined #nixos-borg
cole-h has joined #nixos-borg
<cole-h>
I'm actually an idiot lol
<cole-h>
"comment filter seems dead again" -> because I posted `@ofborg build` and not `@ofborg build list of attrs`
<cole-h>
Derp
<LnL>
FYI either my connection was _extremely_ unstable yesterday or something fishy is going on
<cole-h>
re: all the stalled darwin alerts?
<LnL>
yeah check the graph
<cole-h>
I don't need to, I'll just go through my notification history hehe
<LnL>
gchristensen: you happen to know the semantics of amqp heartbeats?
<LnL>
one thing I noticed that looks slightly suspicious is that a build which publishes a lot of logs seems to receive inconsistent heartbeats
<LnL>
so I'm wondering if it's possible for the channel to get congested so the heartbeat can't get through in time
<cole-h>
Look at the log message collector logs
<cole-h>
"WARN:amqp::session: Error dispatching packet to channel 1: Full! Blocking until there is space."
<cole-h>
Maybe related?
<gchristensen>
"Any traffic (e.g. protocol operations, published messages, acknowledgements) counts for a valid heartbeat. Clients may choose to send heartbeat frames regardless of whether there was any other traffic on the connection but some only do it when necessary."
<LnL>
ok so it's not a mandatory separate thign
<gchristensen>
note the log collector does not do many protocl operations
<cole-h>
OK, so probably unrelated x)
<LnL>
but it sure looked like that staging died while sending logs each time
<cole-h>
Next thing I would like to see is if it's possible to set up coloring based on the `level` field
<cole-h>
Right now, warning, info, etc all show up as gray
<LnL>
oh hmm that should be working
<cole-h>
Look at the "description is too long" messages -- level WARN, but grey
<cole-h>
(Like how I used gray in one sentence and then grey in the next? :D)
<LnL>
hmm don't find anything with that
<LnL>
{unit=~"ofborg.*service"} |~ "description too long"
<LnL>
oh you mean the nix-instantiate output?
MichaelRaskin has quit [Ping timeout: 256 seconds]
<cole-h>
Yeah, those lines
<cole-h>
(sorry I should have written it literally)
<cole-h>
"description is over 140 char"
<cole-h>
Also gchristensen (or LnL if he's given you access to the other machines): if you could optimise or gc the eval nodes, that would be great :) I'm getting red x's because "no space left on device" lol