samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs: https://logs.nix.samueldr.com/nixos-infra/
MichaelRaskin has quit [Ping timeout: 240 seconds]
Guest97996 is now known as JJJollyjim
JJJollyjim is now known as Guest13908
Guest13908 has quit [Quit: authenticating]
Guest139081 has joined #nixos-infra
cole-h has quit [Ping timeout: 268 seconds]
MichaelRaskin has joined #nixos-infra
<lukegb> https://hydra.nixos.org/build/141975028 hrm, "Hydra failure" isn't very specific
<hexa-> "below" also is difficult to follow, given that the page is too short to jump anywhere
<hexa-> fwiw, staging-next was merged
<hexa-> (there is no nix-error anchor)
<lukegb> gchristensen: hmm, how large is hydra's DB
<lukegb> I was contemplating doing some analysis on builds to work out which ones are good candidates for being marked big-parallel
<gchristensen> neat
<gchristensen> you need a few hundred gb to hold the db
<gchristensen> the .sql is 30G
<lukegb> hm, that's alright
<MichaelRaskin> Hmm. Year-cut tab-separated values could be nice for analytics experiments…
<lukegb> I was tempted to just dump the thing into BigQuery for the moment but that's because, well, I'm familiar with it
<lukegb> but TSVs are more generally useful
<MichaelRaskin> Yeah, one could have a few years at a time in RAM and do whatever analytics desired
<gchristensen> lukegb: if you come up with an export query and PR it to github.com/grahamc/hydra-pg-load-dump I'll run it for you
<lukegb> Ooh, that's neat
<gchristensen> the only rule is don't get the users table :)
<MichaelRaskin> Hmm, are psql options allowed?
<gchristensen> it is an arbitrary script
<gchristensen> MichaelRaskin: what are you wondering?
<gchristensen> / thinking
<MichaelRaskin> Nah, I don't want to play gotcha in this setting, just thinking about copy.sql as a TSV export
<lukegb> ah, shellcheck says no
<gchristensen> ruh roh, heh
<gchristensen> d'you mind PRing a fixup on that? a bit unfair to ask you to fix something you can't easily check in automatic CI, but ... :)
<lukegb> yeah, will do
cole-h has joined #nixos-infra
<gchristensen> lukegb: looks pretty good, some useless trivia is that when assigning variables you never need quotes. of course, this is not an objection to adding quotes :D
<lukegb> TIL :p
<gchristensen> erm, okay, an issue I'll need to address elsewhere :D
<lukegb> ah, hah
<lukegb> gchristensen: ooh thanks
* lukegb keeps an eye on it
<gchristensen> note the data is a bit stale. this does the dump and load on a secondary zpool, and the automation that copies it from my archival zpool to the working zpool was stalled since 2021-04-17T13:10:00Z
<gchristensen> but we'll get you this data, then after the working pool catches up we can get you a fresher copy
<lukegb> Sounds good! Thanks :)
* gchristensen grumbles grumpily about zrepl not copying the dataset over, ignoring the part where I didn't ask it to
<hexa-> so, about that tested job failure, how would we get more details on that?
<lukegb> hexa-: for the moment I think just waiting for the eval's jobs to finish and then trying to see if we can replicate it locally
<gchristensen> I wonder if postgres's dump program could output more info ... https://buildkite.com/grahamc/postgres-load-dump/builds/51#679e1a7a-d395-4c4e-8f50-538b52ad4585/24-422
<MichaelRaskin> I guess there is an option to just run a loop printing the output file sizes in the background…
<gchristensen> maybe the output could be told to go to stdout, then pv
<gchristensen> actually, is that going to create a tsv? :)
<MichaelRaskin> I thought pg_dump is _slightly_ not TSV?
<gchristensen> pg_dump makes all sorts of outputs
<MichaelRaskin> Well, sure, if you ask it insistently enough
<gchristensen> no, really, like
<gchristensen> https://linux.die.net/man/1/pg_dump we're passing -Fc
<gchristensen> c custom Output a custom archive suitable for input into pg_restore. This is the most flexible format in that it allows reordering of loading data as well as object definitions. This format is also compressed by default.
<gchristensen> maybe this script should do the full dump as a separate buildkite task
<lukegb> it depends how often you plan on running it and how much spare I/O you have, tbh
<gchristensen> I'd like to run it weekly to say hey your backups aren't valid enough to restore
<lukegb> I'm thinking about metrics I'd like and trying to work out if they exist already
<gchristensen> cool
<lukegb> among other things: Hydra builder occupancy (i.e. how many jobs could a node have vs. how many are scheduled on it at the moment)
<gchristensen> that one is knowable now
<gchristensen> importantly, though, the hydra scheduler does not prioritize filling machines
<lukegb> oh?
<gchristensen> afaik, it has an ordered list of jobs it wants to run, and distributes them as possible
<lukegb> hmm, I see
<gchristensen> this does confuse people :) and one of the things I'd like to revisit
thefloweringash has quit [Ping timeout: 245 seconds]
nh2[m] has quit [Ping timeout: 245 seconds]
roberth has quit [Ping timeout: 245 seconds]
garbas[m] has quit [Ping timeout: 245 seconds]
thefloweringash has joined #nixos-infra
nh2[m] has joined #nixos-infra
roberth has joined #nixos-infra
garbas[m] has joined #nixos-infra
<gchristensen> lukegb: ping see dm
<lukegb> ty
supersandro2000 has quit [Killed (verne.freenode.net (Nickname regained by services))]
supersandro2000 has joined #nixos-infra