samueldr changed the topic of #nixos-infra to: NixOS infrastructure | logs:
aminechikhaoui has quit [Ping timeout: 240 seconds]
aminechikhaoui has joined #nixos-infra
aminechikhaoui has quit [Quit: Ping timeout (120 seconds)]
aminechikhaoui has joined #nixos-infra
<gchristensen> I started working on a job to load the hydra database snapshot and do a pgdump on it, thinking that by verifying the pg_dump output is about 30G+, we'll have a fairly well tested backup
<gchristensen> anyway, after a few minor tweaks today, it is working: it is dumping with "pg_dump hydra --create --format=directory --exclude-table users --verbose -U hydra" -- excluding the user table with the idea of making the dumps publicn somehow
<gchristensen> here is the repo:
<gchristensen> I'm thinking people could add batch queries they'd like run, too, and have those results uploaded
<gchristensen> any opinions?
<gchristensen> it'd be nice for it to be on infra managed by the foundation and not my home system, but there are a few of requirements we can't easily meet at the moment: an always-on system (precludes Packet spot market) with a significant amount of disk space for receiving the snapshots. I'd also like it if it continued using buildkite or something similar where the logs were public and easily viewable,
<gchristensen> but I wish that were true about a bunch of the scheduled tasks we have :)
cole-h has joined #nixos-infra
qyliss has joined #nixos-infra
MichaelRaskin has joined #nixos-infra
LnL has joined #nixos-infra
<niksnut> what do you mean with 'add batch queries they'd like run'?
<gchristensen> like LnL would like to have a query similar to this run on a semi-regular basis, and be able to fetch the results: COPY (SELECT * FROM builds as b JOIN jobsetevalmembers as m ON = JOIN jobsetevals as e ON = m.eval WHERE e.project = 'nixpkgs' AND e.jobset = 'trunk' AND > 1515735) TO '/tmp/builds.csv';
<LnL> that's mostly just a one time thing
<gchristensen> ah
<MichaelRaskin> Periodic would be «date_part('epoch',now())-e.timestamp < 3.3e7» at least…
<LnL> the actual meta queries should probably run against all the data
<gchristensen> we don't need to do that part :P (that said, if there were queries people'd want to run, it would be a lot easier for them to be able to submit the query and fetch results than have ~300G of disk space to download and import the db dump. but again, don't need to do this part)
<LnL> yeah so for package metadata these are a bunch of relatively small queries, but for each package in nixpkgs which adds up
tilpner has quit [Remote host closed the connection]
tilpner_ has joined #nixos-infra
tilpner_ is now known as tilpner