#nixos-borg on 2020-06-22

2018-04-19 20:36 gchristensen changed the topic of #nixos-borg to: https://www.patreon.com/ofborg https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1&from=now-1h&to=now "I get to skip reviewing the PHP code and just wait until it is rewritten in something sane, like POSIX shell. || https://logs.nix.samueldr.com/nixos-borg

00:08 orivej has quit [Ping timeout: 264 seconds]

00:32 tilpner_ has joined #nixos-borg

00:35 tilpner has quit [Ping timeout: 240 seconds]

00:35 tilpner_ is now known as tilpner

01:49 andi- has quit [Ping timeout: 272 seconds]

01:55 andi- has joined #nixos-borg

03:08 LnL has quit [Ping timeout: 260 seconds]

03:10 LnL has joined #nixos-borg

04:54 orivej has joined #nixos-borg

05:02 orivej has quit [Ping timeout: 240 seconds]

05:03 orivej has joined #nixos-borg

05:04 cole-h has quit [Quit: Goodbye]

05:25 orivej has quit [Ping timeout: 246 seconds]

05:25 orivej has joined #nixos-borg

05:47 orivej has quit [Ping timeout: 240 seconds]

05:48 orivej has joined #nixos-borg

06:15 orivej has quit [Ping timeout: 240 seconds]

06:16 orivej has joined #nixos-borg

06:31 orivej has quit [Quit: No Ping reply in 180 seconds.]

06:32 orivej has joined #nixos-borg

06:43 orivej has quit [Ping timeout: 258 seconds]

08:48 orivej has joined #nixos-borg

08:59 orivej_ has joined #nixos-borg

09:01 orivej has quit [Ping timeout: 256 seconds]

10:16 orivej_ has quit [Read error: Connection reset by peer]

10:17 orivej has joined #nixos-borg

10:27 orivej has quit [Ping timeout: 240 seconds]

10:45 kiwiirc has joined #nixos-borg

10:47 <kiwiirc> what's nixos-borg?

10:48 <hexa-> the channel related to the ofBorg service

10:49 <hexa-> see https://github.com/NixOS/ofborg

10:51 <kiwiirc> ah nice

11:32 orivej has joined #nixos-borg

11:58 orivej_ has joined #nixos-borg

12:01 orivej has quit [Ping timeout: 256 seconds]

12:07 orivej has joined #nixos-borg

12:08 orivej_ has quit [Ping timeout: 240 seconds]

12:17 orivej has quit [Quit: No Ping reply in 180 seconds.]

12:19 orivej has joined #nixos-borg

12:45 orivej has quit [Ping timeout: 264 seconds]

12:45 orivej has joined #nixos-borg

12:53 orivej has quit [Quit: No Ping reply in 180 seconds.]

12:54 orivej has joined #nixos-borg

13:07 orivej has quit [Quit: No Ping reply in 180 seconds.]

13:09 orivej has joined #nixos-borg

13:17 orivej_ has joined #nixos-borg

13:17 orivej has quit [Ping timeout: 264 seconds]

13:28 orivej_ has quit [Quit: No Ping reply in 180 seconds.]

13:30 orivej has joined #nixos-borg

13:47 adisbladi is now known as adisbladis

13:54 orivej has quit [Ping timeout: 256 seconds]

13:54 orivej has joined #nixos-borg

14:14 orivej has quit [Quit: No Ping reply in 180 seconds.]

14:15 orivej has joined #nixos-borg

14:25 orivej has quit [Quit: No Ping reply in 180 seconds.]

14:26 orivej has joined #nixos-borg

14:35 orivej has quit [Quit: No Ping reply in 180 seconds.]

14:38 orivej has joined #nixos-borg

14:47 orivej_ has joined #nixos-borg

14:49 orivej has quit [Ping timeout: 240 seconds]

15:09 orivej_ has quit [Ping timeout: 260 seconds]

15:09 orivej has joined #nixos-borg

15:21 orivej has quit [Ping timeout: 256 seconds]

15:21 orivej has joined #nixos-borg

15:45 orivej has quit [Ping timeout: 246 seconds]

15:46 orivej has joined #nixos-borg

16:02 orivej has quit [Quit: No Ping reply in 180 seconds.]

16:03 orivej has joined #nixos-borg

16:14 orivej has quit [Quit: No Ping reply in 180 seconds.]

16:14 orivej has joined #nixos-borg

16:37 orivej has quit [Read error: Connection reset by peer]

16:37 orivej_ has joined #nixos-borg

16:46 cole-h has joined #nixos-borg

16:54 orivej_ has quit [Ping timeout: 265 seconds]

16:54 orivej has joined #nixos-borg

17:01 <cole-h> gchristensen: Is the fact that we only have 3 "managed

17:01 <cole-h> " evaluators the reason why eval-4 is reporting those "no space left" errors?

17:01 <gchristensen> ouch

17:01 <gchristensen> probablyso

17:02 <cole-h> e.g. would it be resolved by just redeploying with 6 again? I don't know how the provisioning works (would it give us back those old machines? would it create 3 more so we have a grand total of 9?)

17:03 <cole-h> (alternatively, somehow do a manual decommission of those 3 extras)

17:04 <gchristensen> we have 3 good ones,right?

17:05 <cole-h> Well, really only -4 seems to be misbehaving

17:05 <cole-h> But yes, 1,2,3 are fine

17:05 <cole-h> (And tracked by grafana and friends)

17:07 <gchristensen> cool

17:13 orivej has quit [Ping timeout: 260 seconds]

17:14 orivej_ has joined #nixos-borg

17:24 orivej_ has quit [Ping timeout: 260 seconds]

17:25 orivej has joined #nixos-borg

17:33 orivej has quit [Ping timeout: 265 seconds]

17:33 orivej has joined #nixos-borg

17:48 orivej_ has joined #nixos-borg

17:50 orivej has quit [Ping timeout: 256 seconds]

17:57 orivej_ has quit [Ping timeout: 256 seconds]

17:58 orivej has joined #nixos-borg

18:15 orivej has quit [Quit: No Ping reply in 180 seconds.]

18:16 orivej has joined #nixos-borg

18:26 orivej has quit [Quit: No Ping reply in 180 seconds.]

18:26 orivej has joined #nixos-borg

18:47 orivej has quit [Ping timeout: 260 seconds]

18:47 orivej has joined #nixos-borg

19:03 orivej has quit [Quit: No Ping reply in 180 seconds.]

19:05 orivej has joined #nixos-borg

19:25 orivej has quit [Quit: No Ping reply in 180 seconds.]

19:26 orivej has joined #nixos-borg

19:46 orivej has quit [Ping timeout: 256 seconds]

19:46 orivej has joined #nixos-borg

20:31 evanjs has quit [Read error: Connection reset by peer]

20:31 evanjs has joined #nixos-borg

20:33 orivej has quit [Ping timeout: 265 seconds]

20:33 orivej has joined #nixos-borg

20:41 <hexa-> creating file '/nix/store/603m6dp953vlnl7m3vk0y5lwr1jvqakj-packet-spot-eval-5/pkgs/desktops/gnome-2/platform/libbonobo/default.nix': No space left on device

20:42 <hexa-> that seem unrelated to my python packages :)

20:42 <hexa-> https://github.com/NixOS/nixpkgs/pull/91317

20:42 <{^_^}> nixpkgs#91317 (by mweinelt, 24 minutes ago, open): pythonPackages.mido: init at 1.2.9

20:47 <cole-h> Yeah, eval-4 is out of space

20:47 <cole-h> Oh

20:47 <cole-h> eval-5 now too

21:00 orivej has quit [Quit: No Ping reply in 180 seconds.]

21:01 orivej has joined #nixos-borg

21:23 orivej has quit [Quit: No Ping reply in 180 seconds.]

21:24 orivej has joined #nixos-borg

21:26 <gchristensen> ouch

21:26 <gchristensen> let me get those redeployed ...

21:28 <cole-h> Hm

21:28 <cole-h> Or we can just drop them?

21:28 <gchristensen> sure

21:28 <gchristensen> that'll work too i think, try it!

21:29 <cole-h> How do you mean? I think it needs to be done manually :^) We already redeployed back to 3 evaluators, and 4, 5, and 6 didn't die.

21:29 <gchristensen> oh interesting

21:29 <cole-h> https://buildkite.com/grahamc/ofborg/builds/90 > OFBORG_PACKET_HOSTS="3"

21:29 <cole-h> So destroying them with nixops didn't decommission them.

21:30 <gchristensen> destroyed

21:32 orivej has quit [Ping timeout: 260 seconds]

21:32 <cole-h> Yep, confirmed. I see 3 evaluators and builders in grafana.

21:32 orivej has joined #nixos-borg

21:33 <cole-h> Is that something nixops-packet should be responsible for?

21:34 <cole-h> (The actually-destroying of previously-provisioned systems)

21:35 <gchristensen> it is, you have to pass --kill-obsolete

21:36 <cole-h> https://github.com/ofborg/infrastructure/blob/master/build/build.sh#L42 Like this? :P

21:37 <gchristensen> uh ...

21:37 <gchristensen> yeah ...

21:37 <cole-h> x)

21:40 orivej has quit [Ping timeout: 246 seconds]

21:41 orivej has joined #nixos-borg

21:43 <gchristensen> cole-h: maybe replace http://147.75.194.151/result/x86/netboot.ipxe with ipxeScriptUrl = "http://images.packet.net/nixos/installer-pre/x86/netboot.ipxe";

21:44 <cole-h> What should that do?

21:44 <gchristensen> https://github.com/ofborg/infrastructure/blob/master/nixops/network.nix#L109

21:44 <gchristensen> it'll keep working when I don't control that IP anymore :)

21:44 <cole-h> Oh, so not totally related to the above issue :P

21:45 <gchristensen> right

21:45 <cole-h> Will do, boss.

21:47 <cole-h> hexa-: btw, space issue should be resolved now that the machines are no longer a part of the network :^)

21:49 orivej has quit [Ping timeout: 260 seconds]

21:50 orivej has joined #nixos-borg

22:12 <cole-h> https://github.com/ofborg/infrastructure/pull/35 Done.

22:12 <{^_^}> ofborg/infrastructure#35 (by cole-h, 33 seconds ago, open): Use packet domain for ipxe script

22:13 orivej has quit [Quit: No Ping reply in 180 seconds.]

22:14 orivej has joined #nixos-borg

22:44 orivej has quit [Quit: No Ping reply in 210 seconds.]

22:45 orivej has joined #nixos-borg

22:54 orivej has quit [Ping timeout: 260 seconds]

22:54 orivej_ has joined #nixos-borg

23:11 orivej_ has quit [Ping timeout: 258 seconds]

23:51 <hexa-> thanks everyone