jtojnar has quit [Remote host closed the connection]
jtojnar has joined #nixos-aarch64
efraim has quit [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]
efraim has joined #nixos-aarch64
jtojnar has quit [Remote host closed the connection]
jtojnar has joined #nixos-aarch64
orivej has joined #nixos-aarch64
orivej has quit [Ping timeout: 272 seconds]
FRidh has joined #nixos-aarch64
orivej has joined #nixos-aarch64
FRidh has quit [Quit: Konversation terminated!]
orivej has quit [Ping timeout: 244 seconds]
shad has quit [Remote host closed the connection]
shad has joined #nixos-aarch64
shad has quit [Remote host closed the connection]
shad has joined #nixos-aarch64
orivej has joined #nixos-aarch64
sphalerite has quit [Quit: WeeChat 2.0]
sphalerite has joined #nixos-aarch64
orivej has quit [Ping timeout: 244 seconds]
jtojnar has quit [Remote host closed the connection]
jtojnar has joined #nixos-aarch64
Thra11 has joined #nixos-aarch64
<gchristensen> LnL: hmm! so I upgrade my client?
jtojnar has quit [Quit: jtojnar]
<LnL> yes, or the daemon on the other side
jtojnar has joined #nixos-aarch64
<gchristensen> ok, let's see if 2.1.2 on my end fixes it :)
<gchristensen> 2.1.2 on my client didn't work
<gchristensen> maybe I'll downgrade to 1.x for this part.
<sphalerite> gchristensen: I frequently get connection timeouts when sshing into the aarch64 community box. Any idea why this might be?
<sphalerite> particularly existing connections just no longer responding to input
Thra11 has quit [Ping timeout: 252 seconds]
<gchristensen> + nix-copy-closure --to root@147.75.65.54 /nix/store/a6b2aasayn3nfy0l1cjsanh3wglz53ml-netboot.drv
<gchristensen> copying 1387 paths...
<gchristensen> so close
<sphalerite> to..?
<sphalerite> also the community box seems to be completely unreachable for me right now..?
Thra11 has joined #nixos-aarch64
jtojnar has quit [Quit: jtojnar]
jtojnar_ has joined #nixos-aarch64
<gchristensen> deploying
<gchristensen> so close to 1337
<gchristensen> also yeah I've been being paged about it, hehe
<sphalerite> ooooh right
<sphalerite> nice alerts
<gchristensen> should be up now? mog you're good now.
<sphalerite> yep I'm back in :)
<mog> gchristensen, actually i accidently destroyed the private key can i send you a new pr?
<gchristensen> yes
<sphalerite> gchristensen: I was thinking I'd do a PR adding a MOTD warning about non-persistence to the machine, but am not sure in which section of configuration.nix to put it…
<gchristensen> sphalerite: make a new file, see my latest commit
<gchristensen> (just pushed seconds ago)
<gchristensen> btw if there are any prometheus junkies http://aarch64.nixos.community:9100/metrics
Ericson2314 has joined #nixos-aarch64
<gchristensen> mog, sphalerite: eta? ideally your eta's will be close to each other :)
<sphalerite> like 5-10min
<gchristensen> cool
<sphalerite> or less
<{^_^}> nix-community/aarch64-build-box#28 (by mogorman, 15 seconds ago, open): accidently destroyed previous key
<mog> thanks again gchristensen i backed it up this time to my server just in case
<sphalerite> aarch64-build-box#29
<sphalerite> nix-community/aarch64-build-box#29
<{^_^}> https://github.com/nix-community/aarch64-build-box/pull/29 (by lheckemann, 22 seconds ago, open): Add MOTD warning about lack of persistence
* gchristensen builds
<sphalerite> lol this means my build will stop again doesn't it xD
<sphalerite> maybe I should remove the mass-rebuild commits from it
<gchristensen> how long is the build?
<sphalerite> idk, haven't finished it before
<sphalerite> but I've cancelled it now so go right ahead
<gchristensen> mog: do you have stuff you want to bild ~now?
<gchristensen> I'm a few minutes away frombeing able to deploy
<gchristensen> ok deploying
<gchristensen> ok should be all set mog, sphalerite, LnL
Ericson2314 has quit [Changing host]
Ericson2314 has joined #nixos-aarch64
Ericson2314 has quit [Quit: Leaving]
Ericson2314 has joined #nixos-aarch64
Ericson2314 has quit [Changing host]
Ericson2314 has joined #nixos-aarch64
Ericson2314 has quit [Remote host closed the connection]
Ericson2314 has joined #nixos-aarch64
Ericson2314 has quit [Changing host]
Ericson2314 has joined #nixos-aarch64
Thra11 has quit [Ping timeout: 252 seconds]
<samueldr> sphalerite: I concur, recently ssh seems unresponsive to the aarch64 community machine
<samueldr> I wonder though if it's just load
<gchristensen> is it sessions timing out or something?
<samueldr> in my case it always comes back seconds later
<gchristensen> hrm
<samueldr> with a quick (maybe too quick) test, it doesn't seem to be having issues right now
<gchristensen> I should probably make this a % instead of absolute, now :)
jtojnar_ has quit [Quit: jtojnar_]
jtojnar has joined #nixos-aarch64
gchristensen changed the topic of #nixos-aarch64 to: Get access to the NixOS Community aarch64 build box: https://github.com/nix-community/aarch64-build-box ... todo: https://hydra.nixos.org/jobset/nixpkgs/trunk#tabs-jobs
gchristensen changed the topic of #nixos-aarch64 to: Get access to the NixOS Community aarch64 build box: https://github.com/nix-community/aarch64-build-box ... todo: https://hydra.nixos.org/jobset/nixpkgs/trunk#tabs-jobs build box status: https://monitoring.nix.ci/d/hkRCcV0mk/instance-metrics?var-instance=aarch64.nixos.community&orgId=1
<gchristensen> now you can go and see if something is up when having problems with the builder.
<gchristensen> you can run arbitrary queries here https://nix.ci/prometheus/graph
<gchristensen> for example: https://nix.ci/prometheus/graph?g0.range_input=1h&g0.expr=node_network_receive_bytes%7Binstance%3D~%22%5Eaarch64.nixos.community%3A.*%22%7D&g0.tab=0
<samueldr> neat!
<samueldr> argh, another day out of my element!
<sphalerite> ?
<samueldr> playing around with kvm and arm
<gchristensen> love/hate those dayshaha
<samueldr> using this as a base https://nixos.wiki/wiki/NixOS_on_ARM/QEMU
<sphalerite> time outside your element is the only way to expand the definition of your element ;)
<samueldr> yes!
<samueldr> using this command: qemu-system-aarch64 -machine virt,highmem=off -bios uboot-qemu_arm_defconfig-2018.07_u-boot.bin -drive if=none,file=sd-image-armv7l-linux.qcow2,id=mydisk -device ich9-ahci,id=ahci -device ide-drive,drive=mydisk,bus=ahci.0 -netdev user,id=net0 -device virtio-net-pci,netdev=net0 -nographic -smp 2 -m 512M -enable-kvm -cpu host,aarch64=off
<sphalerite> ooft
<samueldr> you see, adding "-enable-kvm -cpu host,aarch64=off"
<samueldr> the boot is MUCH faster... for what it boots
<gchristensen> :D
<samueldr> but it seems to fail somewhere along the storage drivers
<gchristensen> so about those graphs ^ I've been wanting to do that _forever_ because I have literally no idea how much the thing is actually used
<samueldr> just sharing here first in case someone has experience, now digging to see if I can corner the issue
<gchristensen> (oops, I didn't mean to step on the existing chat with idle banter, I wasn't thinking)
<samueldr> no worries
<samueldr> but if it works, it'd be useful (though not on the aarch64-only cpus like the thunderx)
<samueldr> (AFAIUI)
<sphalerite> gchristensen: following multiple independent interleaved conversation threads is a key skill to have for online communications, no? ;)
<sphalerite> samueldr: it would indeed! There's a decently-powerful box at packet which gchristensen got me access to a while back which supports aarch32 as well as aarch64
<sphalerite> it's not booting currently unfortunately
<sphalerite> but yeah
<samueldr> and is it me or --option system armv7l-linux fails on aarch64?
<sphalerite> gchristensen: those little things that you want to do forever and never get around to!
<sphalerite> samueldr: I did some fiddling with that
<samueldr> maybe it's a raspi thing, builder for '/nix/store/wlmx3k9zdqc57r3w7wc3nd65zdc5x2ry-bootstrap-tools.drv' failed due to signal 31 (Bad system call)
<sphalerite> samueldr: https://github.com/NixOS/nix/pull/1916 --extra-platforms armv7l-linux might fix it
<{^_^}> nix#1916 (by lheckemann, 29 weeks ago, merged): Support extra compatible architectures
<sphalerite> ah yes that was a seccomp issue
<samueldr> nix-build -A hello --option system armv7l-linux
<samueldr> it's not part of 2.0 as I can see
* samueldr tres
<samueldr> tries*
<sphalerite> I think what you might want instead is nix-build --extra-platforms armv7l-linux --argstr system armv7l-linux
<samueldr> will probably need a more recent nix anyway?
<sphalerite> it was already in 2.0 I think
<samueldr> -> un_s_able to add
<sphalerite> ah no you're right
<sphalerite> whaaaaat I never noticed that typo!
<samueldr> sorry :|
<samueldr> :)
<sphalerite> x)
<samueldr> don't worry, everyone makes typos :(
<sphalerite> maybe it was muscle memory wanting to write unstable
<sphalerite> :D
Thra11 has joined #nixos-aarch64
<sphalerite> aaaaah there's another typo, missing close paren D:
<gchristensen> _sighhhh_ compiling the kernel again.
<gchristensen> even with -j8 it is slow.
<gchristensen> https://grahamc.com/blog/an-epyc-nixos-build-farm need one of these
<sphalerite> gchristensen: also in that category, I set up DNS for my VPN the other day. It's really nice being able to type names instead of numbers, and use ipv6!
<gchristensen> :o! how did you do it?
<sphalerite> I generate the zone file and the tinc config from a single configuration
<samueldr> gchristensen: on armv7l; for an orangepipc (so not beefy), building a non-generic kernel took around twenty hours IIRC
<gchristensen> uggggh
<sphalerite> so now I have names like sudoku.sphalerite.tech for my main laptop (of course these aren't very useful outside the VPN but hey)
<gchristensen> sphalerite: oh that box should be booting now, but I can't look yet.
<samueldr> at least it worked the second try :')
<sphalerite> I also really like the .tech TLD :D
<sphalerite> gchristensen: ooooh awesome!
<sphalerite> samueldr: really, that long? On my first chromebook it took about 5-6h for the nixos one with all the modules iirc
<samueldr> chromebooks were beefier I think
<samueldr> and it might have been a bit generic
<sphalerite> huh ok
<sphalerite> but by *that* much?
<samueldr> maybe storage speed? while SD card for both, maybe a bad bus + a bad card?
<sphalerite> I do think the main bottleneck was storage, so it might — lol
Thra11 has quit [Ping timeout: 252 seconds]
<sphalerite> great minds, eh? :D
<samueldr> I'm now booting the raspberry pi off a usb drive, seems better
<samueldr> (can't do usb hard drive though, u-boot doesn't want to start it, even though it loaded from it)
<sphalerite> oh yeah, I ordered a fancy big microSD card for my chromebook the other day
<sphalerite> that's… odd?
<samueldr> maybe not, I'm probably wrong in some aspect, but AFAIUI hard drives and usb drives may talk different usb protocols
<samueldr> sphalerite: use `f3` (f3write, f3read) to both check the speed and verify the card is legit
<sphalerite> stick the kernel on an SD card and the rootfs on the hard drive?
<samueldr> sphalerite: that's an alternative I was thinking of
<sphalerite> ooh, thanks for the tip. Never heard of that
<sphalerite> but samsung ought to be good, no?
<samueldr> if it's a genuine one, yeah
Ericson2314 has quit [Ping timeout: 252 seconds]
<samueldr> it's not made for benchmarking, but still gives a good ballpark
<samueldr> I started putting them in public gists hoping people searching model numbers will find them to see write speeds https://gist.github.com/samueldr/7a7ff37feb12b59b73c107978b81e215
<sphalerite> also it's crazy how much the price of microSD cards has dropped
<samueldr> as often they're quoted ONLY for UP TO read speeds
<sphalerite> oooh good work!
<samueldr> sphalerite: dunno if you have looked, and how much you've looked the past 5 years, but SSD too have dropped creazy
<samueldr> crazy*
<sphalerite> I mean more recently
<sphalerite> I bought a 32GB one in February, found the other day that a 64GB one with a higher speed class was now cheaper
<samueldr> speed classes :( what a crapload
<samueldr> especially if you think UHS greated than 1 will help
<samueldr> you need a reader that has the second row of contacts AFAIR
<sphalerite> yeah I'm not so convinced by those either, but I'm sure they affect the price
<sphalerite> https://bunniestudios.com/blog/images/microsd_1.jpg misread the number on the card as the left as "Adolf" :|
<samueldr> lol
<samueldr> I hate buying SD cards because of the fact that it apparently is hard to make sure it's genuine
<samueldr> even e.g. buying from a reputable brick and mortar shop :/
<gchristensen> this dang kernel is still compiling.
<gchristensen> it has taken longer than the house cleaning.
<sphalerite> https://www.bunniestudios.com/blog/?page_id=1022 this article was really fascinating
<sphalerite> "Unfortunately, you can’t buy Samsung-branded microSD cards on the retail market, as far as I know" well this has changed :D
<mog> gchristensen, Received disconnect from 147.75.79.198 port 22:2: Too many authentication failures
<mog> wont let me in by normal ssh either
<mog> im sure im using the key
<samueldr> (ssh works here)
<samueldr> good username?
<samueldr> did you turn it off and on again? (jk)
<mog> can you verify user mog is on the system?
<samueldr> oh, and if you have more than three keys in your agent I think this could happen
<samueldr> ssh with a couple of -v may show this
<samueldr> I don't see your username in /etc/passwd
<mog> seems likely i am not all the way inputed then?
<samueldr> gchristensen: I assume I would see mog in /etc/passwd if everything was right, right?
<mog> i mean you are there correct samueldr ? and other people in the key dir i assume
<samueldr> that's what I see
<samueldr> which is why I assume so
<samueldr> and you are in the users.nix file on master
<mog> maybe he hasnt done rebuild on it yet?
<sphalerite> oh yeah it's not printing the motd either
<sphalerite> gchristensen: the deploy didn't work right!
<gchristensen> hrm?
<gchristensen> how X)
<samueldr> you tell us :)
<gchristensen> hmm
<samueldr> sharing the failures I see on a raspi 3 (not plus) with kvm and armv7l https://gist.github.com/samueldr/d20c34ac233ea3bf397761fadccf28db
<samueldr> ooh, though I think I may have been impatient for those without panics; it might have been working :/ (retrying)
orivej has joined #nixos-aarch64
<mog> also are you running fail2ban or something similar on this box? im not seeing anything when i try to connect now
<gchristensen> it is redeploying
<gchristensen> but maybe? the whole config is public if you want to look
<mog> good point
<mog> nope nothing to ban
<gchristensen> try now mog
<mog> its asking for a password
<mog> i am specifying the key on the cli
<mog> let me do -v -v to be sure its sending it
<samueldr> [samueldr@arm-community:~]$ grep mog /etc/passwd || echo "oh no"
<samueldr> oh no
<gchristensen> ???
<mog> he means im not in the /etc/passwd file
<sphalerite> yep
<sphalerite> and you should be because I am
<gchristensen> Postponed keyboard-interactive for invalid user mog
<gchristensen> yeah I know, I just don't understand why
<sphalerite> and the motd still isn't showing
<sphalerite> I think it's just not booting the new thing?
<mog> i added the pub key to authorized keys on one of my other boxes
<mog> and sshed in using the private key and it did work
<mog> so my ssh-agent isnt doing anything crazy
<gchristensen> it is definitely not you
<mog> just wanted to be sure i wasnt the problem
<gchristensen> well Nth time is hopefully the charm
<mog> failure :(
<gchristensen> it isn't done deploying :)
<gchristensen> sorry for that not being more transparent
<mog> IM IMPATIENT!!!!
<mog> sorry
<gchristensen> the deploy process involves instantiation, copying the drv to a build host, building, scp'ing a netboot image to a web host, and then issuing a reboot
<mog> yup, i just misinterpreted your message and am excited to see it work
<gchristensen> love2watch enterprise grade hw boot
<gchristensen> takes long enough you can make a tea
<gchristensen> (it is tea time...)
<samueldr> motd!
<mog> Welcome to the NixOS community aarch64 build box!
<mog> huzzah
<gchristensen> yay!
<gchristensen> phew
<samueldr> details?
<gchristensen> cache coherency problems on the upstream iPXE infrastructure
<gchristensen> one part of their system updated teh boot url properly
<gchristensen> the other didn't
<samueldr> this sounds... painful :)
<gchristensen> it was good to watch the boot log :)
* mog builds erlang and elixir with all the cores
<gchristensen> the cores are your oyster
<mog> sooo fast
<gchristensen> the cores are slow
<gchristensen> there are just a LOT of them ;)
<mog> erlang normally takes 30 minutes or so on my laptop
<mog> the cores did it in 5 or so
<gchristensen> nice
<mog> can i build non-free things on builder by just copying over config? one of things im building wants jre8 which is nonfree
<samueldr> no restrictions, other than not having root access
<samueldr> do note that it may not be enough for java stuff :/
<mog> ive run the program on pi before when it was running debian stuff
<mog> its not using a lot on it
<sphalerite> mog: yeah if you're using remote building it'll jsut use your local config
<sphalerite> mog: otherwise you can enable unfree stuff same as you would locally, although of course it'll disappear sometime in the not-all-too-distant future :)
<mog> cool cool cool
<samueldr> hmmm, anyone has a pair of rootfs *image* + kernel for armv7l handy just to try qemu kvm on aarch64? (not nixos + u-boot based)
<samueldr> just making sure it *should* work
<sphalerite> samueldr: dezgeg's installer images?
<samueldr> u-boot based
<samueldr> those are the one failing
<samueldr> the fun thing is there isn't much documented online
<samueldr> and most of those with pre-built artifacts lead to 404s :(
<gchristensen> never good news:
<gchristensen> waiting for the machine to finish
<gchristensen> rebooting....[down]..................................................................................................................................................................................................................................................................................................................................................................................................
<gchristensen> ...............................................................................
<gchristensen> .....................................................................................................................................................................................................................................................................................................................................................................................................................
<gchristensen> .....................................................................................................................................................................................................................
<samueldr> ...
<sphalerite> ...
<samueldr> oh! we *do* have debootstrap in nixpkgs
<mog> !!!
* samueldr schemes
<sphalerite> samueldr: there's a bug with it which might bite you
<{^_^}> #27396 (by lheckemann, 1 year ago, open): debootstrap --foreign does not produce compatible results
<samueldr> foreign?
<samueldr> anyway, I'll try
<sphalerite> yeah when you do only part of the bootstrapping, not everything, allowing use for a foreign arch
Thra11 has joined #nixos-aarch64
Thra11 has quit [Ping timeout: 252 seconds]
<samueldr> well, it failed not due to foreign. something about missing every binaries in the world (sed, cut, etc), so probably somewhat related
<Dezgeg> you could probably use edk2 to EFI boot most other distros
<Dezgeg> or u-boot to EFI boot
<samueldr> right now most of my issue seems to be related to not finding prebuilt $everything for other distros :/
<samueldr> and for nixos+u-boot, u-boot failing to see the disk with virtio-blk-device, and with the same "ide-drive" setup as in the nixos wiki, panicing
<Dezgeg> fedora installer images should work
<samueldr> (haven't searched fully with fedora) what I found with fedora was always using virt-builder... which I didn't have the patience to look into building on aarch64
* samueldr checks for installer image
<Dezgeg> nixos+u-boot should also be working, afaik
<samueldr> works fine without kvm
<samueldr> the moment I add kvm, I get panics, but not deterministic panics :/
<samueldr> all of the logs are separate boots
<samueldr> hmm, I don't have a log of the failure with virtio-blk-device
<Dezgeg> weird, IIRC it has worked for me
<samueldr> I don't know how to verify whether virtio-blk should work or not with that u-boot build
<Dezgeg> it shouldn't
<samueldr> at least, for that, I'm seeing the expected result I think
<Dezgeg> yes
<Dezgeg> you could try this guy's branch: http://u-boot.10912.n7.nabble.com/PATCH-00-27-virtio-Introduce-VirtIO-driver-support-td341928.html but I don't think it will help with your actual problem
<samueldr> the panics, any idea, tips, or clues?
<Dezgeg> those seem to be just init and other apps crashing with SIGILL, maybe there's a way to get a coredump from the initrd shell or something
<clever> Dezgeg: if you `ulimit -c unlimited`, it will write to `core` by default
<Dezgeg> yes but here we mostly seem to crash before getting into a shell
<clever> Dezgeg: and if you write to /proc/sys/kernel/core_pattern you can give it a full path, and pattern for the core files
<clever> so they land in a common place, and dont overwrite eachother
<clever> and if that pattern begins with a |, then its a binary to spawn, which handles things
<samueldr> not sure is helpful in the initrd
<clever> yeah, you would need to at least mount something writable before it can really work
jtojnar has quit [Remote host closed the connection]
jtojnar has joined #nixos-aarch64