<vielmetti>
hey, if I was going to get a single-board computer or three to run nixos arm64 on, what's your current thinking? I know you have some new gear samueldr and wonder whether you'd repeat that again or look for something else
<samueldr>
hmm, none of the new gear shipped yet :/
<samueldr>
(oh, except one board which isn't really new)
<samueldr>
I think in the end it all depends on the desired use case, and other than one major "catch", there's not much that's nixos-specific in the choice to make
<samueldr>
the major catch is: prefer mainline-supported boards (by u-boot and linux), otherwise it may become harder either to bootstrap oneself and find the right downstream or community maintained forks to build the system
<{^_^}>
#51924 (by cyounkins, 1 day ago, open): Put kernel and initrd on ext4 in ARM images
<cyounkins>
samrose: I'm neither of those but I'm pretty sure yes. I just installed on RPi 3B. Had a few problems with HDMI.
<samueldr>
cyounkins: the most used I'm pretty sure
<samueldr>
supported it depends on the features; there are differences between the mainline kernel and the linux_rpi kernel. I don't know much about those differences, but they may make some third party accessories harder to use
<samueldr>
oops, samrose ^
<samueldr>
cyounkins: which problems with hdmi?
<samrose>
samueldr: are there some boards out there that are known to support mainline kernel and u-boot?
<cyounkins>
Rpi 3 should use mainline?
<samueldr>
now that there's linux_rpi for aarch64, maybe not
<samueldr>
didn't get around to test and see personally if it's better
<samueldr>
there are *some* issues when the board is detecting an under-volt, it draws that dastardly lightning bolt on the screen, and the mainline kernel doesn't deal well with it :/
<samueldr>
also, the vc4 modesetting kernel only starts working at stage-2 (haven't still looked at making it work at stage-1)
<samueldr>
and you may need to adjust the cma value which AFAIUI is the allocated ram for the gpu
<samrose>
made it all the way through the build much quicker, but it failed after compiling kernel, so I am trying again, as I had also set DRM_HDMI = yes in the sam common-configuration on the advice of Sunxi corp. I took that line out and trying again, thinking maybe DRM_HDMI could have been culprit
<samueldr>
I don't know about known boards to be supported both in mainline u-boot, mainline linux and nixos; I think the issues with the allwinner boards right now could be entirely due to configuration
<samueldr>
ideally, one would look at those upstream projects, see the defconfig, look at the "what's supported" page of those boards (e.g. at the linux-sunxi wiki), but your mileage may vary in the end :/
<samrose>
samueldr: it seems like you are right with allwinner, because there are working ubuntu, and debian distros for these boards (albeit kinda hacked together) and the differences seem to boild down to drivers and patches
<samueldr>
patches -> not mainline :)
cyounkins has quit [Remote host closed the connection]
<samueldr>
but yeah, I've read that mainline without patches should work for my board, I need to track a pre-built 4.20-based mainline build that works with ethernet to figure out the issue (here hdmi works)
<samrose>
the patches seemed to be in some cases.
<samrose>
samueldr: I wonder if my bpim64 board is using same ethernet hardware as yours?
<samueldr>
plausible
<samrose>
if I can get through one of these builds to see HDMI working, and if I lose ethernet, I have connection to engineers at sunxi and can also ask them what they recommend, but from my last communication they have only made it up to 4.4 on their end
<samrose>
and somehow, on 4.4 kernel, they were using "DRM_HDMI" but I wanted to try your recommendation of 4.20 + SUN8I_DE2_CCU = yes first before I mess with what they are saying
<samueldr>
yeah :/ downstream kernels are always a bummer
<samueldr>
after all, I can get android-specific downstream kernels to build and work on devices with nix and nixos :)
<THFKA4>
got a couple ODROIDs running 4.14 though
<THFKA4>
NixOS
<samrose>
THFKA4: :-O
<samueldr>
THFKA4: and you didn't contribute to the wiki?
<samueldr>
;)
<THFKA4>
my contributions would be 1) the current ARM images are out of date with some annoying bugs that force you to rebuild nix before doing system-rebuild
<THFKA4>
2) 2GB is not enough RAM to build Rust. the kernel runs out and the board resets
<samrose>
THFKA4: by "rebuild nix" is this referring to like a total recompile of kernel etc?
<samueldr>
the aarch64 image doesn't?
<THFKA4>
i mean the images from September that were up on the wiki for a while
<samueldr>
I'm thinking more about the board-specific things
<samueldr>
yeah, I'm thinking about those
<samueldr>
never had to rebuild nix
<THFKA4>
yeah, so they wouldn't be useful contributions is all i'm saying haha
<samueldr>
though maybe you're not wrong? maybe there's something you're doing I'm not :)
<THFKA4>
samrose: by "rebuilding nix" i mean pull the package manager, because that's where the annoying bug was
<THFKA4>
some hash didn't match and your system switch would abort
<samrose>
THFKA4: aha
<THFKA4>
i think pulling channels was also necessary
<THFKA4>
but anyway, i hear you guys have new images since yesterday
<samueldr>
the "pulling channels" will still be an issue: the nixos-18.09 release doesn't wait for a complete -aarch64 build :/
<samueldr>
(same for unstable)
<samueldr>
so if you update the channel as it is right now, it may end up on an incomplete rebuild of the system
<samueldr>
maybe that's why you had to rebuild nix? especially in the last week, there were a couple compounding issues with the aarch64 builds on hydra
<THFKA4>
the issue i ran into was solved in September, just after the images got released
<samueldr>
then maybe not :/
<THFKA4>
yeah, probably not
<samrose>
one thing I noticed about the images produced by for instance bananapi mfr vs the nixos images, was that there is way less in the bananapi ubuntu image. So I was thinking I would eventually do well to go through the kind of kitchen sink of sd-aarch64 image build, and figure out what is not needed for my board, and then produce an image stripped of things not neeed
<samrose>
and maybe this would solve some of the issues of running out of memory, space etc
<THFKA4>
the Rust thing is really weird, it dies even with 130GB of swap and --max-jobs 1
<THFKA4>
i think it's because it's kmem that it's running out of
<samrose>
(although we will require users of our system to augment with several GB extra USB storage that will be automunted by udiskie)
<THFKA4>
i was bringing up an AWS ARM server to try to get around this, but then found this channel
<samrose>
THFKA4: I was lazy and used packet.net :)
<samrose>
THFKA4: were you just doing a standard rust build?
<samrose>
samueldr: I wonder if there is a way to systematically compare images released for boards vs mainline + uboot ? I would volunteer some time toward this in the weeks/months to come
<samueldr>
you mean comparing downstream/community releases of u-boot/linux to mainline?
<samrose>
yes, and then maybe also compare to nixos aarch64
<samueldr>
no idea, and then the userland shouldn't matter much, so it'd be (I think) all down to the source :/
<samrose>
that is what i was foggily thinking about, diff of sources that could show the gaps
<samueldr>
maybe of better value is other mainline linux based builds compared to nixos, especially the kernel config
<samueldr>
but then, have you ever tried to diff 4.4 to 4.20? ;)
<samrose>
yeah, likely not a small one
<samueldr>
for allwinner, samrose, maybe better to ask around on #linux-sunxi; not a high traffic channel, but that's the main chat communication channel for the upstreaming efforts
<samueldr>
(there are far more knowledgeable people there than me)
<samrose>
samueldr: ok I will check it out. On the comparing releases, I was thinking beyond my own needs to giving anyone that comes across this effort some kind of short cut clues on the gaps
<samueldr>
I would be betting that as far as hardware support (so ignore anything like rust failing to compile) the userlands don't matter, so at least there's a huge chunk of the builds that can be ignored
<samueldr>
so you'd be left with u-boot and the kernel, the kernel can be summed AFAIK to /proc/config.gz
<samrose>
samueldr: I think you are right, for instance in my case it all boiled down to defconfig as you said so far
<samrose>
hmmm cool
cyounkins has joined #nixos-aarch64
<samrose>
it seems like in u-boot there is actually a lot of support for boards already
<samrose>
and u-boot is (kind of) well documented
<cyounkins>
samueldr: to answer your question from earlier, my HDMI issues on RPi 3 B with aarch64 image were a blank screen after "starting kernel...", then the monitor thinking the cable was disconnected. I had to change cables and monitors to get the display up, after which I could switch back to the first. I had the dreaded "vblank wait timed out" https://gist.github.com/cyounkins/abc75a51579df67993edb656a8ced2
<samrose>
THFKA4: I had run into this here and there that some system rebuilds, or nix package builds were running out of either storage, or memory on SBC and had me worried in general about the feasibility of my own use case
<samrose>
although I think once I have the minimum hardware working, there will probably be a way to address these issues
<THFKA4>
i'm pretty used to adding swap whenever cc gets kiled, but this case seems a little different
<samrose>
THFKA4: yes you mentioned you ran out of kmem as far as you knew
<THFKA4>
the whole machine usually becomes unresponsive and dies, even the heartbeat LED
<samrose>
damn
<THFKA4>
i left it running with dmesg -w and it complains about memory, yeah
<samrose>
THFKA4: interesting that it triggers rust install in the package
<THFKA4>
i think it's one of the addons
<samrose>
the "optional plugins" ?
<THFKA4>
i didn't adjust anything, so whatever is included in the default closure
<THFKA4>
yeah, plugins, not addons
<THFKA4>
the only "special" thing about this machine is it's running on an encrypted btrfs root
<THFKA4>
so that's pretty taxing
<samrose>
hmmm yeah, could be a factor
globin has quit [Ping timeout: 250 seconds]
Thra11 has joined #nixos-aarch64
<peel>
hey, what's the story with odroid-c2? i'm about to resurrect my rpi/odroid cluster and obviously i'd prefer running nixos.
<gchristensen>
I think I got it to work once, but I don't remember - I have one too. if you get it sorted, I'd love to also get it sorted :)
<THFKA4>
just tried it last week and it wouldn't boot
<THFKA4>
bad ARM image magic
<samueldr>
gchristensen: with the latest stock u-boot and the uefi iso I think you got it to boot?
<samueldr>
(uefi iso not yet built by hydra and WIP)
<peel>
well, well.. i remember last time i tried i ended up with arch and ansible, but ugh. needed that for a talk but no rush this time. will have a closer look.
<gchristensen>
I can try this afternoon
<samueldr>
right, I didn't open a PR for the odroid u-boot since it needed a bunch of cleanup
<samueldr>
(and I don't have the device, so I hate to just send a PR in that case)
<samueldr>
and finally, #51397 implements it for AArch64 on our iso, so you can `dd if=our.aarch64.nixos.iso of=/dev/usbdrive` and then boot from it and install just like on any other uefi machine; as long as the SBC has some uefi support
<samueldr>
once flashed, I can usb boot the UEFI usb drive without having an eMMC nor an SD card :)
<cyounkins>
Nice!
<samueldr>
my raspberry pi is booted via uefi on a USB hard drive
<samueldr>
there's no real advantage, though
<samueldr>
(yet?)
<makefu>
samueldr: well, the raspi will most likely not eat your hard drive as it does with sd-cards
<samueldr>
you had me confused for a while: I was talking about booting using UEFI, not the USB part :)
<samueldr>
as of right now with nixos, there are not advantages to go out of your way to boot using UEFI
<makefu>
ah
<gchristensen>
update: I mentioned I might try and get nixos on my odroid-c2, but unless someone tells me fairly specific set of commands to do it, I won't be able to tonight as I work on upgrading Packet to 18.09 images.
<gchristensen>
(nobody should feel obligated to do that)
<samueldr>
though now I'm thinking it'll be hell to partition the emmc right so it won't lose its u-boot install :/
<gchristensen>
I've got to run to a dinner date.
<samueldr>
oops, just realised I linked to a useless comment in the PR :)
<samueldr>
on an aarch64 machine, with the PR applied to a nixpkgs tree, `nix-build -A config.system.build.isoImage -I nixos-config=nixos/modules/installer/cd-dvd/installation-cd-minimal.nix --show-trace nixos/default.nix` would build the iso image that's uefi bootable
<samueldr>
(or otherwise I belive the sd image should be bootable, but I don't know how it'll work on your device)
<samueldr>
and both of those are contingent on mainline kernel support
<samrose>
yes kernel built but image failes with this zfs-kernel which is apparently part of the sd-aarch64 image build on testing
<samrose>
This is the result of running nix-build nixos -I nixos-config=nixos/modules/installer/cd-dvd/sd-image-aarch64.nix -A config.system.build.sdImage