lordcirth_ has quit [Remote host closed the connection]
<colemickens>
Does it not make sense to split sd-image-rpi4.nix into two separate files so that users can build their own custom SD card image without getting the installer bits too?
<samueldr>
that specific derivation will go away once mainline linux can boot on the rpi4
<samueldr>
now, about the actual subject of the question, can we make it easier for end-users to use nixpkgs to build their custom SD card images without installer bits
<samueldr>
I don't see why not
<colemickens>
:) okay. sometimes with some of the various image modules I'm always a bit unsure of how intentional certain choices were, versus someone just trying to get something working, etc.
<colemickens>
I'll play around with it as I get my rpi4 this week and start playing with it and try to help out. I am trying to avoid any interactive login at all hopefully.
<samueldr>
I haven't changed much of the sd-image stuff, it's as it has been written a long time ago
<samueldr>
so I'm not sure either :)
orivej has quit [Ping timeout: 268 seconds]
ryantrinkle has quit [Ping timeout: 240 seconds]
zupo has joined #nixos-aarch64
orivej has joined #nixos-aarch64
zupo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
ryantrinkle has joined #nixos-aarch64
zupo has joined #nixos-aarch64
zupo_ has joined #nixos-aarch64
zupo has quit [Ping timeout: 240 seconds]
makefu has quit [Ping timeout: 245 seconds]
WilliButz has quit [Ping timeout: 246 seconds]
fpletz has quit [Ping timeout: 248 seconds]
pbb has quit [Ping timeout: 246 seconds]
pbb has joined #nixos-aarch64
fpletz has joined #nixos-aarch64
makefu has joined #nixos-aarch64
WilliButz has joined #nixos-aarch64
zupo_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
zupo has joined #nixos-aarch64
zupo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
zupo has joined #nixos-aarch64
zupo has quit [Client Quit]
zupo has joined #nixos-aarch64
ryantrinkle has quit [Ping timeout: 265 seconds]
<Raito_Bezarius>
grw: I pulled all nix channel updates again and I'll try to see again, how much time does it take on your machine (with which specs)?
<Raito_Bezarius>
And I think I understand more the issue with qemu compilation
<Raito_Bezarius>
I don't think someone has succeeded to use the user emulation mode to generate aarch64 SD image, the problem might be that user emulation do not implement the syscall cptofs is using properly and is faking it's working when in truth, it is doing nothing
<Raito_Bezarius>
What could prove this theory would be to enable full emulation rather than user emulation for qemu, but unsure how Nix perform the choice between user/full, is there any parameter to control this?
orivej has quit [Ping timeout: 268 seconds]
zupo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<sphalerite>
Raito_Bezarius: are you sure qemu is involved here? This looks like regular cross to me.
zupo has joined #nixos-aarch64
<Raito_Bezarius>
sphalerite: This was a cross compilation
<Raito_Bezarius>
Sorry for the confusion, I'm trying simultaneously the both ways
<Raito_Bezarius>
QEMU emulation to build a SD image
<Raito_Bezarius>
Cross compilation to build a SD image
<Raito_Bezarius>
I'm not super great with Nix, how can I apply this patch to my nixpkgs locally?
<Raito_Bezarius>
Should I just find the proper /nix/store/xxx and patch it directly?
<samueldr>
that's just a git diff, so git apply the patch source, should work, or remove those lines manually
<samueldr>
ah, you're not working from a git checkout?
<Raito_Bezarius>
You're right, I should just work from a git checkout…
<Raito_Bezarius>
Anyway, meanwhile, I tried to mimic what the ext4 image creation process does locally without qemu, it works
<Raito_Bezarius>
But, something is bugging me, how does Nix choose its QEMU binary?
<Raito_Bezarius>
Is it like in some /nix/store/<path> ?
<samueldr>
depends on how it's configured, but AFAIK it should be through the binfmt mechanism of the kernel
<Raito_Bezarius>
This is super strange
<Raito_Bezarius>
I see a certain binary /usr/bin/qemu-aarch64-static
<Raito_Bezarius>
But I uninstalled it
<samueldr>
oh, you're not on nixos?
<Raito_Bezarius>
Yes, I'm on Arch Linux
<samueldr>
nix doesn't deal with selecting a qemu binary, (unless something new passed me)
<Raito_Bezarius>
So I suppose the kernel puts in memory/cache the binfmt handlers
<samueldr>
so it's the binfmt mechanism of your host system
<Raito_Bezarius>
And I should reboot ™ to use the other mechanism
<Raito_Bezarius>
s/mechanism/binary
<Raito_Bezarius>
Alright, I patched & reran the build, but it says that he wants an aarch64-linux
<Raito_Bezarius>
Where should I patch the aarch64 platform stuff?
<samueldr>
$ nix-build nixos -I nixos-config=nixos/modules/installer/cd-dvd/sd-image-aarch64.nix -A config.system.build.sdImage
<Raito_Bezarius>
Right, thanks…
<samueldr>
what command were you using?
<Raito_Bezarius>
I added some --argstr system which of course triggered the stuff
<samueldr>
that's what I assumed :)
<Raito_Bezarius>
Now, it's telling me about the RPi firmware pkg not available on x86_64
<samueldr>
did you apply the patch?
<Raito_Bezarius>
Yes
<Raito_Bezarius>
error: Package ‘raspberrypi-firmware-1.20190819’ in /nix/store/r3nrnqlvgl6lwbrmbq9s5cfgar8212z6-nixpkgs-20.03pre200828.1e7ddf233a5/nixpkgs/pkgs/os-specific/linux/firmware/raspberrypi/default.nix:20 is not supported on ‘x86_64-linux’, refusing to evaluate.
<samueldr>
you're running the command from the root of that git checkout?
<Raito_Bezarius>
I have $NIXPKGS env var set to the root of the git checkout
<Raito_Bezarius>
Maybe, it's not using this env var at all
<Raito_Bezarius>
I guess it's NIX_PATH rather
<samueldr>
I don't think NIXPKGS is used for anything :)
<Raito_Bezarius>
Okay, I'm using NIX_PATH and it's doing something "new" :D
THFKA4 has quit [Ping timeout: 260 seconds]
THFKA4 has joined #nixos-aarch64
<Raito_Bezarius>
Okay I confirm
<Raito_Bezarius>
It's working
<Raito_Bezarius>
It passes the dreadful step of cptofs some stuff around
<samueldr>
right, so it's most likely the qemu binfmt stuff
<Raito_Bezarius>
or cptofs under qemu
<samueldr>
that's what I meant, under qemu-user
<Raito_Bezarius>
Because at some point, the qemu binfmt stuff was correct but I wanted to try qemu upstream
<Raito_Bezarius>
Ah okay
<Raito_Bezarius>
Truthfully, I have no idea what's going on
<Raito_Bezarius>
Thank you very much samueldr
<samueldr>
qemu-user takes a binary for a target architecture (here aarch64) and executes it through emulation, up until syscalls, where it translates to native syscalls from your system
<samueldr>
so here it runs cptofs code as aarch64 emulation (among others), but when it calls a syscall, it runs natively, which is much quicker
<Raito_Bezarius>
I understood this part
<samueldr>
ah
<Raito_Bezarius>
But, except if qemu-aarch64 user emulation mode do not implement some syscalls that lkl is using
<Raito_Bezarius>
I don't see why is there such a difference
<samueldr>
I don't know
<Raito_Bezarius>
As far as I understood by peeking cptofs code, it looks like they're mounting the img file in RAM under a RAM kernel or something like this
<samueldr>
lkl is a userspace port of parts of the kernel
<Raito_Bezarius>
So it's executing a very small kernel blob in userland, right?
<samueldr>
it's running as much as possible the same code as the kernel, but as an application / a library
<Raito_Bezarius>
BTW, is it necessary to run cptofs as aarch64?
<Raito_Bezarius>
If I understood correctly, it should be enough to cptofs as host as long as the data is crafted for aarch64
<samueldr>
no, but nix doesn't really have ways to cheat
<Raito_Bezarius>
Alright, I imagined so
<samueldr>
though *you* could do it I guess
<clever>
samueldr: you can just re-import nixpkgs, and not set system, when getting the cptofs utils
<samueldr>
clever: yeah, that's what I had in mind
<clever>
you kind of need 2 types of cross-compile
<clever>
one that actually cross-compiles, and uses host tools to generate ext images
<clever>
and a second, that native compiles, but still uses host tools for generating images
<Raito_Bezarius>
Would it be better if host tools were used for generating ext images? Is there any drawback to this cheat of "reimport nixpkgs, unset system, perform <some stuff as host>"?
<clever>
potentially, you loose overlays and config= when re-importing nixpkgs, and it harms the eval performance some
<samueldr>
building stuff through qemu-user is not supported, so it's not really a case that's thought of
<clever>
samueldr: the 2nd cross-compile type also allows using an rpi to build stuff, but x86 to squashfs it
<samueldr>
the supported way is a native build, and the getting supported way is full-blown cross-compilation
<clever>
and the 2nd cross-compile type can still be used in exactly that manner
<Raito_Bezarius>
So I suppose the simplest fix for me is just to create a qemu full emulation VM, set up NixOS in it, register it as a remote builder and hop (?)
<samueldr>
that could work, though expect it to be slow as emulating a system is slow
<clever>
Raito_Bezarius: i also have a nix expression that could be adjusted to do that...
<clever>
the host one, defaults to whatever your host is (darwin was the original idea)
<clever>
the linux one, forces x86 linux (you would change this to aarch64 linux)
<clever>
it then native builds an entire nixos for use within the qemu
<Raito_Bezarius>
wow, nice clever, I'm going to play around with it whenever I have time then
<Raito_Bezarius>
clever: how slow is it on your machine?
<clever>
then grabs the host qemu, and configures it to run that nixos
<clever>
ive not tried it on aarch64
<clever>
it was meant to run x86-linux on x86-darwin, with full hw accel
<samueldr>
Raito_Bezarius: another option for you could be to setup another raspberry pi 3 aarch64 system, and nix-build natively on it
<Raito_Bezarius>
hm makes sense
<Raito_Bezarius>
samueldr: Yeah, I try to avoid to rely on this solution because I have exactly 1 RPi 3 and I want to set it up without keyboard/HDMI monitor
<samueldr>
if there are distros that allow networking configuration to happen before burning the image
<clever>
oh, heh, line 11-14 enables qemu-user, inside qemu! lol
<samueldr>
that would obviously be only to kickstart yourself
<clever>
because it was meant to give qemu-user powers to darwin
<Raito_Bezarius>
samueldr: Yes, I could do this but I want to stick full NixOS, it's also a way to see how to do stuff fully in Nix™
<Raito_Bezarius>
(I'm not getting really shit done, I just want to stick to this path and evaluate what can be done, etc.)
<samueldr>
right now the way to stick to full nixos and fully nix is to build natively ;)
<Raito_Bezarius>
Last resort, I could start some aarch64 machine on Packet to register as a remote builder :p
<Raito_Bezarius>
Haha
<clever>
Raito_Bezarius: aarch64 community box
<Raito_Bezarius>
clever: I don't have access to it
<samueldr>
likely to lack the recognition needed
<clever>
Raito_Bezarius: you can just file a PR to add yourself to it
<Raito_Bezarius>
I'm not sure I would be accepted as I'm not a lot in the community
<samueldr>
(yet)
<Raito_Bezarius>
FWIW, I am in CNCF for some security research around NixOps/Kubernetes
<thefloweringash>
If you have the toplevel built you could potentially stitch it together manually with `nix copy` and the extlinux builder. Would be delicate though.
<Raito_Bezarius>
(rather, I have access to CNCF Packet account for this task)
<Raito_Bezarius>
(samueldr: :D)
<samueldr>
since it hangs in cptofs, I figure it's complete
<samueldr>
Raito_Bezarius: you could also look at the git history of the sd imag builder
<samueldr>
and find out the command used before cptofs
<Raito_Bezarius>
samueldr: Yes, I have seen the magic script of before
<samueldr>
it's the ext[2,3,4] debug tool
<samueldr>
maybe reverting to it would work better in your case
<Raito_Bezarius>
Unsure of what was its drawbacks though?
<samueldr>
I don't remember
<Raito_Bezarius>
Oh, that's a good idea
<samueldr>
one of them though is that it's ext specific
<Raito_Bezarius>
I really want to play with clever qemu config though haha
<Raito_Bezarius>
Ah
<samueldr>
so no way forward with *that* tool if we want to do f2fs or other fancy fs
<thefloweringash>
I would love to see a squashfs based arm image be a standard option. I tend to nixos-install most of my arm things anyway
<Raito_Bezarius>
I'm not sure if something else than ext4 is a fine idea on a RPi
<Raito_Bezarius>
Like does it make sense to put zfs on a SD card?
<clever>
thefloweringash: not-os does use a squashfs
<samueldr>
thefloweringash: just do it :)
zupo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<clever>
Raito_Bezarius: i have been considering testing that on a pi, since i want snapshots, rollbacks, and better atomicity
<samueldr>
but don't mistake it for justdoit.nix
<thefloweringash>
When I find a moment I guess I’ll PR it
<samueldr>
(I personally already do by booting the efi iso on aarch64)
<clever>
samueldr: i wonder..... would justdoit.nix work on a pi, out of the box? lol
<Raito_Bezarius>
clever: That would be an interesting experiment
<samueldr>
clever: armv7l though I guess
zupo has joined #nixos-aarch64
<Raito_Bezarius>
what is justdoit.nix BTW exactly?
<thefloweringash>
Which boards have efi support?
<clever>
Raito_Bezarius: i'm also attempting to write open source firmware for the rpi4, to replace every blob that can be replaced
<samueldr>
thefloweringash: u-boot does
<Raito_Bezarius>
thefloweringash: AFAIK, RPi 3 has it
<samueldr>
Raito_Bezarius: it can through tianocore yeah
<samueldr>
and rpi4 is getting in shape too
<samueldr>
(it also can through u-boot)
<Raito_Bezarius>
clever: Okay, I actually just used to infect some VPS to NixOS, it works super well haha
<clever>
Raito_Bezarius: basically, you add justdoit.nix to your install media (iso, netboot, normal install to usb, kexec), and you set some options to define how you want it installed
<clever>
Raito_Bezarius: then you boot that install media, ssh in, and run justdoit
<Raito_Bezarius>
can it be done that justdoit run automatically at boot?
<clever>
easily
<Raito_Bezarius>
super nice
<clever>
just generate a systemd service called `goodbye-data.service` and have it run justdoit + shutdown, lol
zupo has quit [Client Quit]
<clever>
my attacks on the rpi4 mask rom have currently hit a roadblock, i need to either re-implement an sdhci controller and sd card, or add vc4 support to qemu
<clever>
i'm attempting to add vc4 support to qemu, but its resulting in a cpu-less vm
<Raito_Bezarius>
can't you reuse some sdhci controller/sd card from the linux kernel?
<clever>
i need the controller, not the driver
<Raito_Bezarius>
ah, it's some proprietary blob?
<clever>
and once thats done, all i have is a fifo pushing bytes to an SD card
<clever>
so i ten have to emulate the SD card itself
<clever>
Raito_Bezarius: yep, i have a blob (from the mask rom) that talks to the sdhci controller to load firmware files
<Raito_Bezarius>
that looks super interesting, are you attempts somewhere on some repository?
<clever>
not yet
<clever>
i have a custom bootcode.bin, that can somewhat boot linux on an rpi3
<clever>
and recovery.bin appears to be the same basic format
<clever>
but there is undocumented validation against recovery.bin