<gchristensen>
I'm building one big netboot image which contains all the hardware support for a bunch of hardware classes by including their hardware.nix, which each individually set that console option
<worldofpeace_>
lol, so it's only duplicated a few times. no sweat 😅
Scriptkiddi has joined #nixos-dev
das_j has joined #nixos-dev
<gchristensen>
:P
<gchristensen>
doesn't seem to cause trouble, so I'll just go for it
<samueldr>
gchristensen: the log will output only on the last (or first?) console
<clever>
Profpatsch: only problem i can forsee with that, is that it complicates compiling for another platform
<clever>
Profpatsch: i cant just realize a given path that hydra built, i am forced to ssh out to a build machine of the right arch
<clever>
but thats an edgecase, where i would already be paying that cost for every single drv
<clever>
so its a choice between always paying the narinfo lookup cost (remove allowSubstitutes) or sometimes paying the remote-build cost
<Profpatsch>
clever: Hm, you mean when allowSubstitutes is set you can’t use it if you don’t have a builder for the arch
<Profpatsch>
clever: If it’s already in the store, there is no substitution anyway. If you *could* subsitute it, you have to push on the builder, yes
<Profpatsch>
But only if you either don’t build or it’s a different arch
orivej has joined #nixos-dev
<clever>
Profpatsch: yeah
FRidh has quit [Quit: Konversation terminated!]
<clever>
gchristensen: sure
<gchristensen>
clever: so I can accept CPR instructions now and partition and mount disks based on the CPR data
<gchristensen>
including ZFS on /
drakonis has joined #nixos-dev
<clever>
gchristensen: do the machines have 2 or 3 disks?
<gchristensen>
clever: one second, I'll get to that
<gchristensen>
clever: https://gist.github.com/grahamc/b85725db4f90aa03004583f9bac56225 here are the files from /etc/nixos/packet/ does this look good to you? note: I plan on moving the users.users.root.openssh.authorizedKeys.keys definition to a different file (named credentials.nix or something) so it can be easily ignored
* clever
looks
<clever>
gchristensen: somewhat related, what is packet.net's policy on repairs when a hdd fails?
<gchristensen>
they don't do them
<clever>
just report it, destroy the instance, and re-spawn a new one?
<gchristensen>
yep
<clever>
no problems with by-id then
<clever>
id normally prefer by-uuid, so when you clone the disk during repair, it can survive
<clever>
but if you cant repair, it doesnt matter
<gchristensen>
*SOMETIMES* they will do it, but they make no guarantees it'll work, and they only do it for customers who they know have very high technical competence and know what to do to make it work
<gchristensen>
so in this case the by-id is sufficient
<clever>
and with hdd swaps, there is always the risk of swapping the wrong device
<gchristensen>
yeah, so they really just don't do them
<clever>
gchristensen: will these machines always be efi, or are some legacy?
<gchristensen>
almost all of them will be MBR
<gchristensen>
c2.medium.x86 and their ARM machines are EFI, the rest are MBR
<clever>
gchristensen: youll want a symlink like this in boot.loader.grub.devices, when on legacy
<gchristensen>
yep, I'll get to that part :)
<clever>
and if you have redundancy, i think you can use 2, but ive never tested that part of grub
<gchristensen>
yep
<samueldr>
(pedantic samueldr alert!) Legacy Boot*; MBR/GPT is the partitioning scheme, and both are likely to work on somewhat recent implementations of both EFI/Legacy boot firmwares :)
<gchristensen>
with the exception of the authorizedKeys part, does this metadata layout for this system look good?
<clever>
still reading over it
<clever>
samueldr: yeah, ive seen efi on mbr work with the rpi&tianocore
<clever>
and i often do legacy on gpt
<samueldr>
the most important bit of that distinction is, rightly so, booting Legacy Boot GRUB on GPT; if you misuse "MBR boot" then it'll be assumed you don't need the BIOS Boot partition!
<clever>
gchristensen: in slug.nix, i would rather you use mkMerge, that will give better error messages
<gchristensen>
oh cool, can you show me an example of that?
<clever>
gchristensen: if you want legacy on gpt, you must create a bios boot partition, no fs, not formated, not mounted, 1mb in size
<clever>
gchristensen: when you boot.loader.grub.devices = "/dev/sda", grub will search for the bios boot partition, and automatically use it
<samueldr>
(or was it mkMerge that he needed an example of?)
<gchristensen>
clever: please focus on just this metadata for this one instance :)
<clever>
its pretty much identical to how your doing it with imports
<gchristensen>
perfect
<clever>
i think the module system will translate the above, into { config.services = mkMerge ( { foo = ...; } { bar = ...; } ); }
<clever>
which lets lazyness figure out what config.environment is, and figure out services later
<clever>
and it will recursively push it down, while keeping track of which file to blame for errors
<gchristensen>
sounds good
<gchristensen>
other than the mkMerge and moving the users.users.root.openssh.authorizedKeys.keys bit out of this file, looks good?
<clever>
looking over it more...
<clever>
nixpkgs.config.allowUnfree = true?
<gchristensen>
needed for hardware.enableAllFirmware = true;
<clever>
gchristensen: and why does it need all firmware?
<gchristensen>
they may not, I'll make a note to look in to that
<clever>
the rest of that gist all looks good
<gchristensen>
cool, and matches your needs for automation?
<clever>
gchristensen: i think so, the only thing missing is the choices for pool layout, which i think would be in the CPR data?
<gchristensen>
right, we're not talking about that yet :)
<gchristensen>
so for a MBR sernver I have this in the configuration: boot.loader.grub.devices = [ "/dev/disk/by-id/wwn-0x55cd2e414fbc4ee2-part1" ]; where part1 starts at sector 2048 and goes until sector 4096 (1M) and is the "MBR partition scheme" type. however, grub-install tells me: grub-install: error: unable to identify a filesystem in hostdisk//dev/sda; safety check can't be performed ... any thoughts?
<clever>
gchristensen: nope, you must never point that to a partition
<gchristensen>
ahh
<clever>
it must be pointed to the root device, like sda or sdb
<clever>
for MBR, grub will install a stub to sector 0, and then a stage 1.5 at sector 1024 i think (between sector 0 and partition 0)
<clever>
for GPT, the tables get in the way, and they wanted to ban using "unused" space :P
<clever>
so you must create a bios boot partition, and grub will lookup the offset on its own
<clever>
in both cases, you point grub to sda, a stub goes into sector 0, and the stage 1.5 goes "somewhere else"
<clever>
gpt just clearly defines that "somewhere else" as in use
<gchristensen>
perfect
<gchristensen>
this makes my code nicer too
<clever>
the LBA offset of "somewhere else" is baked into the sector0 code, so it can be found without supporting the partition tables
<clever>
and stage1.5 includes the grub kernel, and the grub modules required to mount /boot/
<clever>
sda here is an MBR disk, a stub is in sector 0, and stage 1.5 is directly after it at sector 1!! (my 1024 before was wrong)
<clever>
stage 1.5 includes fshelp, ext2, part_msdos, and biosdisk, along with a string saying to mount (,msdos2)/grub (the 2nd partition on the same device as the current MBR)
<clever>
by omiting the device, that drive can change to sdb or sdc, and still find the 2nd partition
<clever>
if your doing devices = [ "sda" "sdb" ];, then grub will include a device, so the MBR on both disks, will look for a /boot on one of them
<clever>
which creates a weak link in the redundancy
<clever>
i'm not sure how to setup fully raid'd /boot, but bootinfoscript will help confirm if you did it right
<clever>
the sdb in my above gist, is a gpt disk
<clever>
Device Start End Sectors Size Type
<clever>
/dev/sdb1 2048 67583 65536 32M BIOS boot
<clever>
so now its using the offset defined in the partition tables (and i made that way too big)
<clever>
that grub is not in use anymore, and it expects /boot/grub to be a directory on a zfs dataset called /root
<clever>
(i dont remember even doing that? lol
<clever>
ahh, sdb is from my old ssd+ssd mirror
drakonis has quit [Quit: WeeChat 2.6]
drakonis has joined #nixos-dev
drakonis has quit [Client Quit]
<clever>
gchristensen: as for the specifics of the CPR data, it depends on how many disks are present, and choices the user will want to make
<clever>
which will also impact the legacy grub config
<clever>
oh, you can also do legacy+efi on gpt, for extra redundancy
<gchristensen>
yeah, I think I accounted for the number of disks and a large number of choices
<clever>
ive not tested it, but i suspect raid'ing and efi might also work, with some extra effort
orivej has quit [Ping timeout: 240 seconds]
<clever>
some specific choices i'm thinking of
<gchristensen>
I don't support that
<clever>
1 disk, no choice, just bpp+boot+swap+zfs
<clever>
choice a: /boot is vfat, it must be tagged as the efi system partition in gpt
<gchristensen>
right
<clever>
choice b: /boot can be anything (for zfs, id recomend an ext4 partition), and then /boot/ESP is the ESP and vfat
<clever>
oops, thats /boot/EFI
<clever>
boot.loader.efi.efiSysMountPoint = "/boot/EFI"; is required, to make choice b work
<gchristensen>
how about /boot/efi
<clever>
that also works
<clever>
the path doesnt really matter
<clever>
choice b also allows /boot to be on the same partition as /, but zfs complicates that
<gchristensen>
I support either /boot or /boot/efi being the ESP partition
<clever>
the benefit of choice b, is that only the efi binaries live on the vfat
<clever>
the kernels and initrds can then live on ext4
<clever>
so vfat can be much smaller, and it rarely changes, so you dont have to worry about vfat lacking a journal
<clever>
another option, that i think some nixos users stumble into by accident, is a combination efi+legacy install
<clever>
if you have both a bios-boot partion, and an efi-system partition, and you set both boot.loader.grub.efiSupport and boot.loader.grub.devices = [ "/dev/sda" ];
<clever>
then grub installs both efi and legacy binaries to /boot, and configures both .efi executables and the MBR
<clever>
then if efi ever borks, you can just legacy boot the same disk
<clever>
but i dont know if packet.net lets you force legacy on an efi machine?
<clever>
legacy is also x86 specific
<clever>
if you have 3 disks, then your choices for zfs open up more
<clever>
i'm thinking, an option in the json, to make blkdiscard get ran on every disk, before it partitions
<gchristensen>
it does
<gchristensen>
every disk is completely discarded by the time my stuff runs
<clever>
perfect
<clever>
gchristensen: any particular reason you used sdd for the boot disk?
<gchristensen>
that is what Packet does out of th ebox
<clever>
ah
<gchristensen>
clever: the rules around boot devices are: if EFI: there must be single vfat filesystem mounted at /boot or /boot/efi and it must not be RAID-backed, ... if MBR: at least one disk must have a partiton with the label "BIOS", and every disk which has a partition labeled "BIOS" is added as a grub device
<gchristensen>
that is, those are the rules my tooling checks for
<clever>
ahh
<clever>
still writing...
<gchristensen>
cool
<clever>
its going to be over-engineered as heck :P
<gchristensen>
oh dear
<clever>
gchristensen: disks array is done, on to filesystems...
<gchristensen>
:)
<edef>
do we have a policy for external completions addon things?
<edef>
like, i'm packaging the gcloud completions for fish
<clever>
gchristensen: do we know the block size for these drives?
<gchristensen>
they're probably enterprise SSDs
ris has joined #nixos-dev
<gchristensen>
that is all I know though :)
justanotheruser has quit [Ping timeout: 245 seconds]
<clever>
gchristensen: `fdisk -l /dev/sda` can reveal that
<edef>
not unless you have very specialised drives
<clever>
Profpatsch: and now i need runCommandLocal, lol!
<gchristensen>
clever: I am concernedabout what you're creating :P
justanotheruser has joined #nixos-dev
<gchristensen>
btw CPR is available when you have reserved the instance, so you would know if you drive was 4k or not because you can craft the CPR data for that host
<clever>
gchristensen: datasets are now defined
<clever>
gchristensen: but i might need to boot something on the instance to inspect it? and then re-run CPR against it afterwards
<clever>
gchristensen: though, your tool could support this specific flag on its own
<gchristensen>
you would need to inspect the host to learn about it yes
<gchristensen>
that is why CPR is only available on reserved hw :)
<clever>
ashift, must be set to N, where the block size is 2^N
<clever>
ahh
<clever>
i was thinking you could just restrict cpr to fit within a template
<clever>
i want a machine with 2 disks and _ cpu, here is a CPR template that would work on any 2 disk machine
<gchristensen>
yea
<gchristensen>
they just don't do that right now
<clever>
cpr_zfs.pools and .datasets are done
<clever>
i think that just leaves mounts
<clever>
gchristensen: i think its done?, giving it a read-only
<clever>
over*
<gchristensen>
cool
<clever>
currently, its efi only
<clever>
and swap could be handled differently, but thats an experiment for later
<gchristensen>
clever: property values must all be strings
<ma27[m]>
is the `-unstable-` infix supposed to be part of `pname` or `version`? (IIRC this has been mentioned in nixos office hours some time ago and it was suggested to do `pname = "pkg-unstable";`, but I'm not 100% sure anymore)
<gchristensen>
31.60% of uncompressed filesystem size (1494488.69 Kbytes)
<gchristensen>
but even with 96 cores it takes several minutes
<clever>
7 # For zstd compression you can use "zstd -Xcompression-level 6".
<clever>
select COMPRESSION compression. Compressors available: gzip (default), lzma (no kernel support), lzo, lz4 and xz.
<clever>
-b BLOCK_SIZE
<clever>
Use DICT_SIZE as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc.
<clever>
gchristensen: interesting, there is an `-Xbcj arm` "filter" that does .... something