samueldr changed the topic of #nixos-dev to: #nixos-dev NixOS Development (#nixos for questions) | NixOS 19.09 is released! https://discourse.nixos.org/t/nixos-19-09-release/4306 | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html | https://r13y.com | 19.09 RMs: disasm, sphalerite | https://logs.nix.samueldr.com/nixos-dev
worldofpeace_ has joined #nixos-dev
drakonis has quit [Remote host closed the connection]
drakonis has joined #nixos-dev
<jtojnar> hmm, there is also SEML
<gchristensen> tomorrow is Office Hours again! it'll be a bit casual, if you have PRs you want to look at or questions about how to get involved, etc, come chat with worldofpeace_ and me =) https://discourse.nixos.org/t/nixos-office-hours-2019-11-29/4956
drakonis has quit [Read error: Connection reset by peer]
orivej has quit [Ping timeout: 276 seconds]
<jtojnar> I already have prior engagement for today’s evening, will definitely try to come for the next one
<gchristensen> no worries :)
<gchristensen> is it safe to assume /dev/disk/by-*/* will be symlinks to /dev/sdx[y]?
<worldofpeace_> jtojnar: please please for the next one :D we do soo much cool stuff together
ris has quit [Ping timeout: 265 seconds]
<gchristensen> +1!
<aanderse> speaking of cool stuff you do together... how is the pantheon desktop looking on nixos these days? :)
<aanderse> i've never used it but am interested in trying
niksnut has quit [Ping timeout: 240 seconds]
<gchristensen> unionfs is indeed making nixos-install take a long time on my images
<gchristensen> 400% CPU and 10 on iowait, and definitely not on the receiving end :P
das_j has quit [Remote host closed the connection]
Scriptkiddi has quit [Remote host closed the connection]
<worldofpeace_> aanderse: maintained very well by me.
<worldofpeace_> I even contribute upstream
<worldofpeace_> latest work is to support latest mutter api's, instead of using a compat package. https://github.com/elementary/greeter/commit/15c80e458856ba8640f1b52fc4a0adb7b72f44e2 #73906
<{^_^}> https://github.com/NixOS/nixpkgs/pull/73906 (by worldofpeace, 6 days ago, open): Pantheon use latest mutter
<aanderse> great! i look forward to trying it out then :-D
<worldofpeace_> but everything pretty much works, thanks to a lot of help from jtojnar.
<ashkitten> zzA
<gchristensen> hehe: init=/nix/store/p5hidl14l3s1z8rvigdjg37zrah9qx2i-nixos-system-install-environment-19.09pre-git/init initrd=initrd console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 console=ttyS1,115200n8 initrd=initrd loglevel=4
<gchristensen> I wonder if this will result in problems
<worldofpeace_> looks in nixos wiki for what you could be doing https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi#Serial_console
<gchristensen> hehe
<gchristensen> I'm building one big netboot image which contains all the hardware support for a bunch of hardware classes by including their hardware.nix, which each individually set that console option
<worldofpeace_> lol, so it's only duplicated a few times. no sweat 😅
Scriptkiddi has joined #nixos-dev
das_j has joined #nixos-dev
<gchristensen> :P
<gchristensen> doesn't seem to cause trouble, so I'll just go for it
<samueldr> gchristensen: the log will output only on the last (or first?) console
<gchristensen> right on
<gchristensen> perfect :)
<{^_^}> #42255 (by dezgeg, 1 year ago, open): WIP: Log stage-{1,2} output to secondary consoles
orivej has joined #nixos-dev
worldofpeace_ has quit [Quit: worldofpeace_]
orivej has quit [Ping timeout: 250 seconds]
justanotheruser has quit [Ping timeout: 245 seconds]
orivej has joined #nixos-dev
justanotheruser has joined #nixos-dev
tilpner has quit [Quit: tilpner]
Jackneill has joined #nixos-dev
tilpner has joined #nixos-dev
niksnut has joined #nixos-dev
justanotheruser has quit [Ping timeout: 276 seconds]
FRidh has quit [Ping timeout: 240 seconds]
marek has quit [Ping timeout: 240 seconds]
ckauhaus has joined #nixos-dev
psyanticy has joined #nixos-dev
FRidh has joined #nixos-dev
FRidh has quit [Remote host closed the connection]
FRidh has joined #nixos-dev
justanotheruser has joined #nixos-dev
__monty__ has joined #nixos-dev
<domenkozar[m]> I think I can no longer recommend nixops
<domenkozar[m]> it's just broken software
<domenkozar[m]> supporting a few people shows that it's too fragile
<domenkozar[m]> and since the split that's going to get just worse
<adisbladis> domenkozar[m]: I've been dreaming of a nixops replacement that leverages terraform
<domenkozar[m]> polyrepo was the nail in the coffin
<domenkozar[m]> you take undermaintained and make it polyrepo
<domenkozar[m]> it's a recipe for disaster
<domenkozar[m]> software*
<domenkozar[m]> @adisbladis: yup :)
<adisbladis> We simply don't have the amount of people to support all the services out there
<domenkozar[m]> I think that's fine, the problem is we pretend that's not the case
<domenkozar[m]> too much complexity
<domenkozar[m]> that's why terraform is successful regardless of their resources
<clever> domenkozar[m]: what parts of nixops have you found are fragile?
orivej has quit [Ping timeout: 268 seconds]
<domenkozar[m]> it took like 5 fixes before hetzner would boot
<domenkozar[m]> beforehand it was at least nixpkgs<->nixops that was hard to keep in sync, across all nixpkgs branches
<domenkozar[m]> now that's so much worse
<domenkozar[m]> the underlying issues is that neither python nor nix have a good story for breaking changes
<clever> domenkozar[m]: i think a lot of that, is to blame on the hetzner bootstrap process, that tries to run nixos-install on ubuntu
<clever> https://github.com/NixOS/nixops/issues/1189 aims to entirely skip that problem, by just booting nixos via kexec
<{^_^}> nixops#1189 (by cleverca22, 15 weeks ago, open): plan for supporting custom partition layouts and custom FS's on any backend
<clever> then you nixos-install like normal, from a proper nixos host
orivej has joined #nixos-dev
<domenkozar[m]> clever: that's one area of the issues
phreedom_ has quit [Ping timeout: 260 seconds]
phreedom has joined #nixos-dev
orivej has quit [Ping timeout: 240 seconds]
das_j has quit [Remote host closed the connection]
Scriptkiddi has quit [Remote host closed the connection]
das_j has joined #nixos-dev
Cale has quit [Remote host closed the connection]
Cale has joined #nixos-dev
__monty__ has quit [Quit: leaving]
<Profpatsch> Okay, before I finally snap, let’s just add runCommandLocal finally https://github.com/NixOS/nixpkgs/pull/74642
<{^_^}> #74642 (by Profpatsch, 49 seconds ago, open): pkgs/build-support/trivial-builders: add runCommandLocal
Scriptkiddi has joined #nixos-dev
<clever> Profpatsch: only problem i can forsee with that, is that it complicates compiling for another platform
<clever> Profpatsch: i cant just realize a given path that hydra built, i am forced to ssh out to a build machine of the right arch
<clever> but thats an edgecase, where i would already be paying that cost for every single drv
<clever> so its a choice between always paying the narinfo lookup cost (remove allowSubstitutes) or sometimes paying the remote-build cost
<Profpatsch> clever: Hm, you mean when allowSubstitutes is set you can’t use it if you don’t have a builder for the arch
<Profpatsch> clever: If it’s already in the store, there is no substitution anyway. If you *could* subsitute it, you have to push on the builder, yes
<Profpatsch> But only if you either don’t build or it’s a different arch
orivej has joined #nixos-dev
<clever> Profpatsch: yeah
FRidh has quit [Quit: Konversation terminated!]
<clever> gchristensen: sure
<gchristensen> clever: so I can accept CPR instructions now and partition and mount disks based on the CPR data
<gchristensen> including ZFS on /
drakonis has joined #nixos-dev
<clever> gchristensen: do the machines have 2 or 3 disks?
<gchristensen> clever: one second, I'll get to that
<gchristensen> clever: https://gist.github.com/grahamc/b85725db4f90aa03004583f9bac56225 here are the files from /etc/nixos/packet/ does this look good to you? note: I plan on moving the users.users.root.openssh.authorizedKeys.keys definition to a different file (named credentials.nix or something) so it can be easily ignored
* clever looks
<clever> gchristensen: somewhat related, what is packet.net's policy on repairs when a hdd fails?
<gchristensen> they don't do them
<clever> just report it, destroy the instance, and re-spawn a new one?
<gchristensen> yep
<clever> no problems with by-id then
<clever> id normally prefer by-uuid, so when you clone the disk during repair, it can survive
<clever> but if you cant repair, it doesnt matter
<gchristensen> *SOMETIMES* they will do it, but they make no guarantees it'll work, and they only do it for customers who they know have very high technical competence and know what to do to make it work
<gchristensen> so in this case the by-id is sufficient
<clever> and with hdd swaps, there is always the risk of swapping the wrong device
<gchristensen> yeah, so they really just don't do them
<clever> gchristensen: will these machines always be efi, or are some legacy?
<gchristensen> almost all of them will be MBR
<gchristensen> c2.medium.x86 and their ARM machines are EFI, the rest are MBR
<clever> lrwxrwxrwx 1 root root 9 Nov 25 08:31 /dev/disk/by-id/wwn-0x5000cca33ad2b4c7 -> ../../sda
<clever> gchristensen: youll want a symlink like this in boot.loader.grub.devices, when on legacy
<gchristensen> yep, I'll get to that part :)
<clever> and if you have redundancy, i think you can use 2, but ive never tested that part of grub
<gchristensen> yep
<samueldr> (pedantic samueldr alert!) Legacy Boot*; MBR/GPT is the partitioning scheme, and both are likely to work on somewhat recent implementations of both EFI/Legacy boot firmwares :)
<gchristensen> with the exception of the authorizedKeys part, does this metadata layout for this system look good?
<clever> still reading over it
<clever> samueldr: yeah, ive seen efi on mbr work with the rpi&tianocore
<clever> and i often do legacy on gpt
<samueldr> the most important bit of that distinction is, rightly so, booting Legacy Boot GRUB on GPT; if you misuse "MBR boot" then it'll be assumed you don't need the BIOS Boot partition!
<clever> gchristensen: in slug.nix, i would rather you use mkMerge, that will give better error messages
<gchristensen> oh cool, can you show me an example of that?
<clever> gchristensen: if you want legacy on gpt, you must create a bios boot partition, no fs, not formated, not mounted, 1mb in size
<clever> gchristensen: when you boot.loader.grub.devices = "/dev/sda", grub will search for the bios boot partition, and automatically use it
<samueldr> (or was it mkMerge that he needed an example of?)
<gchristensen> clever: please focus on just this metadata for this one instance :)
<gchristensen> we'll get to MBR later
<clever> ah
<clever> gchristensen: { config = mkMerge [ { services.foo = ...; } { services.bar = ...; } ]; }
<gchristensen> cool
<clever> its pretty much identical to how your doing it with imports
<gchristensen> perfect
<clever> i think the module system will translate the above, into { config.services = mkMerge ( { foo = ...; } { bar = ...; } ); }
<clever> which lets lazyness figure out what config.environment is, and figure out services later
<clever> and it will recursively push it down, while keeping track of which file to blame for errors
<gchristensen> sounds good
<gchristensen> other than the mkMerge and moving the users.users.root.openssh.authorizedKeys.keys bit out of this file, looks good?
<clever> looking over it more...
<clever> nixpkgs.config.allowUnfree = true?
<gchristensen> needed for hardware.enableAllFirmware = true;
<clever> gchristensen: and why does it need all firmware?
<gchristensen> they may not, I'll make a note to look in to that
<clever> the rest of that gist all looks good
<gchristensen> cool, and matches your needs for automation?
<clever> gchristensen: i think so, the only thing missing is the choices for pool layout, which i think would be in the CPR data?
<gchristensen> right, we're not talking about that yet :)
<gchristensen> so for a MBR sernver I have this in the configuration: boot.loader.grub.devices = [ "/dev/disk/by-id/wwn-0x55cd2e414fbc4ee2-part1" ]; where part1 starts at sector 2048 and goes until sector 4096 (1M) and is the "MBR partition scheme" type. however, grub-install tells me: grub-install: error: unable to identify a filesystem in hostdisk//dev/sda; safety check can't be performed ... any thoughts?
<clever> gchristensen: nope, you must never point that to a partition
<gchristensen> ahh
<clever> it must be pointed to the root device, like sda or sdb
<clever> for MBR, grub will install a stub to sector 0, and then a stage 1.5 at sector 1024 i think (between sector 0 and partition 0)
<clever> for GPT, the tables get in the way, and they wanted to ban using "unused" space :P
<clever> so you must create a bios boot partition, and grub will lookup the offset on its own
<clever> in both cases, you point grub to sda, a stub goes into sector 0, and the stage 1.5 goes "somewhere else"
<clever> gpt just clearly defines that "somewhere else" as in use
<gchristensen> perfect
<gchristensen> this makes my code nicer too
<clever> the LBA offset of "somewhere else" is baked into the sector0 code, so it can be found without supporting the partition tables
<clever> and stage1.5 includes the grub kernel, and the grub modules required to mount /boot/
<clever> https://github.com/arvidjaar/bootinfoscript will parse all of those blobs, and explain the steps, as a debug tool
ckauhaus has quit [Quit: WeeChat 2.6]
<clever> sda here is an MBR disk, a stub is in sector 0, and stage 1.5 is directly after it at sector 1!! (my 1024 before was wrong)
<clever> stage 1.5 includes fshelp, ext2, part_msdos, and biosdisk, along with a string saying to mount (,msdos2)/grub (the 2nd partition on the same device as the current MBR)
<clever> by omiting the device, that drive can change to sdb or sdc, and still find the 2nd partition
<clever> if your doing devices = [ "sda" "sdb" ];, then grub will include a device, so the MBR on both disks, will look for a /boot on one of them
<clever> which creates a weak link in the redundancy
<clever> i'm not sure how to setup fully raid'd /boot, but bootinfoscript will help confirm if you did it right
<clever> the sdb in my above gist, is a gpt disk
<clever> Device Start End Sectors Size Type
<clever> /dev/sdb1 2048 67583 65536 32M BIOS boot
<clever> so now its using the offset defined in the partition tables (and i made that way too big)
<clever> that grub is not in use anymore, and it expects /boot/grub to be a directory on a zfs dataset called /root
<clever> (i dont remember even doing that? lol
<clever> ahh, sdb is from my old ssd+ssd mirror
drakonis has quit [Quit: WeeChat 2.6]
drakonis has joined #nixos-dev
drakonis has quit [Client Quit]
<clever> gchristensen: as for the specifics of the CPR data, it depends on how many disks are present, and choices the user will want to make
<clever> which will also impact the legacy grub config
<clever> oh, you can also do legacy+efi on gpt, for extra redundancy
<gchristensen> yeah, I think I accounted for the number of disks and a large number of choices
<clever> ive not tested it, but i suspect raid'ing and efi might also work, with some extra effort
orivej has quit [Ping timeout: 240 seconds]
<clever> some specific choices i'm thinking of
<gchristensen> I don't support that
<clever> 1 disk, no choice, just bpp+boot+swap+zfs
<gchristensen> https://gist.github.com/grahamc/52e80664eb5774f928f36c625c7370bc here is the CPR data for the c2.medium.x86 which had the .nix files I showed you
<clever> 2 disks, you can choose between redundancy or capacity+speed
drakonis has joined #nixos-dev
<clever> for zfs, your choices are mirror or not (just zpool create /dev/sda /dev/sdb)
<clever> for swap, you can either mdadm 2 partitions together (the system should survive a failure while on)
<clever> or swap could be 2 partitions, with the same priority set (it will crash upon loosing a disk, but can still boot)
<clever> 40 #{ device = "/dev/disk/by-partlabel/swap1"; priority = 10; }
<clever> 41 #{ device = "/dev/disk/by-partlabel/swap2"; priority = 10; }
<clever> this tells linux to stripe all swap activity between the 2 disks, so you get double the write rates
<gchristensen> neat
<clever> gchristensen: line 11 of that CPR data, is that /boot or bios-boot?
<gchristensen> "BIOS" in this case doesn't mean anything specific
<gchristensen> it is just what upstream does for whatever reason
<gchristensen> but this one is EFI, not MBR
<gchristensen> and you can correlate that partition number to https://gist.github.com/grahamc/52e80664eb5774f928f36c625c7370bc#file-cpr-json-L29-L31 to see it is /boot
<clever> oh, those are labels, not types
<clever> for efi, you have 2 more choices
<clever> choice a: /boot is vfat, it must be tagged as the efi system partition in gpt
<gchristensen> right
<clever> choice b: /boot can be anything (for zfs, id recomend an ext4 partition), and then /boot/ESP is the ESP and vfat
<clever> oops, thats /boot/EFI
<clever> boot.loader.efi.efiSysMountPoint = "/boot/EFI"; is required, to make choice b work
<gchristensen> how about /boot/efi
<clever> that also works
<clever> the path doesnt really matter
<clever> choice b also allows /boot to be on the same partition as /, but zfs complicates that
<gchristensen> I support either /boot or /boot/efi being the ESP partition
<clever> the benefit of choice b, is that only the efi binaries live on the vfat
<clever> the kernels and initrds can then live on ext4
<clever> so vfat can be much smaller, and it rarely changes, so you dont have to worry about vfat lacking a journal
<clever> another option, that i think some nixos users stumble into by accident, is a combination efi+legacy install
<clever> if you have both a bios-boot partion, and an efi-system partition, and you set both boot.loader.grub.efiSupport and boot.loader.grub.devices = [ "/dev/sda" ];
<clever> then grub installs both efi and legacy binaries to /boot, and configures both .efi executables and the MBR
<clever> then if efi ever borks, you can just legacy boot the same disk
<clever> but i dont know if packet.net lets you force legacy on an efi machine?
<clever> legacy is also x86 specific
<clever> if you have 3 disks, then your choices for zfs open up more
<gchristensen> here is an MBR boot device with the .nix files and the CPR data: https://gist.github.com/grahamc/ae368d5d159c0dda05ee065c01288fd5
<clever> you can either go with no redundancy, mirror, or raidzN
<clever> gchristensen: ext4, gpt? all looks good
<clever> i noticed a minor issue with the vdevs on the other one...
<clever> gchristensen: is this raid? mirror? jbod?
<gchristensen> "disk" means "stripe"
<clever> ah
<clever> and zfs doesnt really let you do concat, it will just stripe if you dont specify a type
<gchristensen> you can add any of the vdev types describen in `zpool` under "Virtual Devices (vdevs)"
<gchristensen> in man zpool*
<clever> i also notice, your using sda for some, and sdd3 for others
<clever> that has a minor impact on performance
<gchristensen> I don't care, that is just a demo
<gchristensen> it is up to you to put in sensible data
<clever> ive yet to measure how much though
<clever> yeah, this allows the user to choose those things
<clever> only other thing i can think of thats missing, is device and pool attributes
<gchristensen> it isn't :)
<clever> ah, that gets pool properties, but what about dataset properties?
<clever> oh there
<clever> duh!
<clever> that just leaves one last thing then, lol
<gchristensen> ok
<clever> sometimes, i will install with dedup=on, for the entire pool
<clever> but once installed, i switch to dedup=off
<gchristensen> hehe
<clever> that will reduce the disk usage of the initial install, but eliminate the performance cost of running with dedup
<clever> the only cost remaining, is when deleting blocks created during the install
<gchristensen> I'm not going to support changing options after install for now
<gchristensen> you can do that when you first SSH in :)
<clever> yeah, nixops could always do that step
<gchristensen> ok, here is what I need from you now
<clever> the property fields let nixops specify initial values, and once created, nixops can change to secondary ones
<gchristensen> I need a CPR document which describes a system you'd actually use
<gchristensen> to be certain it actually works
* clever edits
<clever> i'll write one for some dual-disk systems
<gchristensen> https://gist.github.com/grahamc/52e80664eb5774f928f36c625c7370bc#file-cpr-json-L57 this one is from a c2.medium.x86 which I believe is the system you're wanting to use?
<clever> already using that one as the example
<gchristensen> cool
<clever> oh, what about discard stuff
<gchristensen> what about it?
<clever> i'm thinking, an option in the json, to make blkdiscard get ran on every disk, before it partitions
<gchristensen> it does
<gchristensen> every disk is completely discarded by the time my stuff runs
<clever> perfect
<clever> gchristensen: any particular reason you used sdd for the boot disk?
<gchristensen> that is what Packet does out of th ebox
<clever> ah
<gchristensen> clever: the rules around boot devices are: if EFI: there must be single vfat filesystem mounted at /boot or /boot/efi and it must not be RAID-backed, ... if MBR: at least one disk must have a partiton with the label "BIOS", and every disk which has a partition labeled "BIOS" is added as a grub device
<gchristensen> that is, those are the rules my tooling checks for
<clever> ahh
<clever> still writing...
<gchristensen> cool
<clever> its going to be over-engineered as heck :P
<gchristensen> oh dear
<clever> gchristensen: disks array is done, on to filesystems...
<gchristensen> :)
<edef> do we have a policy for external completions addon things?
<edef> like, i'm packaging the gcloud completions for fish
<clever> gchristensen: do we know the block size for these drives?
<gchristensen> they're probably enterprise SSDs
ris has joined #nixos-dev
<gchristensen> that is all I know though :)
justanotheruser has quit [Ping timeout: 245 seconds]
<clever> gchristensen: `fdisk -l /dev/sda` can reveal that
<clever> Sector size (logical/physical): 512 bytes / 512 bytes
<clever> I/O size (minimum/optimal): 512 bytes / 512 bytes
<edef> that's always 512 bytes
<edef> or 4k
<gchristensen> I think fdisk will always report 512
<edef> my NVMe SSD shows 512 bytes
<gchristensen> exactly, disks usually lie
<clever> only because crappy OS's fail hard when the disk tells the truth
<edef> i'm pretty sure that my other samsung SSDs show the same value
<clever> Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
<edef> even though the truth is a multiple of 3
<clever> Sector size (logical/physical): 512 bytes / 4096 bytes
<clever> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
<clever> one of my drives tells the truth!
<edef> yes, that's 4k because it's AF
<clever> AF?
<edef> but the only options are 512 and 4k
<edef> Advanced Format
<clever> ah
<clever> nothing is ever bigger then 4k?
<edef> not unless you have very specialised drives
<clever> Profpatsch: and now i need runCommandLocal, lol!
<gchristensen> clever: I am concernedabout what you're creating :P
justanotheruser has joined #nixos-dev
<gchristensen> btw CPR is available when you have reserved the instance, so you would know if you drive was 4k or not because you can craft the CPR data for that host
<clever> gchristensen: datasets are now defined
<clever> gchristensen: but i might need to boot something on the instance to inspect it? and then re-run CPR against it afterwards
<clever> gchristensen: though, your tool could support this specific flag on its own
<gchristensen> you would need to inspect the host to learn about it yes
<gchristensen> that is why CPR is only available on reserved hw :)
<clever> ashift, must be set to N, where the block size is 2^N
<clever> ahh
<clever> i was thinking you could just restrict cpr to fit within a template
<clever> i want a machine with 2 disks and _ cpu, here is a CPR template that would work on any 2 disk machine
<gchristensen> yea
<gchristensen> they just don't do that right now
<clever> cpr_zfs.pools and .datasets are done
<clever> i think that just leaves mounts
<clever> gchristensen: i think its done?, giving it a read-only
<clever> over*
<gchristensen> cool
<clever> currently, its efi only
<clever> and swap could be handled differently, but thats an experiment for later
<clever> hows that look?
<gchristensen> clever: property values must all be strings
<ma27[m]> is the `-unstable-` infix supposed to be part of `pname` or `version`? (IIRC this has been mentioned in nixos office hours some time ago and it was suggested to do `pname = "pkg-unstable";`, but I'm not 100% sure anymore)
<clever> gchristensen: gist updated
<gchristensen> clever: the very top of the file should be { "customdata": { .... and then this line should be deleted https://gist.github.com/cleverca22/43f098096007c332a3f1409a8b38bafa#file-build1-json-L62
<clever> $ jq < build1.json '.customdata.cpr_storage.disks[0].device'
<clever> "/dev/sdd"
<clever> gchristensen: thats just the sorting of the fields
<clever> let me double-check against yours...
<clever> $ jq < 52e80664eb5774f928f36c625c7370bc/cpr.json '.customdata.cpr_storage.disks[0].device'
<clever> "/dev/sdd"
<clever> gchristensen: yep, the customdata is at the right point in mine
<gchristensen> no it isn't :P
<clever> it is at the root of the json object
<gchristensen> { customdata = { cpr_zfs = ...; cpr_store = ...; } }
<clever> oh, cpr_zfs is at the wrong point?
* clever compares
<clever> i see
<clever> cpr_zfs is a child of customdata, not a sibling
<gchristensen> yep
<clever> the indenting was hard to follow in your json
<gchristensen> aye
<gchristensen> let me know when it is updated and I'll try this
<clever> min-free strikes again
<clever> gchristensen: gist updated
<clever> gchristensen: this also defines 2 json files, for raidz1 and disk arrangements
<gchristensen> yeah you could do raidz1 in there
<gchristensen> the format is { "vdevtype": [ "disk", "disk", "disk" ] }
<clever> hydra is using raidz1 so the database isnt lost upon failure
<clever> while the build machines, who cares
<gchristensen> and it accepts a list of these vdevtype objects
<gchristensen> ah dang clever you can't mkMerge a function
<clever> the function args should be one level up
drakonis has quit [Ping timeout: 265 seconds]
<gchristensen> can't
<clever> and passed down as normal vars
<gchristensen> that slug.nix file is generated from other files
<clever> just leave it as imports then for now, and it can be delt with better at a later time
<gchristensen> yep
<edef> clever: note that there is ashift autodetection
<edef> clever: it hardcodes some device IDs
<clever> edef: ive not heard of that, ahh, so its not as smart as querying the disk directly
<edef> clever: disks lie
<clever> yeah
<clever> i would both query, and hard-code a list of overrides
<clever> for the liers
<clever> i also see some 8k's in that list!
<edef> i don't think they report that
<edef> the 860s definitely have page sizes that are multiples of 3
drakonis has joined #nixos-dev
psyanticy has quit [Quit: Connection closed for inactivity]
drakonis has quit [Ping timeout: 265 seconds]
<niksnut> gchristensen: hm, looks like some build machines have the wrong time: https://nix-cache.s3.amazonaws.com/log/f45iziab6xhwfq9c7qphmx422y6hlkkr-nix-2.4pre7116_f102d793.drv
<gchristensen> how do I view that?
justanotheruser has quit [Ping timeout: 246 seconds]
<gchristensen> niksnut: what makes you think so?
<gchristensen> I can't tell
MichaelRaskin has joined #nixos-dev
<niksnut> sorry, I restarted the build so the log is gone
<gchristensen> I did manage to see the log
<gchristensen> but I didn't see anything susipcious
<gchristensen> clever: ping
<clever> gchristensen: pong
<gchristensen> clever: ssh -o UserKnownHostsFile=/dev/null root@147.75.75.222
<gchristensen> please validate the config and disks are right
<clever> gchristensen: almost done...
<clever> efi vars are a little odd, but it boots so it doesnt matter much
<gchristensen> uh oh
<gchristensen> the aarch64 build box is a bit busted
<gchristensen> it is asking for a password at boot time :o
<samueldr> uh oh!
<clever> gchristensen: minor nit-pick, boot-phone-home persists in the config after installation
<gchristensen> yep
<gchristensen> I know
<clever> gchristensen: you could delete that file (and remove it from packet.nix) after you nixos-install
<clever> so it will apply on bootup, but not persist after a nixos-rebuild
<gchristensen> it does that by default on the installatino
<gchristensen> but this one I had to hand fix becaus eof the lib thing
<clever> ah
<gchristensen> so don't worry about this one :P
<clever> slug.nix also doenst contain any functions, this time
<gchristensen> I know
<gchristensen> I fixed it by hand
<clever> ahh
<clever> gchristensen: everything looks perfect for a build-machine (builder1.json)
<gchristensen> perfect
<clever> gchristensen: oh, i just noticed, the disks are mis-matched in sizes
<clever> 447 + 447 + 223 + 223
<gchristensen> yep
<clever> that will cause problems with the raidz1 file
<clever> mostly wasted space
<gchristensen> you'll need to account for that, yeah
<gchristensen> could do 2 mirrors
<clever> raidz1 is pointless if you dont have 3 disks of equal size
<clever> so 2 mirrors would be better, for that hardware
<clever> how predictable are factors like disk size and number of disks?
<gchristensen> since it is on reserved hardware, 100% ;)
<gchristensen> I can't say any more details than that, using CPr on unreserved hardware is 100% unsupported
<clever> more about what i'll get when asking for a reserved box for example?
<clever> or would i just say what i want on the reserved box?
<gchristensen> you'll get at least what is here: https://www.packet.com/cloud/servers/c2-medium-epyc/
<clever> ah, they are calling it "boot" and "storage" drives
<gchristensen> yeah
<clever> i tend to just lump everything into a single pool
<clever> so mis-matched sizes like that mess up my plans
<clever> checking the disks on this box closer..
justanotheruser has joined #nixos-dev
<clever> gchristensen: feel free to terminate that box whenever you want
thonkpod has joined #nixos-dev
<gchristensen> thanks
<clever> gchristensen: one solution i can think of for slug.nix, is to bring hnix into the mix
<clever> it may be a simple matter to parse a list of nixos modules, merge them together into one, and spit it back out
<clever> then it doesnt even need mkMerge
<clever> it would primarily target simple { ... }: { <stuff> } things, and not allow let blocks
<gchristensen> that would be fine
<gchristensen> these are all calculated at build time, not during the installation -- so adding hnix wouldn't impact the install environment at all
<clever> the tool could also be a fairly small binary, and kept in a binary cache
<gchristensen> adisbladis and I were wondering why mksquashfs is *so slow* earlier
<gchristensen> somebody make a mksquashfs but fast :P
<adisbladis> If I was the maintainer I'd just make it fast
<gchristensen> good call
<MichaelRaskin> Just make it fast means that generated images are allowed to be incorrect, right?
<gchristensen> I'm pretty sure mksquashfs is slower than it needs to be
<MichaelRaskin> Well, you said _but_ fast, I was replying to _just_ fast
<MichaelRaskin> (not sure what compression level squashfs achieves, maybe compression level it uses is actually expensive)
<clever> gchristensen: mksquashfs is very good at using many cores, when compressing
<clever> 8 comp ? "xz -Xdict-size 100%"
<clever> 25 mksquashfs nix-path-registration $(cat $closureInfo/store-paths) $out \
<clever> 26 -keep-as-directory -all-root -b 1048576 -comp ${comp}
<gchristensen> yeah Filesystem size 472322.35 Kbytes (461.25 Mbytes)
<gchristensen> 31.60% of uncompressed filesystem size (1494488.69 Kbytes)
<gchristensen> but even with 96 cores it takes several minutes
<clever> 7 # For zstd compression you can use "zstd -Xcompression-level 6".
<clever> select COMPRESSION compression. Compressors available: gzip (default), lzma (no kernel support), lzo, lz4 and xz.
<clever> -b BLOCK_SIZE
<clever> Use DICT_SIZE as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc.
<clever> gchristensen: interesting, there is an `-Xbcj arm` "filter" that does .... something
<clever> gchristensen: oh god no, dont use bcj on anything nix is involved in! lol
<clever> gchristensen: it mutates assembly code to make it more compression friendly!
<gchristensen> nonono
<clever> say goodbye to all your hashes and validation :P
janneke_ has joined #nixos-dev
janneke has quit [Ping timeout: 240 seconds]
<clever> gchristensen: so, you can choose between xz, lz4, zstd, lzo, lzma, and gzip...
<clever> gchristensen: ive also only used mksquashfs on machines of up to 8 cores
xvapx has joined #nixos-dev
xvapx has quit [Client Quit]