phreedom_ has quit [Remote host closed the connection]
phreedom has joined #nixos-dev
ma27 has joined #nixos-dev
ma27 has quit [Ping timeout: 256 seconds]
ma27 has joined #nixos-dev
MichaelRaskin has joined #nixos-dev
ma27 has quit [Ping timeout: 256 seconds]
ma27 has joined #nixos-dev
ma27 has quit [Ping timeout: 240 seconds]
__Sander__ has joined #nixos-dev
ckauhaus has joined #nixos-dev
jtojnar has quit [Remote host closed the connection]
xeji has joined #nixos-dev
zybell_ has joined #nixos-dev
ma27 has joined #nixos-dev
ma27 has quit [Ping timeout: 240 seconds]
orivej has joined #nixos-dev
xeji has quit [Ping timeout: 276 seconds]
ma27 has joined #nixos-dev
<WilliButz>
does anyone know how exactly 'Parallel building is broken in OpenSSL' according to 'e5497ca0434e45fef3750213c26922987f53f641'? I see that core utilization is just as bad as when parallel building is disabled for 1.0 but it builds fine, even when I build it with --check. For openssl 1.1 the core utilization is fine and build duration decreased from 'real 2m54.008s' to 'real 1m18.176s' on a machine with 16
<WilliButz>
hyperthreads.
<WilliButz>
niksnut: /cc ^
<dtz>
So is there a history behind Triton[1] regarding why forking was necessary or desired? Would be nice to work together and take advantage of that effort!
<copumpkin>
general disagreements of how the project should work
<copumpkin>
they wanted lots of knobs, most of us didn't, they didn't want to have to think about macOS, we did, they wanted to rearrange all the package layout, we didn't, etc.
<copumpkin>
I think it started around deeply modeling the set of ./configure options on lots of packages
<copumpkin>
where lots of us felt it was overkill and implied a false amount of variability to expose several dozen knobs on each pacakge, even though realistically only one (or maybe two) combination of them would ever be tested
<srhb>
NixtooOS
<copumpkin>
niksnut: did you see the M5 volume resizing PR?
<dtz>
Tyvm copumpkin
<infinisil>
copumpkin: But tbh, the package layout in nixpkgs is horrible..
<copumpkin>
I agree
<dtz>
WilliButz: since the commit is 5 years old that seems worth revisiting
<copumpkin>
the question is whether it's worth the pain to rearrange it
<copumpkin>
and I'm not sure I like the triton one either
<dtz>
Especially since there is no linked issue and that wasn't even the purpose of that commit
<srhb>
That's not a good way to fish for help. :-)
* dtz
tries to help re:that issue, we'll see O:)
<dtz>
it does seem he doesn't get Nix way of doing things, and that issue has too much sass about that IMO
<dtz>
other projects might close an issue outright for taking such a tone
<dtz>
"accept your software is wrong/bad" generally doesn't go very well
<infinisil>
Regarding the pkgs structure, how about using directories like `pkgs/{aa,ab,ac,...}` for the starting letters of the package name (and potentially another level). Then to get the lazyness we use a generator that produces a default.nix similar to the current all-packages.nix from all of these directories
<infinisil>
Seems like that could work nicely
ckauhaus has quit [Quit: Leaving.]
<copumpkin>
niksnut: I'm gonna merge https://github.com/NixOS/nixpkgs/pull/39164 unless you object :) I do think it should be backported to 18.03 and that we should make new AMIs, but will defer to you if you disagree
<dtz>
builtins.autoTopLevel
<dtz>
lol
<niksnut>
copumpkin: sure
* dtz
pours some friendliness and attention on those issues and hopes for the best
<niksnut>
it probably doesn't solve the whole problem the c5/m5 instances though
<copumpkin>
what's left?
<copumpkin>
he tested the AMIs and got resizing behavior
<copumpkin>
and I thought we already had the various ena/nvme drivers from earlier commits
<aminechikhaoui>
but it's more of a NixOps issue I guess
<copumpkin>
oh okay
<copumpkin>
yeah, I don't use nixops at all, so don't really know that stuff very well
<aminechikhaoui>
copumpkin: basically nixops waits for devices such as /dev/xvdf etc and doesn't work with nvme devices names
<aminechikhaoui>
but now that I look at it I don't think it's a blocker for AMI generation
<copumpkin>
yeah
<dtz>
someone asked me what command would install their software on NixOS, since they want to put it on their website (in response to me letting them know I'd packaged it). Thoughts on how to best answer this? I'm trying not to drag in too much "well, depending on what channel(s) you use..."
<infinisil>
Well I'd just say `nix-env -i rw`, the people that know Nix/NixOS would probably know themselves that -A would make it faster and that it might not be available on their channel
<dtz>
I wanted to recommend `nix-env -iA nixos.rw` or `nix-env -iA nixpkgs.rw` but... only one of those works and it depends on if using NixOS, I think?
infinisil has quit [Quit: Configuring ZNC, sorry for the join/quits!]
<dtz>
well but the sort of user that might just copy-paste a command will find that doesn't work--since it's not in 18.03 (although it's "harmless" so maybe it could be but this seems like a strange motivation lol)
<dtz>
hmm okay
<dtz>
i forgot about not using '-A', not even kidding xD
infinisil has joined #nixos-dev
<dtz>
(I tend to deal with enough same-package-but-built-with-different-compiler-or-something that 'name' matching would be difficult lol)
<niksnut>
copumpkin: yeah, what aminechikhaoui said
<MichaelRaskin>
I think the two strange issues in question are about installing with a writeable store
<niksnut>
I think there was also something with udev rules
<niksnut>
the official amazon image has some udev script for doing xvd <-> nvme renaming
<niksnut>
unfortunately it was written in python and we don't want to pull that into the base system
<copumpkin>
ah, never noticed that
<copumpkin>
mind recording details of that in a ticket so we don't forget it?
<copumpkin>
does it really affect much? is it just so folks don't need to know about the nvme naming scheme?
<niksnut>
well, you have to use sd/xvd in block device mappings
ma27 has quit [Ping timeout: 255 seconds]
<niksnut>
so if you write fileSystems."/foo".device = "/dev/xvdi" then it will hang waiting for /dev/xvdi
<niksnut>
and you can't write device = "/dev/nvm..." because NixOps doesn't recognize it so it won't create the disk
<niksnut>
probably the best way forward is to make nixops perform the nvm -> sd/xvd translation internally
<niksnut>
so you would write device = "/dev/nvm...'
<gchristensen>
hard to know if being consistent with real life., or with aws, is the best way forward here
ma27 has joined #nixos-dev
ma27 has quit [Ping timeout: 265 seconds]
<zybell_>
why not rewriting the rename in a language that is available in the base? Could even be as simple as `grep oldname tablefile`.
<dtz>
grahamc: are you setup to "test" the multi-user installer in vagrant without too much trouble? I'm trying to explore two issues (minor, or at least non-breaking) or opportunities for improvement :). Might happen with single-user installer too.... dunno. Anyway 1)'nix verify --all' claims the Nix installed is "corrupt"-- which can be "repaired" via substituter which presumably has a copy of it anyway :). It'd be nice if we
<dtz>
didn't initialize things to "corrupt", if nothing else to avoid scaring new users who try verify xD
<gchristensen>
whoa
<gchristensen>
so my test phase is `vagrant init debian/jessie` (or comparable) and then just use scp to xfer .tar.bz2 in, then run it
<dtz>
2) "suspicious writable paths"-- I know I saw this plenty previously but I haven't non-NixOS'd in a while so I forget what the deal is with this. Nothing in the store should be writable, right? Perhaps during install we should `chmod -R a-w /nix/store`? haha
<dtz>
gchristensen: hahaha oh okay that sounds like my kind of recipe! :)
<gchristensen>
I'd love an "install grid" thing which spawned lots of VMs of different varieties, tried the installer and a few fitness tests, and reported back a table of results... but *looks at the clock, looks at the todo list*
<dtz>
hahaha yeah for real
<gchristensen>
tbh with something that is good at parallel SSH *prepares to say forbidden words* Ansible might be good at this *cringes*
<Dezgeg>
I think nix-store --verify should be fixed since the change to use closureInfo
<dtz>
is travis diverse enough re:supported platforms that we could use it for that?
<dtz>
(heck, even in a separate repo if needed lol)
<gchristensen>
dtz: it is much, much, much too painful
<dtz>
(not sure but hopefully travis VM's are reasonable stand-in's for what a user'd have)
<gchristensen>
on a moderately busy day, you can wait in line for an hour
<dtz>
hahaha okay
<gchristensen>
I'd rather, say, pay the costs for a handful of AWS instances and not pay the delay
<gchristensen>
or pay the cost for a big Packet server and run VMs on that... but the AWS route would be simpler
<Dezgeg>
I was wondering if there's a way to convert EC2 AMIs to something that runs on QEMU
<Dezgeg>
but probably having that would be counterproductive to amazon's business :P
<gchristensen>
oh oh oh oh we have the runInLinuxVM thing that maybe could work
<niksnut>
Dezgeg: there is nixos/tests/ec2.nix, which runs an EC2 disk image in qemu, simulating the metadata server etc
<Dezgeg>
but can you fetch other distros' AMIs easily somehow?
<gchristensen>
I don't believe you can export an AMI you don't own
<Dezgeg>
I guess most of them host the same images somewhere on their infra anyway
ma27 has joined #nixos-dev
<ciil>
simp_le is broken on master, because 479d38c588c5455cbc2f038c607ebb6e29eb8445 updated certbot/acme to 0.23, but simp_le 0.8 depends on acme 0.22 exactly. only the upcoming simp_le version 0.9 is gonna support 0.23. What's the best action here? Revert the commit, update the simp_le drv to current master commit over there, which has just included the update to acme 0.23, or patch it for the time being?
ma27 has quit [Ping timeout: 246 seconds]
<infinisil>
Would it be possible to implement a `builtins.fromDir` which transforms a directory into an attrset and lazyness? Or are attribute names strict so that's not possible without a lot of changes?
<infinisil>
s/and lazyness/with lazyness
<gchristensen>
O.o
<gchristensen>
tbh I don't see why you couldn't implement that with some fancy functions and a fixed point =)
<shlevy>
infinisil: attrsets are strict in the names
<gchristensen>
each readdir would be strict but the subdirs not?
<infinisil>
I see
<shlevy>
Correct
<gchristensen>
right
<shlevy>
You wouldn't need to add any extra builtins
<infinisil>
niksnut mentioned that if pkgs was done with readdir and autocall, it would do a lot of IO, but I still don't get why, because a directory listing (needed for the attrsets) is rather cheap
<gchristensen>
this sounds like an evil and fun thing
<shlevy>
gchristensen: If you look through the git logs, it used to exist for our idris infra :)
<gchristensen>
﴾͡๏̯͡๏﴿
pie_ has quit [Ping timeout: 240 seconds]
<infinisil>
s/needed for the attrsets/needed for the attrnames
<gchristensen>
infinisil: it seems that if you're looking to replace all-packages.nix, a first step would be making it auto-generatable, which is a big task on its own. is that what you're hinking?
<infinisil>
Nah I think autogen isn't even neede
<infinisil>
d
<infinisil>
Just a single directory with a subdir for every attribute
<infinisil>
and readDir that
<gchristensen>
yes, not _needed_ but not an unreasonable first step
<infinisil>
I don't see why you want to use autogen?
<gchristensen>
the complaint: readdir is too much io. solution: do the same thing, but memoize the readir.
<infinisil>
How is readDir too much IO though?
<gchristensen>
I'm not saying it is
<gchristensen>
to prove the value and feasibility case without letting people be afraid of the io being a problem
<shlevy>
profile, profile, profile :)
<infinisil>
FWIW `time nix-instantiate --eval -E 'builtins.readDir /nix/store'` returns in 0.6 seconds for me
<infinisil>
over 100000 paths
<gchristensen>
my point is not that readdir is a problem. my point is readdir happening at one point or another is the same thing, so eliminate that complaint while you build out the restof it.
<shlevy>
infinisil: compare to something like '(import <nixpkgs> {}) == null'
<shlevy>
(not quite fair since a nixpkgs import is doing a bit more than defining the top-level set, but a good first approximation)
<gchristensen>
+1 I find the transparency of all-packages.nix to be really useful
orivej has quit [Ping timeout: 246 seconds]
<gchristensen>
moving to a world where as-much-as-possible was generated would be cool and could feasible actually make it alphabetically sorted :)
<infinisil>
shlevy: Well nixpkgs would only have about 10000 attributes, and i think my 0.6s time is mostly printing all the paths to stdout. Also importing of nixpkgs isn't done more than once usually anyways
<infinisil>
gchristensen: There wouldn't be much difference with a single directory and a subdirectory for every attribute, it would be even simpler then because people don't need to look up files for attributes in all-packages.nix
<gchristensen>
yeah I disagree
<gchristensen>
right now you can start at all-packages.nix and searching that file get to every package defined in nixpkgs without having to switch between looking at the files referenced and implicitly all the directories around.
<gchristensen>
oh its _one of those_ attributes that isn't mapped to a directory
<infinisil>
gchristensen: tree | less ?
<gchristensen>
I think I've already proved my point
<gchristensen>
IMO the explicitness is a feature
<gchristensen>
there is already enough magic to throw people off
<infinisil>
Considering the amount of weird and undocumented functionality in nixpkgs, I'd think using a directory listing for attribute declaration would be one of the clearer ones
<infinisil>
But yeah I get what you said, it's one more indirection step, I think people will just manage fine with it though
<gchristensen>
and yet a big cost compared to what we have now
<gchristensen>
especially since we could get most of the benefit you offer by simply aggressively evaluating the readdir instead of lazily
<infinisil>
aggressively=recursively?
<gchristensen>
I speak of course of the evaluation optimization where you universally cache the results in all-packages.nix at commit time
<infinisil>
Huh, never heard of that
<gchristensen>
(that is to say, you generate it and stick it in all-packages.nix)
* gchristensen
waits for the stares
<infinisil>
Huh
<infinisil>
You mean to transform `callPackage ./foo {}` into `callPackage ({ stdenv, ...}: mkDerivation { ... }) { }`?
<gchristensen>
no
<infinisil>
Ah you mean to basically do `ls` and transform that into a nix file that callPackage's on every file and output that to all-packages.nix?
<gchristensen>
I mean ./update-all-packages.nix.sh -> does the readdir() thing via `find` and emits myattr = callPackage ./myattr { };`
<infinisil>
I see
<infinisil>
I guess if people don't like the idea of the readDir magic this could be done first
<gchristensen>
it would be a good start
<infinisil>
What to do about packages that are called with `callPackage ./foo { bar = some.thing; }` though?
<infinisil>
I have an idea on how to get rid of those overrides that specify different versions
<infinisil>
I'll think about it some more when I have more free time, but thanks for the discussion gchristensen
__Sander__ has quit [Quit: Konversation terminated!]
<dtz>
on "real" machine, ptmx has similar permissions. hmm.
<Dezgeg>
the permissions are set by the mount options on /dev/pts, I think
<dtz>
anyway might be good to track this sort of thing down and ensure it works as expected as part of sandbox'ing so builds from machines can be trusted even if they're not running NixOS
<dtz>
okay, ty I'll look into that
zybell_ has quit [Ping timeout: 276 seconds]
<Dezgeg>
well, impurities caused by old kernels are inevitable sooner or later
<dtz>
well okay so that at least pretty clearly explains why IOTty might be having problems :D
<gchristensen>
ok so I think I have an editor which makes editing docbook super nice, and everybody will hate it an equal amount b/c it is vscode and not $theirfavoriteeditor
<Dezgeg>
but that's a real popular editor nowadays actually
<aminechikhaoui>
niksnut: :o
<gchristensen>
ah dang I'll have to integrate it with thee ACME editor then ;)
<niksnut>
gchristensen: I think none of those xz problems apply to us
<MichaelRaskin>
Well, it could be a reason to provide an lzipped tarballs as an option.
<MichaelRaskin>
For nix releases etc.
<simpson>
I see .xz stuff in the store occasionally. It'd be nice to at least be *aware* of xz's failings.
<MichaelRaskin>
Well, we don't care as much about fragility and versioning, because we have an external checksum and a recorded xz version that did unpack the file.
<simpson>
Yes, but people don't just unpack tarballs with Nix. Sometimes we pack tarballs too.
<dtz>
(also it appears CONFIG_DEVPTS_MULTIPLE_INSTANCES was removed eventually? https://lwn.net/Articles/689539/ haha; it's set on my centos machine but absent on my NixOS (weeeeee))
<Dezgeg>
get the centos sources and start reading?
<MichaelRaskin>
simpson: for the project caring about it would mean providing lzipped tarballs as the advertised default, which I would support; what upstreams provide is less of an issue because we have extra verification.
<simpson>
MichaelRaskin: Sure. I also mean that we don't just publish official NixOS tarballs, but also there are other folks using nixpkgs for their build tools.
infinisil has quit [Quit: Configuring ZNC, sorry for the join/quits!]
infinisil has joined #nixos-dev
infinisil has quit [Client Quit]
infinisil has joined #nixos-dev
infinisil has quit [Client Quit]
infinisil has joined #nixos-dev
infinisil has quit [Remote host closed the connection]
<dtz>
of course that's only after I was neck-deep in kernel source and git history
<dtz>
what my friend refers to as "source code achaeology" (only he uses fancy ae char lol)
Lisanna has joined #nixos-dev
pie_ has joined #nixos-dev
<gchristensen>
niksnut: do you have a timeline on 2.0.1? some people are asking me about getting the new installer. I'm also thinking I should send a PR to the homepage documenting the `sh -s --daemon / --no-daemon` part
<domenkozar>
heh I managed to squeeze in my fix :P