ekleog changed the topic of #nixos-dev to: NixOS Development (#nixos for questions) | https://hydra.nixos.org/jobset/nixos/trunk-combined https://channels.nix.gsc.io/graph.html https://r13y.com | 18.09 release managers: vcunat and samueldr | https://logs.nix.samueldr.com/nixos-dev
asymmetric has quit [Ping timeout: 258 seconds]
lassulus has quit [Ping timeout: 250 seconds]
yl has joined #nixos-dev
drakonis has quit [Quit: WeeChat 2.3]
yl has quit [Quit: WeeChat 2.2]
yl has joined #nixos-dev
yl has quit [Read error: Connection reset by peer]
orivej has joined #nixos-dev
<samueldr> hmm, nixos/tests/switch-test.nix will always pass even when failing due to using | tee in a non pipefail script :/
<gchristensen> oh jeeeeze
<samueldr> yeah
<gchristensen> samueldr: did you see the patch from Kees?
<samueldr> yes
<samueldr> that's triple-checking it works and waiting on a kernel build to reply
<samueldr> oops
<samueldr> that's why I found out about the switch-to-configuration test being a bit... off
<gchristensen> oooh
<samueldr> they passed while they should have failed :/
<gchristensen> good job on your mail, seems to have rattled the right cages
<samueldr> (always reproduce the failure before testing for success!)
<gchristensen> :)
<gchristensen> I can't stop thinking about this shebang https://gist.githubusercontent.com/dtzWill/5d4324eb3ef9d3c7dfbb0f690768dc91/raw/217d62d8ff88a2579eaf79e5a7c35445044f0be3/biber and how its shebang would have taken 50% of a commodore 64's RAM
<gchristensen> entire NES games took up less space
<gchristensen> pretty cool
<clever> gchristensen: shit like that is why hydra-notify takes 2+ seconds to even start
<clever> gchristensen: buildEnv!!!
yl has joined #nixos-dev
<domenkozar> :D
yl has quit [Quit: WeeChat 2.2]
pie__ has joined #nixos-dev
pie___ has quit [Ping timeout: 245 seconds]
yl has joined #nixos-dev
jtojnar has quit [Quit: jtojnar]
phreedom_ has quit [Remote host closed the connection]
phreedom has joined #nixos-dev
yl has quit [Quit: WeeChat 2.2]
<clever> domenkozar: and now ive run into this once more! https://github.com/commercialhaskell/stack/issues/793#issuecomment-418151204
lassulus has joined #nixos-dev
lassulus has quit [Quit: WeeChat 2.2]
lassulus has joined #nixos-dev
<domenkozar> :D
asymmetric has joined #nixos-dev
__Sander__ has joined #nixos-dev
jtojnar has joined #nixos-dev
<tilpner> I know you want to press shiny merge buttons: #55729 #55450 #53873
<{^_^}> https://github.com/NixOS/nixpkgs/pull/55729 (by tilpner, 13 hours ago, open): groff: Remove indeterminism in manpages
<{^_^}> https://github.com/NixOS/nixpkgs/pull/55450 (by tilpner, 5 days ago, open): fuse-overlayfs: 0.2 -> 0.3
<{^_^}> https://github.com/NixOS/nixpkgs/pull/53873 (by tilpner, 4 weeks ago, open): tahoe-lafs: 1.12.1 -> 1.13.0
FRidh has joined #nixos-dev
init_6 has joined #nixos-dev
<tilpner> Oh hey, it worked :)
<tilpner> gchristensen++
<{^_^}> gchristensen's karma got increased to 75
<tilpner> sphalerite++
<{^_^}> sphalerite's karma got increased to 43
<gchristensen> clever: notify-sent uses a built env?
init_6 has quit []
asymmetric has quit [Ping timeout: 250 seconds]
init_6 has joined #nixos-dev
lassulus has quit [Ping timeout: 268 seconds]
lassulus_ has joined #nixos-dev
lassulus_ is now known as lassulus
<gchristensen> PERL5LIB="/nix/store/7yf3fh95ljf90nnw6cv70dry5jvqin0l-perl-5.28.1/lib/perl5/site_perl:/nix/store/w322q6nfg2gar4h86yh5l3p6vxb8ik1r-perl5.28.1-File-Next-1.16/lib/perl5/site_perl..." in the env-vars of the build
<gchristensen> omg stdenv knows perl
lassulus has quit [Ping timeout: 250 seconds]
jtojnar has quit [Read error: Connection reset by peer]
jtojnar has joined #nixos-dev
lassulus has joined #nixos-dev
<LnL> gchristensen: are you or somebody else still looking at a generic solution?
<gchristensen> I stopped looking at a generic solution and started thinking about lessgeneric solutions
<gchristensen> I think using buildEnv at choice points is a fine choice -- we already do it for many packages with many dependencies
<gchristensen> for example, this is what hydra does https://github.com/NixOS/hydra/blob/master/release.nix#L47
<LnL> the custom python builder I wrote does something similar, but links runtime dependencies to the output instead
<gchristensen> cool, that sounds good
asymmetric has joined #nixos-dev
<gchristensen> samueldr: should we roll back the channels or something?
asymmetric has quit [Remote host closed the connection]
asymmetric has joined #nixos-dev
Profpatsch has quit [Quit: WeeChat 0.4.3]
Profpatsch has joined #nixos-dev
<gchristensen> the channel URLs are being rolled back
<Taneb> If the URLs are being rolled back I think it makes sense to roll back the channels correspondingly
<gchristensen> that does roll back (most) channels already
<gchristensen> but not nixpkgs-channels, which we shouldn't force-push to
init_6 has quit [Ping timeout: 245 seconds]
drakonis has joined #nixos-dev
asymmetric has quit [Quit: Leaving]
<clever> gchristensen: it doesnt, and thats part of the problem
<gchristensen> clever: eh?
<clever> gchristensen: ive strace'd hydra-notify, and a decent chunk of its startup time is searching ~10 different places for every single module it imports
<clever> that buildEnv isnt good enough
<gchristensen> ah
<clever> there needs to be exactly 1 path in the PERL5LIB
<gchristensen> that'll be tough to fix, requires fixing stdenv
<clever> gchristensen: is it the stdenv, or a setup hook, that is doing it?
<clever> gchristensen: its a setup hook, so thats easier to modify, somewhat
<clever> cant use a buildEnv, but you can build the env in $out
<gchristensen> gotcha
<clever> addEnvHooks modifies the stdenv some, and will then run addPerlLibPath for every input, and all propagating inputs (which cant be known at nix time)
<clever> you would need to just set PERL5LIB to something like $out/lib/perl, and then lndir things into it
<clever> would still be a mass-rebuild, i expect a lot of things to depend on perl
grw has quit [Ping timeout: 252 seconds]
__Sander__ has quit [Quit: Konversation terminated!]
drakonis has quit [Quit: WeeChat 2.3]
<samueldr> gchristensen: yeah, I wasn't sure about rolling back, I think we'll need to post-morten the whoe situation and see what should and could have been done?
<gchristensen> I agree
<samueldr> among those I'm thinking we'll maybe need to do basic tests (e.g. switch-to-configuration) for all upstream kernels?
<gchristensen> lots of interesting stuff could come out of such a discussion
<samueldr> (hopefully to never be needed again ever
<gchristensen> let's ask 5-why's and get all rub-magic-salt-on-itey
<gchristensen> "Also, if you have a CODEOWNERS file in your repository, a draft pull request will suppress notifications to those reviewers until it is marked as ready for review."
<samueldr> nice
<gchristensen> its a shame you can't mark as draft for a move-to-staging
infinisil has quit [Quit: Configuring ZNC, sorry for the joins/quits!]
infinisil has joined #nixos-dev
<LnL> so, what's the perl status?
<clever> LnL: i'm thinking of patching the perl setup hook to handle things better
<LnL> yeah same, I can look at it this weekend if nobody else is working on it
<gchristensen> please do
<gchristensen> a kernel upsteram update is the author of the offending patch agrees it should be reverted
<gchristensen> but we should fix perl anyway
<clever> and fixing perl will also improve performance of hydra
<clever> we keep getting a backlog of 2k events in the notification channel
<clever> which delays PR merging
<gchristensen> hydra is already somewhat fixed
<gchristensen> because it only searches 10 places, not 400
<clever> gchristensen: hmmm, i notice hydra is using wrapProgram, rather then the #! line
<clever> so hydra itself cant run into #! problems, just slow module loading
<gchristensen> right
<clever> guessing pkgs.biber
<gchristensen> I dunno
<gchristensen> I guess so
<clever> yeah, the #! on biber is huge
<clever> 30kb, i see what you meant about the c64
<gchristensen> :)
<clever> perlPackages.buildPerlModule
<clever> { column = 3; file = "/home/clever/apps/nixpkgs/pkgs/top-level/perl-packages.nix"; line = 24; }
<clever> 21 buildPerlPackage = callPackage ../development/perl-modules/generic { };
<clever> 38 builder = ./builder.sh;
<clever> not clear yet how the PERL5LIB gets into #! ...
<gchristensen> LnL found that beauty
<gchristensen> "beauty"
<clever> gchristensen: that might be what i was looking for, it patches the files in the source, before the build even began
<gchristensen> yeah
<clever> tricky thing, is that this means my $out/lib/perl5 thing would have to be made before the configurePhase
<gchristensen> yeah
<gchristensen> it is very tricky to fix
<clever> which coudl seriously hamper nix-shell usage
<gchristensen> IMO we shouldn't try to make it 100% better
<clever> but, how does this custom builder even interact with nix-shell?
<gchristensen> just ... bettuer
<LnL> I threw together something for use lib, but making it not append entries when running multiple times is annoying
<samueldr> gchristensen: just a reminder that a syslinux update is ready to be merged for r13y :)
<gchristensen> oh! link?
<samueldr> #55500
<{^_^}> https://github.com/NixOS/nixpkgs/pull/55500 (by samueldr, 4 days ago, open): syslinux: 2015-11-09 -> 2019-02-07
<gchristensen> woot
<gchristensen> a bit boring that most of the patches have gone to staging :P
<clever> gchristensen: it looks like the sed in builder.sh, is just converting the PERL5LIB into a -I list, and adding it to the #!
<gchristensen> yea
<clever> so we just need to shrink the PERL5LIB the setup-hook was building
<gchristensen> true
<clever> error: infinite recursion encountered, at undefined position
<clever> how did i break it that badly? lol
<clever> i think its due to lndir causing a cycle in the stdenv bootstrap
<LnL> you'll probably have to move lndir to an earier stage then
<LnL> you can see what stage breaks by moving up with stdenv.__bootPackages.stdenv...
<clever> and its hidden under xlibs.lndir, will need to carefully check its deps
<gchristensen> I'm not real sure we need to go from the worst possible case to the best possible case
<LnL> also does lndir and buildEnv work together properly, I seem to remember I had some kind of problem
<clever> LnL: buildEnv isnt an issue, due to how propagated inputs work, and not having access to the list at nix time
FRidh has quit [Quit: Konversation terminated!]
<LnL> can't reproduce, maybe it was my first attempt
<LnL> make sure to add -silent tho, otherwise it spams the build log
<clever> +++ b/pkgs/development/interpreters/perl/default.nix
<clever> + propagatedNativeBuildInputs = [ lndir ];
<clever> LnL: thats what i had done, but it caused recursion
<gchristensen> what is it that lndir does which buildEnv doesn't?
<clever> gchristensen: run at bash time
<clever> due to how propagated inputs work, you cant know the entire set at nix time, so you cant generate a buildEnv derivation
<clever> so the merged site_perl must be made at bash time
<LnL> also buildEnv is perl :p
<gchristensen> ouch
<clever> LnL: that reminds me, the nixpkgs buildEnv WAS copied into nix, because nix-env -i requires a buildEnv
<clever> and nix-env needs it to manage nix-channel!
<clever> but, when nix was de-perled, it got rewritten in c++
<LnL> yeah, there's builtin:buildenv IIRC
<clever> and ive been wondering, can nixpkgs take advantage of that now?
<andi-> lndir depends on X11 :/
<andi-> not sure that should go into stdenv then
<LnL> that c++ -> binary would be nice
<clever> andi-: thats what i was fearing when i saw xlibs.lndir
<andi-> it is just a single C file
<andi-> trying to figure out how complicated it would be to port
<clever> 1208 buildInputs = [ xorgproto ];
<clever> 2625 buildInputs = [ libXt ];
<LnL> clever: don't remember if it's 2.0 or 2.1
<clever> lol, why!?
<clever> 1013 buildInputs = [ libICE xorgproto libSM libX11 ];
<clever> lndir depends on ice, session manager, x11, xt .....
<LnL> lol
<clever> but only at build-time, at runtime it only depends on glibc
<andi-> it looks tirvial.. just some msg and quit functions
<gchristensen> ...
<clever> what if i just rewrite it in bash? :P
<gchristensen> please lord
* gchristensen is dying
<gchristensen> what if we dropped all the -I business and just did the `use lib` bits
<clever> `use lib`?
<gchristensen> s/PR/patch/
<clever> ah
<clever> still has the same basic problem, needing either a long-ass search list, or a buildenv/lndir output
<gchristensen> I would rather take this fix in two steps
<gchristensen> 1) stop doing the long shebangs
<gchristensen> 2) make the loading faster
<clever> 1 looks easy, *tries*
<LnL> yeah, that was my plan
<clever> 20 sed -i "$fn" -e "s|^#\!\(.*[ /]perl.*\)$|#\!\1\nuse lib $PERL5LIB|"
<clever> maybe?
<clever> Unknown regexp modifier "/e" at t/bind.t line 2, at end of line
<clever> syntax error at t/bind.t line 2, near "site_perl:"
<clever> oh, i think i forgot a ;
<clever> use lib /nix/store/bsllvmy9vjikanbxwvmf8gbmzw7g50s9-perl-5.28.1/lib/perl5/site_perl:/nix/store/bsllvmy9vjikanbxwvmf8gbmzw7g50s9-perl-5.28.1/lib/perl5/site_perl:/nix/store/bsllvmy9vjikanbxwvmf8gbmzw7g50s9-perl-5.28.1/lib/perl5/site_perl:/nix/store/24mvl35rs6fvmlrghwzll7dcd1c6y0wm-perl5.28.1-gettext-1.07/lib/perl5/site_perl;
<clever> gchristensen: does that look right?
<gchristensen> no
* gchristensen looks for the gist from zimbatm
<LnL> you want use lib ('<path>', '<path>');
<clever> oh, i see
<clever> wasnt reading the concatMapStringsSep close enough
<clever> and then the tricky part is the sep part, in bash...
<LnL> not that bad, check L11
lassulus has quit [Ping timeout: 250 seconds]
<clever> LnL: of which file?
<LnL> my gist
<clever> ah, was mixing that link up with the pr
<samueldr> I thought the tricky part would be map
<samueldr> (set a b c d ; export IFS=x; echo "$*")
<clever> LnL: ah, nice
* samueldr sees it's already how LnL did it
<samueldr> or not
<gchristensen> samueldr: kees's patch was incorporated in to some tree which might get to linux-next which might go in to the releases, I guess?
<gchristensen> did you get that email too?
lassulus has joined #nixos-dev
<samueldr> gchristensen: got it, didn't open yet
<clever> use lib 'foo''bar''baz'; ok, missing a char...
<clever> yeah, missed a :
<clever> gettext builds, letting -j rip...
<gchristensen> woohoo
<LnL> multiline sed is the most annoying part, this thing keeps adding use lib entries if it runs more than once
<LnL> but maybe that's fine
<clever> 154 things to build
<clever> [90 built (1 failed)]
<clever> builder for '/nix/store/cqqja3ipaac0vc0151imh3yd7zsvv48f-perl5.28.1-Test-Simple-1.302141.drv' failed with exit code 2; last 10 log lines:
<clever> Failed 3/228 test programs. 6/2602 subtests failed.
<clever> # Failed test at t/Legacy/fail-like.t line 68.
<clever> # got: '# Failed test at t/Legacy/fail-like.t line 63.
<clever> # expected: '# Failed test at t/Legacy/fail-like.t line 62.
<clever> gchristensen: *facepalm*, line-numbers are baked into the expected failure
<clever> and inserting `use lib ...;\n` is breaking them
<gchristensen> LOL
<clever> [5/10/163 built, 0.0 MiB DL] building perl-5.28.1 (configurePhase): getgrent() found.
xeji has joined #nixos-dev
<clever> gchristensen: biber built with my changes!!
<gchristensen> nice!
<clever> #!/nix/store/n4jc5jkvdv57j8vyzwi9r59y5wwz55j3-perl-5.28.1/bin/perl
<clever> use lib '/nix/store/n4jc5jkvdv57j8vyzwi9r59y5wwz55j3-perl-5.28.1/lib/perl5/site_perl', '/....
<{^_^}> #55786 (by cleverca22, 1 minute ago, open): improve perl shebang lines by switching to `use lib ...;`
<gchristensen> nice!
<LnL> that if tho
<clever> LnL: yeah, thats ugly
<gchristensen> oh, haha, yeah, better to patch TestSimple
<LnL> so q: is use lib necessary during the build if PERL5LIB is set?
<gchristensen> env vars are too limiting for a lib set like bibir has
<clever> LnL: probably not, but this patching of the source, is what persists to the scripts being installed to $out
<clever> so this is what causes the $out files to be patched enough to survive at runtime
<LnL> sure, but that can happen during fixup which is after the tests
<clever> yeah, i could try moving it down more...
<clever> i'll just take the entire while loop and move it down a bit...
<gchristensen> if the tests expect to be able to use the libs the package requires, the tests will fail if it depends upon too many packages
<LnL> also preFixupPhases+=" perlFixupPhase" is much nicer than the way it replaces preConfigure
<emily> can't you just put the use lib on the same line as the next line
<clever> that reminds me, of the time i tried to apply a phase based override to rp-pppoe
<emily> rather than having an extra newline
<emily> as a cheap and simple fix
<clever> then i discovered, rp-pppoe, isnt even stdenv
<clever> emily: more difficult with sed, would maybe need ed, to insert something at the start of line 2
<clever> gchristensen: 13 pages have passed, so this may work...
<ekleog> isn't it just 2s/^/foo/
<emily> i feel like slightly more advanced text manipulation is perhaps still simpler than patching a test library to offset all the line numbers by one
<gchristensen> LnL: what am I missing? we know the env var isn't sufficient :/
<ekleog> after testing: yes, `sed '2s/^/foo'` inserts `foo` at the start of line 2
<clever> emily: patching it after testing is simpler still
<ekleog> erhm, `sed '2s/^/foo/'`
<tilpner> sed '2ifoo' does too
<clever> i think inserting it into the start of line 2 is worse, because you then have 30kb of "use lib" followed by 1 "line" of code youll never notice
<clever> and that could make reading the script imposible
<ekleog> tilpner: TIL the `i` command :)
<LnL> gchristensen: we do?
<ekleog> clever: add an inline comment that reminds the reader that there's something after all the `use lib`
<gchristensen> yes, we have cases where the value of the env var would be much longer than the maximum value of an env var
* ekleog doesn't know what the original problem is, just proposes solutions
<gchristensen> like biber's perl5lib value would be way too long
<LnL> -_-
<emily> $code='...contents of first line...'; use lib ...; eval $code
<emily> any questions
<clever> ekleog: if the #! line is over 127 chars, the kernel refuses to execute the script, i think
<samueldr> it's uh, more complicated than that
<gchristensen> emily: first line of code includes 's?
<ekleog> oh we're still on this problem? wasn't Kees OK to have a revert that matches our expectations?
<samueldr> ekleog: related https://lkml.org/lkml/2019/2/13/853
<samueldr> yes
<emily> gchristensen: i think perl supports arbitrary delimiters with qq strings so you can just keep generating 128-bit UUIDs until you find one that doesn't clash (very important to handle that case)
<samueldr> though I think we're on cleanup detail of perl stuff since it'd be a good idea anyway
<gchristensen> samueldr: and for the people who successfully switched to a generatino with these busted kernels :)
<ekleog> right, that's nice :)
<samueldr> ooh, that's effectively nice
<ekleog> then I'll stop pushing for hackish solutions, if it's for a cleanup
<LnL> yeah also, other platforms like darwin have different limits aparently ack has been broken for a long time
<clever> LnL: from what ive heard, the darwin kernel doesnt even support #!?
<clever> LnL: and its handled by the libc?
<LnL> could be, the whole libSystem thing is a bit of a mess
<clever> i heard about that, because #! scripts dont work if ran in certain ways
<samueldr> even within linux it's hard™
<samueldr> e.g. the regression returns NOEXEC, fine, don't execute
drakonis has joined #nixos-dev
<clever> bash does cheat, and will bash scripts that entirely lack a #!
<samueldr> except execve from the libc will try to run the file as a /bin/sh script on noexec
<LnL> not sure if it's still the case but scripts are not allowed, only binaries
<clever> samueldr: i thought that was a bash level feature, but it could be libc as well?
<samueldr> clever: I think it could be both
<samueldr> bash had code to check failures near
<samueldr> but man 3 execve is quite explicit
<clever> c2d ~ # man 3 execve
<clever> No entry for execve in section 3 of the manual
<samueldr> so linux peeps thought "we're making it stop execute" but in actuality it was "userspace ends up executing it in the weirdest way!"
<clever> ive only got a 2 variant of it
<samueldr> I think it should be in a libc output
<samueldr> but haven't tracked it yet, I was using "the google" to find it
<samueldr> (not on my workstation) bash *also* has some code to handle shebangs on failure
<clever> samueldr: the aux-vector also comes into play, just after exec
<samueldr> so I'm thinking it's the common way to handle ENOEXEC, which makes me weary that the error code is also wrong assuming one wanted to truncate there :/
<clever> when the kernel starts running a program at _start, the auxv, argv, and envp are all in ram
<gchristensen> clever: I bet we could just substituteinplace the failing test?
<clever> gchristensen: PR updated
<gchristensen> it is?
<clever> git gui defaults to origin
<clever> and i'm used to lacking push, lol
<clever> at least it wasnt master!
<gchristensen> :)
<tilpner> Suggestion: Don't have origin, have remotes "nixos" and "clever"
<gchristensen> can't wait to turn off push to master!
<clever> why is an orange led blinking on my laptop......
<clever> Battery 0: Charging, 6%, 01:43:55 until charged
<LnL> I setup origin to be readonly and name everything else explicitly
<gchristensen> nice
<clever> LnL: how did you make origin read-only? or did you just break the push-url?
<LnL> yep :p
<LnL> git remote -v #=> origin readonly, use "nixos" to push upstream
<tilpner> You could set origin to the https nixpkgs URL, but why bother with having origin at all
<clever> origin git@github.com:NixOS/nixpkgs.git (fetch)
<clever> origin readonly (push)
<clever> tilpner: default thing to pull from
gchristensen is now known as SvenSlootweg
SvenSlootweg is now known as gchristensen
<clever> gchristensen: the setup hook
<clever> dang, was scrolled up 5 pages, lol