lopsided98 has quit [Remote host closed the connection]
lopsided98 has joined #nixos-dev
<shlevy>
angerman: Is crossSystem defined? I've hit a number of cases where a dependency was compiled twice due to buildPackages vs buildPackages.buildPackages (https://github.com/NixOS/nixpkgs/issues/35543)
<angerman>
shlevy: thanks for the pointer to the issue, I'll investigate.
<angerman>
shlevy: btw your ssh accessible VM, could be very helpful for Template Haskell runners.
<shlevy>
angerman: Yeah, would be cool, though TBH I'm a little sad about enabling the sorry state of cross-TH to continue :P
<angerman>
ha. I'm not a big fan of the TH runner approach. I think we ought to find a better solution. One will be to have a multi-target GHC, that can build splices on the build machine and use those.
<angerman>
but there are limits.
<shlevy>
This is naive but I'd much rather we had a new metaprogramming approach that is inherently cross-compilation aware
<angerman>
shlevy: like reader macros in lisp? :p
<shlevy>
Possibly not arbitrary IO, but if arbitrary IO arbitrary IO run in a sandbox with well-defined interfaces for e.g. reading files
<shlevy>
:P
<angerman>
I think I'll end up stying the use case of TH on hackage. And I believe a lot is simple AST expansion, whereas some of the others is resource embedding (git-hash, files, ...)
<shlevy>
Yeah
<angerman>
the AST expansion could be handled by a clearly defined plugin (instead of arbitrary haskell functions from arbitrary libraries); the second part could be solved by some pre-processor like hsc2hs.
<angerman>
that would also allow to contain the _allowed_ IO in the preporcessor.
<angerman>
instead of allowing arbitrary IO.
<shlevy>
Honestly I think qemu-user is a better candidate than a VM
<shlevy>
As syscalls work more like you'd expect
<angerman>
shlevy: but that's confined to linux-only I believe.
<shlevy>
But it's limited to Linux :(
<angerman>
ha :-)
<angerman>
Ideally, I'd prefer not to have to run anything on the target at all.
<angerman>
I've even gone as far as parsing assembly to extract constants (for hsc2hs).
<shlevy>
:D
<shlevy>
I've always been annoyed at how long the C world has been happy with autoconf hell when the compiler could just be extended to answer the relevant questions
<angerman>
the binary-search interrogration of the CC is just unbelievably slow.
<shlevy>
with GHC there's much less of the "portability at all costs" concern
<angerman>
shlevy: as I mentioned on twitter. Gcc and Clang should just provide `--expr <constant expr>`, and dump the result onto stdout.
<angerman>
e.g. `--expr 'sizeof int'`.
<shlevy>
Yeah
<angerman>
maybe we'll end up hacking that into those compiles eventually :-)
<shlevy>
:)
<angerman>
so yea, my current plan is: make it work, then make it pretty.
<shlevy>
:)
<shlevy>
dtz: The systemd musl patches from OpenEmbedded, do you know if they were ever submitted upstream?
<dtz>
Don't know, sorry :)
<dtz>
some were, for musl support, but those were from a few years ago I think. Not sure.
<shlevy>
OK
<shlevy>
There's interest in #riscv in a nice cross setup targeting an in-development musl-based nommu target with shared libs
<shlevy>
I'm hopeful by the time their toolchain is ready we can get cross-compiled NixOS ready for them :)
pxc has quit [Ping timeout: 252 seconds]
pxc has joined #nixos-dev
pie__ has quit [Ping timeout: 252 seconds]
<angerman>
Ohh... so hsc2hs is pulling in the full stage2 native compiler. But we would ship hsc2hs in the cross compiler as native as well. Hmm... that should not happen.
<angerman>
Ohh, _and_ we use Setup.hs :-(
<clever>
one weird thing ive found with haskell in nixpkgs, the environment only has the cabal library (for main = defaultMain), but it lacks a cabal binary
<clever>
so you cant just `cabal build`, you have to create a missing Setup.hs and `runhaskell Setup.hs build`
<angerman>
I'm not really sure how I feel about that.
* angerman
is currently going through the `ghc/head.nix` expression. I hope I won't have to touch how we build packages yet. It does work for ghcjs, so I guess it might just work.
<dtz>
lmfao the perl-cross was the classic need for -fwrapv in perl
<dtz>
i mean I suppose "of course" but just... did not expect to be running into that again years later lol
<dtz>
anyway nice they chased that down
<shlevy>
:D yeah
<shlevy>
Do we need to patch perl proper?
<shlevy>
Or just miniperl
<shlevy>
Haven't read in detail
<dtz>
oh, not sure it's just that perl is special and intentionally relies on undefined behavior
<dtz>
requiring it to be built with -fwrapv since forever
<dtz>
on that issue it claims that's the problem although unclear why it was set previously and not in gcc7
<shlevy>
dtz: if you see the patch there was an explicit case statement that didn't include 7 :D
<dtz>
LOL okay
<dtz>
well it's like every compiler I've cared about for over a decade
<dtz>
so
<shlevy>
Now it just checks a lower bound
<dtz>
:P
<dtz>
okay great
<dtz>
was just gonna say, that doesn't seem like a thing you try to enumerate versions for
<dtz>
if compiler accepts -fwrapv, ship it :)
<dtz>
anyway woo
<shlevy>
dtz: Any idea why riscv gcc is in the cache by the way? I didn't add it to release-cross or anything
<dtz>
shlevy: make-bootstrap-tools-cross.nix is my guess
<dtz>
is that... "bad"? lol
<shlevy>
Aaah right
<shlevy>
Just unexpected :D
<dtz>
:D
<gchristensen>
shlevy: cut the line passes would be kinda cool :)
<shlevy>
hmmm "0E0 builds have been restarted."
<dtz>
any systemd or otherwise expert folks, please take a look at https://github.com/NixOS/nixpkgs/issues/35415 ? Might miss it based on title but seems like serious problem unless I'm misunderstanding ....
<dtz>
haha
<dtz>
build numbers are usually represented in scientific notation, right?
<dtz>
hopefully not as floats :D
<gchristensen>
our code makes funny assumptions like "nix values are safe for bash" sometimes ... :) echo -n ~fdN!d{@UFoU%WM#{xu)G{Yyc&kaKKP.~(B?X,{!nkux(GJ`Qy*pd'D/)Vh]@w > /var/lib/rabbitmq/.erlang.cookie
<shlevy>
dtz: Are you able to test a fix for that systemd issue?
<dtz>
yes, although not quite automatically (have a mostly-written nixos test for it but didn't finish)
<shlevy>
OK, I'll see if I can make one
<dtz>
well I wasn't sure that my loginctl command was right. fpletz seems to like it so in that case I think just logging in as a user and checking would probably be a direct test of that problem
<shlevy>
erm
<shlevy>
mesonFlagsArray+=(-Dsysconfdir=$out/etc)
<dtz>
although I was doing the whole workflow of ssh -> spawn detached screen -> exit -> ssh -> check
<shlevy>
dtz: Can you just change that to /etc
<shlevy>
And see? I'm a bit busy atm sorry.
<shlevy>
I can check later
<dtz>
so if we have it entirely reading from the wrong /etc folder .... and things aren't blowing up everywhere.... it sure seems like we need way more/better tests
<dtz>
lol
<dtz>
i mean kinda surprising haha
<dtz>
okay
<shlevy>
but it was /etc before
<shlevy>
Only updated on 2/11
<shlevy>
fpletz: ^
<dtz>
shlevy: should pamconfdir be set too?
<dtz>
err modulo punctuation there sorry
<shlevy>
Not sure but almost certainly yes
<shlevy>
And it looks like we lost localstatedir being set... Maybe meson doesn't support it?
<fpletz>
shlevy: ah :( but then systemd will need more patching
<shlevy>
I dunno
<shlevy>
fpletz: Ah, OK. Can you take a look then?
<shlevy>
Also, why the hell is systemd using meson
<shlevy>
meson is so bad :(
<shlevy>
(not your fault :P )
<fpletz>
shlevy: I can, only in a few hours though
<dtz>
yeah I'm trying with both changed, will report back after builds and test :)
<dtz>
d'oh fails to build trying to create /etc/pam.d :)
<fpletz>
Mic92 is looking at it :)
<Mic92>
fpletz: are you patching PKGSYSCONFDIR in systemd?
<dtz>
\o/
<dtz>
thanks guys
<fpletz>
Mic92: I only removed some code that installs to weird locations and creates direactories (like in /var)
<pierron>
“nix-env now ignores packages with bad derivation names (in particular those starting with a digit or containing a dot).” => what about the 2048 game?
<dtz>
it now has a leading underscore
<dtz>
iirc
xeji has joined #nixos-dev
jtojnar has quit [Remote host closed the connection]
<Sonarpulse>
fpletz: globin_: I'm embarrassed to admit I forget which if you has 18.03 your second tour of duty as release manager
<gchristensen>
that should go in the topic!
<Sonarpulse>
gchristensen: I was also going to add it to the github milestones
<Sonarpulse>
(retroactively too, because of course!)
<Sonarpulse>
fpletz: oooo i didn't even know we had a systemd fork
<Sonarpulse>
kewl
<fpletz>
not much actually :>
<Sonarpulse>
the right amount then :)
taktoa has quit [Remote host closed the connection]
<Mic92>
Sonarpulse: mostly patching lookup paths.
<Sonarpulse>
Mic92: makes sense
<Sonarpulse>
has anyone every been able to reach poettering on nix?
<Sonarpulse>
I remember his google+ blog post crazy threads with people being like
<Sonarpulse>
"yo, pls know you are re-inventing nix"
<Sonarpulse>
and no response
<Sonarpulse>
TT
<Sonarpulse>
shlevy: btw some of this was covered in my nix con talk, and we are FINALLY (whew!!) getting up the slides later today
<Sonarpulse>
happy to sumerize in thread, but maybe explaining would be easier on IRC?
<Mic92>
Sonarpulse: Well, I suppose systemd has to be more conversative. However they are reasonable, if you provide the right arguments: https://github.com/systemd/systemd/pull/5816
<shlevy>
Sonarpulse: I'd like the record actually
<Sonarpulse>
Mic92: to be clear, I am no systemd/poettering hater
<Sonarpulse>
I have no love for "but Unix always did ..." which seems to be the majority (though not all) of his detractor's arguments
<Sonarpulse>
shlevy: as in on thread?
<Sonarpulse>
sure
<gchristensen>
Sonarpulse: but unix always cared what unix always did!
<gchristensen>
the april fools joke for ofborg will be all tags are just :robot:
jtojnar_ has joined #nixos-dev
<Sonarpulse>
shlevy: also btw, i think the 17.09 manual predates me deciding we need the second axis
jtojnar has quit [Ping timeout: 245 seconds]
jtojnar has joined #nixos-dev
jtojnar_ has quit [Ping timeout: 256 seconds]
jtojnar_ has joined #nixos-dev
jtojnar has quit [Ping timeout: 240 seconds]
jtojnar_ is now known as jtojnar
jtojnar_ has joined #nixos-dev
jtojnar has quit [Ping timeout: 248 seconds]
jtojnar_ is now known as jtojnar
orivej has quit [Ping timeout: 240 seconds]
jtojnar has quit [Ping timeout: 240 seconds]
jtojnar has joined #nixos-dev
<Sonarpulse>
shlevy: I'm.......still not sure how you think depsBuildBuild is more common
<Sonarpulse>
depsBuildBUild -> building a tool I will use during build and then throw away
<Sonarpulse>
depsBuildHost: building a something that goes in a derivation output
<Sonarpulse>
surely the latter is more common?
<Sonarpulse>
(I guess I could have made that a comment, but I thought it was going to come out less articulate :))
<shlevy>
The vast majority of things I put into nativeBuildInputs are the former
<shlevy>
Or, things I intuitively want to put into nativeBuildInputs
<Sonarpulse>
huh
<shlevy>
Lemme find some recent commits...
<Sonarpulse>
I feel like like build tools on the fly is pretty niche
<Sonarpulse>
maybe lots of odds and ends, but the "main" tools
<Sonarpulse>
like stdenv.cc for C project
<Sonarpulse>
GHC for haskell project
<Sonarpulse>
rustc for rust project
<shlevy>
Yes, but as I said those aren't things you think about *in a specific package*
<Sonarpulse>
true
<shlevy>
Those are things you think about for the infrastructure
<shlevy>
They are much more common, of course
<Sonarpulse>
but syntactically less common
<Sonarpulse>
I get that
<shlevy>
But you almost never explicitly put them into nativeBuijdlInputs
<shlevy>
Yeah
<Sonarpulse>
but then again
<shlevy>
They're just ambient
<Sonarpulse>
maybe stdenv.mkDerivation should only be used by langauge-sepcific buildXXXXPackage :D
<shlevy>
Well, stdenv.mkDerivation is actually a combination of "generic mkDerivation framework" and "buildC{,XX}Package"
<Sonarpulse>
right but as a backer of the latter
<Sonarpulse>
it needs something to put in that stdenv.cc or ghc or whatever
<Sonarpulse>
cc-wrapper is the main case for that second axis
<Sonarpulse>
the target platform if cc wrapper
<Sonarpulse>
relates to which sort of dep the env hook gets placed in
<Sonarpulse>
so depsBuildBuild cc-wrapper
<Sonarpulse>
the env hooks vars are envHost*Hooks
<Sonarpulse>
*sorry that's depsBuildHost
<Sonarpulse>
depsBuildBuild would be envBuild*Hooks
<Sonarpulse>
(I probably could stop caring about the second axis for env hooks, but didn't do that mass rebuild yet)
<shlevy>
I'm not denying the coherence of your framework, and its importance to the internals :)
<Sonarpulse>
fair
<shlevy>
I'm just talking about names and defaults.
<Sonarpulse>
shlevy: so I think the most common native build inputs for me are like send and find or whatever
<Sonarpulse>
things that don't care about target platform
<shlevy>
E.g. on master autogen has buildPackages.autogen in nativeBuildInputs
<shlevy>
Right, but even those are more of the "throw away" variety
<shlevy>
And really it should be buildDepsDep
<shlevy>
erm, depBuildBuild :D
<Sonarpulse>
shlevy: err well it depends how are they used
<Sonarpulse>
to be clear, depsBuild* (including nativeBuildInputs) really should be disallowed referenced by default
<Sonarpulse>
themsevles
<Sonarpulse>
that goes for the stdenv.cc and ghc for haskell and main tools too
<shlevy>
I need a host guile to run build scripts, not build guile
<Sonarpulse>
you only want references to the depHost*
<shlevy>
I don't see why
<Sonarpulse>
oh? I didn't think that last line was disagreeing with you
<shlevy>
That is, I odn't see why you can't reference them *during build time*
<Sonarpulse>
oh sure
<Sonarpulse>
i meant like allowedRequisites
<Sonarpulse>
if there is a non-transitive version of that
<Sonarpulse>
it should be populated with the depsHost* and depsTargetTarget
<shlevy>
oh, absolutely
<Sonarpulse>
cool
<Sonarpulse>
btw so before i had 2 axes
<Sonarpulse>
i had one and preNativeBuildInputs
<Sonarpulse>
maybe that got into the 17.0.9 manual?
<Sonarpulse>
not sure
<Sonarpulse>
anyways they way I thought about it was the build->build case
<Sonarpulse>
was a "pre native build input" because it was a tool that created the tool that created the thing i put in the output
<shlevy>
And my claim is people think of nativeBuildInputs as things that *won't* go there and it's more common. But really my real gripe is "depBuildBuild" should be some variant of "BuildInput"
<shlevy>
Oh, sure. It used to be buildNativeInputs :D
<Sonarpulse>
shlevy: yeah i just learned that
<Sonarpulse>
actually viric's original work was a lot closer to mine
<Sonarpulse>
e.g. crossDrv was default not nativeDrv
<Sonarpulse>
and people wanted like hostInput from the get-go
<shlevy>
I guess I think exposing the two axes as a normal interface is a mistake
<Sonarpulse>
I do fear grandfathering in the cruft
<Sonarpulse>
but at least stdenv/setup I do think needs to be general
<Sonarpulse>
everything rests on being able to package these janky compilers out of the box
<shlevy>
They're real, for sure, but there are two normal cases (a thing my package is going to link to/use at runtime, and a thing I need to run at build time and throw away) and one less normal case (a thing that runs here to make stuff there)
<Sonarpulse>
ok now I want to double check
<shlevy>
Oh, yes. There should be clear ways to access those for people packaging a new compiler
<Sonarpulse>
"thing I need to run at build time and throw away"
<Sonarpulse>
that to me means any depBuild*
<Sonarpulse>
like what we discussed about the allowedRequisites by default
<shlevy>
Right, because most of the time we're not thinking about target. But a build-native cross-compiler does *not* feel like a throwaway to me
<Sonarpulse>
depsBuildBuild is "the ting I need to run at build time to make the thing i need to run at build to to throw away"
<shlevy>
And if I explicitly ask for a compiler that isn't the normal compiler my build infra gives me, it's because I want to build and run stuff during the buld
<Sonarpulse>
not thinking about the "host" I think you mean
<shlevy>
Right, this is another problem because if you had tochoose from those three words "target" should clearly mean "the target this build will run on" :P
<Sonarpulse>
I believe you that if the "primary" compiler is build->host, the secondary is often build->build
<Sonarpulse>
shlevy: yes "target" is the best name and "target platform" is the worst platform
<shlevy>
And if the "primary" isn't build->host, you're either not thinking about cross-compiling or you're working on write-once use-everywhere infrastructure
<Sonarpulse>
I guess "build" is the clearest name"
<Sonarpulse>
but "target" just sounds like "the other thing"
<shlevy>
:D
<Sonarpulse>
"And if the "primary" isn't build->host, you're either not thinking about cross-compiling or you're working on write-once use-everywhere infrastructure"
<Sonarpulse>
err the primary is build->host
<Sonarpulse>
almost by definition
<Sonarpulse>
my nativeBuildInputs choice presumes the secondary is also build->host
<Sonarpulse>
your choice presumes its build->build
<shlevy>
Yeah
<shlevy>
Sorry, I'm mangling this terminology
<Sonarpulse>
shlevy: hey, that you understand it and still mangle it demonstrates it really is bad names :)
<Sonarpulse>
shlevy: what about when your build tools need libraries
<Sonarpulse>
those are nativeBuildInputs today
<shlevy>
I guess in a sense from the perspective of building libstdc++ the "primary" is build->target because it's part of the compiler build
<Sonarpulse>
I suppose those are libraries so second axis doesn't matter
<shlevy>
Right
<Sonarpulse>
in the general build != host != target case
<Sonarpulse>
which is good to keep in mind even though we never do it
<shlevy>
I was wondering if you were ever going to get there :D
<Sonarpulse>
shlevy: so we do support it in principle
<Sonarpulse>
but it means we can only have build->* deps as deep as the bootstrapping stage chain length :D
<shlevy>
It may be helpful to think about that case, yeah
<Sonarpulse>
(I figured that out procrastinating studdying for a math final two years ago haha)
<Sonarpulse>
my mind was like "anything but analysis" in that library basement
<shlevy>
coreutils would then be... depBuildBuild, right?
<Sonarpulse>
for all three not equal?
<shlevy>
Oh I guess it still doesn't matter
<Sonarpulse>
yeah
<Sonarpulse>
shlevy: I talked to bgamari about doing a build != host != target CI for GHC
<Sonarpulse>
(at least until we make it multi-target :D:D)
<Sonarpulse>
because it is very good bullshit detector
<shlevy>
OK, so in that case, when would you want depBuildBuild when depBuildHost?
<shlevy>
You want depBuildBuild if you need to compile some helper haskell tool during the build
<shlevy>
You want depBuildHost if you need to compile something that's going to run *later*, but you already *have* that.
<Sonarpulse>
shlevy: if you are doing FFI thing on the fly
<Sonarpulse>
you need depBuildHost
<Sonarpulse>
it seems like this build-tool on fly is mainly C thing
<Sonarpulse>
except for Setup.hs
<Sonarpulse>
the FFI thing is sorta common
<shlevy>
Can you give an example?
<Sonarpulse>
but then again the C compiler is usually brought in in as propagated runtime dep of the other language compiler
<Sonarpulse>
example of FFI?
<Sonarpulse>
like i have c-bits: blah blah
<Sonarpulse>
or maybe someday rust-bits: blah blah
<Sonarpulse>
compile that into object file
<Sonarpulse>
link it with my haskell
<Sonarpulse>
it goes in $out/lib
<shlevy>
Right, but like you said that's a runtime dep
<Sonarpulse>
GHC, rustc, and GCC, are all build->host
<Sonarpulse>
yeah the ffi shim code is runtime
<shlevy>
It'd only be something the *packager* needs to think about if it's not a natively supported FFI interface
<Sonarpulse>
but the compiler is build time
<shlevy>
If it is, the compiler holds a reference to the C or rust compiler