supersandro2000 has quit [Disconnected by services]
supersandro2000 has joined #nix-darwin
squidgy_ has quit [Quit: ZZZzzz…]
squidgy has joined #nix-darwin
fiddlerwoaroof_ is now known as fiddlerwoaroof
squidgy has quit [Quit: ZZZzzz…]
philr_ has quit [Ping timeout: 245 seconds]
Chiliparrot has joined #nix-darwin
Chiliparrot has quit [Ping timeout: 264 seconds]
philr_ has joined #nix-darwin
__monty__ has joined #nix-darwin
squidgy has joined #nix-darwin
philr_ has quit [Ping timeout: 256 seconds]
andremedeiros has quit [Read error: Connection reset by peer]
andremedeiros has joined #nix-darwin
squidgy has quit [Quit: ZZZzzz…]
squidgy has joined #nix-darwin
Chiliparrot has joined #nix-darwin
<abathur>
oliver85: not certain, but I kinda suspect the /bin/sh thing is a dead end
<abathur>
it would've been nice to know if it was easy, but I don't see any reason to fight through figuring out why it doesn't have the right context without some more evidence that it'll lead to a solution
<abathur>
oliver85: responded on GH to part of that
<abathur>
oliver85: I also found another IRC log I had missed before; added it to my list of links on the topic but here it is; much more similar to the arc of your report https://logs.nix.samueldr.com/nix-darwin/2020-11-04
<oliver85>
I am going to open a issue with buildkite and see what are the difference between an agent launched via normal gui login session vs. launchd jobs, since thats the difference between work/doesn't work
<oliver85>
I will paste it here if you are interested to follow
<abathur>
oliver85: my main thought at this point is to carefully step through checking ownership/perms of the relevant files
<abathur>
you can also link it from the issue
<abathur>
I'm not sure if they'll know anything specific or not; at some point I'll ping the guy who runs eclecticlight.co to see if he knows anything, but I figure it's worth reviewing permissions to make sure they make sense
<abathur>
oh, and I guess, implied from my issue comment; if you haven't tried the hosted installer, it may be worth giving that a go in case we're chasing something that it accidentally fixes
<oliver85>
follow up questions regarding nix not auto mount when reboot - should 'auto' option be in /etc/fstab? Does the auto mount behavior requires a initial reboot?
<oliver85>
Because the Amazon Machine Image baking process works like this - packer go create a machine, install all the stuff on it (nix, brew,...etc), capture the state of the machine into an image, then shuts down.
<oliver85>
if a reboot is required to get the automount behavior then thats probably what is msising
<abathur>
previously it had auto I think; specifying noauto in my PR to let the launchdaemon handle mounting
<abathur>
basically we're trying to target a problem case with FDE
<abathur>
where the credentials aren't ready yet at the time it tries to automount
<abathur>
so it fails to do so, so any apps/services restore or launch that depend on anything in /nix as the system boots or user logs in will break
<abathur>
so the system/org.nixos.darwin-store launchdaemon should be handling the mount, but there may be a genuine case for using enableOwnership here, I'm not sure
<oliver85>
sounds like using --daemon install would be the path forward
<abathur>
I suspect the token is the better approach, because I think the chown -R will break the store for the nix-daemon
<oliver85>
you mean token pair with single or multi install?
<abathur>
iirc my PR had to add `chown -R root:nixbld /nix` to smooth over some existing-install issues
<abathur>
token + multi
oliver85 has quit [Quit: Connection closed]
oliver85 has joined #nix-darwin
<abathur>
howard from eclectic light says this doesn't ring any bells for him
oliver85 has quit [Quit: Connection closed]
<__monty__>
I'm trying to understand stdenv. Are the headers used to build cmake-boot (pkgs.cmakeMinimal) in stage1 of the stdenv coming from the bootstrapFiles?
Chiliparrot has quit [Ping timeout: 244 seconds]
<__monty__>
Hmm, looks like it. IIUC this means if I want to bump the SDK I'll have to generate a new bootstrapFiles?
<__monty__>
But building make-bootstrap-tools.nix tries to build my stdenv. Is this chicken and egg or can I build new bootstrap tools with an old stdenv?
<supersandro2000>
probably
<supersandro2000>
how else would you do it?
Chiliparrot has joined #nix-darwin
<abathur>
I do recall discussion of a chicken/egg issue related to the big sur work that I think required using an existing system's stdenv, but it was over my head by enough that I'm not certain if that's exactly what you're describing; thefloweringash or LnL (anyone else?) may have more context
<LnL>
that sounds weird, why do you need the new sdk for cmake?
<__monty__>
I'm working on bumping the SDK. Building stdenv requires cmake-boot.
<LnL>
yes, but that's no different from now
<LnL>
big sur did indeed need 2 steps because the linker in bootstrap tools needed to be update in order to build things using tbd definitions starting from stage1
<__monty__>
Building cmake-boot is failing due to missing symbols, like _futimens.
<__monty__>
I assume they're missing because it's using header that aren't coming from the SDK.
<LnL>
that's unexpected, cmake-boot should build with and link against bootstrap tools so nothing really changes there when the stdenv has changes
<LnL>
it's usually just after everything has been updated that we also bump bootstrap tools to avoid lagging behind and running into trouble next time, but in most cases that's not even strictly required
<__monty__>
I think it's because I regenerated the symbol lists in apple_source_releases/Libsystem.
<__monty__>
No no, _futimens nanoseconds : )
<abathur>
it always takes me ~3 reads to figure out how to parse that name
<LnL>
FYI you can peek at bootstrap tools like this
<LnL>
nix-build -A stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.darwin.Libsystem
<LnL>
but that will have the same stub files in $out/lib/system/libsystem_kernel.tbd since it's not part of the actual bootstrap tarball IIRC
<LnL>
ok so this implies cmake wants new symbols even tho nothing changed?
<LnL>
good thing to verify would is if you get the same on master with nix-build --check
<LnL>
alltho on big sur it's not really possible anymore to detect symbols anymore so that would be a bit surprising
<__monty__>
I'm stuck on High Sierra.
<__monty__>
Trying to bump to 10.13 mainly because it's practical for me to work on, as a way to get familiar so I can bump to 10.14 or 10.15 probably.
<LnL>
ah, it that case it's possible something is leaking in the build (which a check should confirm)
<LnL>
normally this isn't necessary, but with the split of bootstrap tools and stubs it should be much easier to generate stubs for 10.13 and use those without having to do the whole bootstrap dance
<LnL>
you can generate and throw a custom tbd in here and it should get picked up right from stage1
Chiliparrot has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
Chiliparrot has joined #nix-darwin
<__monty__>
A nix-build --check? Nix errors saying evaluating the entirety of Nixpkgs is prevented unless you circumvent that. Do you mean `nix-build -A stdenv --check` or something?
<__monty__>
Is generation of the stubs documented anywhere?
oliver85 has joined #nix-darwin
<oliver85>
nix not mount when I ssh in (this is installation with multi)
<oliver85>
mount -a ec2-user@ip-10-249-9-39 ~ % mount -a
<oliver85>
mount_apfs: volume could not be mounted: Operation not permitted
<oliver85>
mount: / failed with 77
<__monty__>
Ah, the README in the stubs repo documents it I guess : )
<abathur>
oliver85: with the securetoken?
<oliver85>
no, I don't think the token I generated on one machine (the imaging machine) would work on other machine
<oliver85>
but I can give it a shot
Chiliparrot has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
<abathur>
fair :) does the query say it has one already?
squidgy has quit [Quit: ZZZzzz…]
<__monty__>
LnL: Hmm, on master I do get a determinism warning when building llvmPackages_7.clang-unwrapped with --check (this is the smallest attribute I'd found that depended on cmake-boot): may not be deterministic: output '/nix/store/hc44758rqfmm08m4fl79ycvicf1wz52y-clang-7.1.0-lib' differs
<LnL>
no you want to build cmake-boot specifically, --check doesn't recurse down
<LnL>
nix-build -A stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.cmake --check
<LnL>
(assuming it's the one from stage1 that fails ofcourse
<__monty__>
"some outputs of '/nix/store/hg20jd1i1944na6s8k3m0hclnav5m4j5-cmake-boot-3.19.4.drv' are not valid, so checking is not possible"
<__monty__>
I'm assuming it's the one from stage1 because that's the first time it pops up afaict.
<LnL>
ah yeah, check only works if you already have the build locally
<LnL>
so build once without --check to download it from the cache and then with to rebuild
<oliver85>
abathur its disabled. I just tested it by enabling it in the image builder, but the machiune booted using that image doesn't have it enabled
<abathur>
ok
<abathur>
that makes sense, though I'm not quite sure what it means for the prospect of fixing this in the installer
<abathur>
it may make more sense to just detect the condition and warn
<abathur>
at least until we find some other way through
<abathur>
assuming we don't stumble on one soon
<abathur>
I did find some doc mentions of a "Mach bootstrap context" that ~might be part of the explanation for what's going on here, but it seems like discussion of it as a concept has died down a lot in recent years, so I'm not sure
<abathur>
if it is, the basic answer might be that it seems like a global launchdaemon would normally run in the ~root mach bootstrap context, and a launchagent would run in user's mach bootstrap context
<abathur>
but it isn't very clear to me from what I've found so far what happens when it's a launchdaemon specifying a username to run the job under, or for that matter a user invoking launchctl load
philr_ has joined #nix-darwin
<abathur>
oliver85: is your number of machines small enough and static enough that it's feasible to do the securetoken process on each initially and be OK thereafter?
<__monty__>
Ok, --check holds out on master, now for my branch.
<abathur>
oliver85: poking AWS for support may also be an option, if they take it as a buck-stops-here sort of thing
webframp has left #nix-darwin [#nix-darwin]
<oliver85>
abathur after added the token and reboot tho (did nothing else) the nix vol is mounted
<oliver85>
and all nixbld are own by root, is that expected?
<abathur>
not sure I follow the latter q
<oliver85>
ls -la /nix
<oliver85>
total 0
<oliver85>
drwxr-xr-x 4 root nixbld 128 Mar 17 20:53 .
<oliver85>
drwxr-xr-x 22 root wheel 704 Feb 10 01:37 ..
<oliver85>
drwxrwxr-t 54 root nixbld 1728 Mar 17 20:54 store
<oliver85>
drwxr-xr-x 4 root nixbld 128 Mar 17 20:53 var
<abathur>
ah yes
<abathur>
nix should be owned by root; it should be running the nix-daemon as root, but performing the actual builds with nixbld* users that are in the nixbld group
<abathur>
oliver85: I assume the buildkite agent still has the same error on boot?
<__monty__>
LnL: Ok, so without the darwin-stubs changes I get the missing symbols for cmake-boot. With the stubs changes it fails earlier when building bootstrap-stage0-Libsystem, ln: failed to create symbolic link '/nix/store/fp4sa206w5his9a9c2hzxi30irdh7bd4-bootstrap-stage0-Libsystem/lib/libSystem.tbd': File exists
<abathur>
oliver85: if the regular nix-daemon is running without the same sandbox error, and you still see it for the agent, I have another idea: I'm wondering a bit if the sandboxing is being triggered by something about touching the user's home directory while the user isn't logged in, since your job sets a PATH that will try to get Nix via a symlink there
squidgy has joined #nix-darwin
<oliver85>
@abathur sorry for slow response, in a meeting, jumping in and out
<abathur>
oliver85: so maybe replacing the buildkite-agent segment at the head of your PATH with `/nix/var/nix/profiles/default/bin` will work
<abathur>
no worries
<LnL>
__monty__: hmm, what changes did you make exactly?
<LnL>
ah you completely replaced it, something's going wrong with the symlinks
<LnL>
maybe try just replacing that one stub
<LnL>
the stubs are pretty new so there are probably still some rough edges with the scripts
<__monty__>
There were a lot of messages about symlinks not being possible because files didn't exist.
<__monty__>
By one stub you mean the Libsystem stub?
<LnL>
yeah, I assume the missing symbol will also be in the kernel stub
<__monty__>
Are the symbol lists in apple_source_releases/Libsystem still used btw or was that a red herring?
<LnL>
oh those are still there? I think it was cleaned up but could be wrong
<__monty__>
Still there on master yeah.
<oliver85>
abathur still says nix not found, something is funny with the bootstraped shell from buildkite
<oliver85>
wait
<abathur>
is that with it set to /bin/sh, or the default /bin/bash?
<oliver85>
env not set
<__monty__>
LnL: Thanks for your help. It's late though so I'm gonna stop looking into it rn. Continuing next week at the latest.
<abathur>
<3 __monty__ LnL
<{^_^}>
__monty__'s karma got increased to 31
<abathur>
<3 LnL even
<{^_^}>
LnL's karma got increased to 97
<__monty__>
abathur++ LnL++ Spreading the internet points : )
<{^_^}>
abathur's karma got increased to like 21, I think
<{^_^}>
LnL's karma got increased to 98
<abathur>
grin
<LnL>
__monty__: I've been pretty absent, but feel free to ping me if you have more stdenv questions :)
<__monty__>
Will do. One question, what do you think of bumping SDK but keeping deployment target at 10.12?
<__monty__>
Possibly introducing a way for individual packages to override the deployment target if upstream doesn't support that far back?
<LnL>
it should be overridable already
<__monty__>
At the individual package level? To relax it to lower than the 10.12 SDK?
<LnL>
the problem is that updating the stubs will also result in things linking against system libraries with potentially missing symbols
<__monty__>
You mean when deployment target is older than SDK? That's why it'd require overriding per package. If a package is found to be incompatible below version 10.xy then that should be its deployment target.
<emily>
does x86-64 Big Sur work with nixpkgs-unstable yet? (is that going to get merged from staging at the same time as Apple Silicon support? wondering how long to hold off my upgrade)
<__monty__>
I think Big Sur support has been merged.
<__monty__>
At least stdenv is bootstrappable.
<oliver85>
abathur `/nix/var/nix/profiles/default/bin` but saw the same dylb error
<oliver85>
tried adding that to the buildkite-agent's launchd path
<oliver85>
something is funny witht the bootstrap shell
<abathur>
drat
<abathur>
I wonder a little about clearing the other ~/buildkite-agent references out of there as well, may cause other problems but would answer the question of whether it is ~interacting with the homedir that is causing the sandboxing/dylib behavior or not
__monty__ has quit [Quit: leaving]
<abathur>
I guess the HOME, NIX_PATH, latter of the NIX_PROFILES, and I guess we can do the same thing as above to the ca bundle
<abathur>
nix-daemon uses /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt for that one