<abathur>
sorry, helping someone out in #nixos; divided attention :)
<oliver85>
all good
<abathur>
I'm aware buildkite exists but not otherwise familiar with it, not sure if that'll end up being relevant
<oliver85>
Its getting more popular these past couple years. Its basically a build orchestrator and you bring your own agents. In our case our agents are AWS ec2 mac1.metal instances
<abathur>
did you also add the SecureToken to the buildkite user, I guess?
<oliver85>
not on the AMi but yes after I booted up I ssh in and added SecureToken manually
<abathur>
iirc others reported being able to skirt this issue by using a GUI session to add a security exemption (FDE, I think) for /bin/sh, in case you're under pressure to have something working yesterday here
<abathur>
some of the links I listed in the previous post have a little contextual information; I haven't directly done this
<abathur>
I'm not sure if that's the only path or not
<abathur>
before I was thinking that this was about running ~headless systems, because all of the reports involved people trying to set up over ssh, failing, and then describing going in through the GUI
<abathur>
but I'm not quite clear on how you're connected when you describe the commands at the end of your last comment in the issue
<abathur>
i.e. "if I launch the daemon sudo launchctl load -w /Library/LaunchDaemons/com.buildkite.buildkite-agent.plist then I see the dyld: Library not loaded post above when build" and "but I can successfully build with nix if I invoke the agent directly after ssh into the machine
<abathur>
sudo su - buildkite-agent then /usr/local/bin/buildkite-agent start"
<abathur>
so if the first part of that IS with a GUI that might help rule out at least one possibility
<oliver85>
ah let me clarify my last comment on the issue and give a bit more context.
<oliver85>
buildkite-agent can be start (https://buildkite.com/docs/agent/v3/cli-start) by directly invoking the binary at /usr/locl/bin/buildkite-agent. Now an agent is running on my machine. from buildkite.com I can then schedule job that will be pick up by the agent. Jobs like "nix --verison"
<oliver85>
If I start the buildkite agent by manuall ssh into my ec2 instance. run (. /Users/buildkite-agent/.nix-profile/etc/profile.d/nix.sh) then kick off the agent "/usr/local/bin/buildkite-agent start" as buildkite-agent user. Then the job pick up by this agent seems to run nix command just fine.
<oliver85>
But if I have launchctl kick off the agent (agent is set to start on boot), then jobs picked up by this agent shows the dyld error
<oliver85>
I am not sure what you meant by "GUI session to add a security exemption"
<abathur>
well, it's a good sign that it will run over ssh if set up correctly
<abathur>
I guess it might be failing for some reason related to the system not having a logged-in user, or something about the launchd context compared to how you're invoking it
<abathur>
re: GUI; sorry, I said FDE instead of FDA earlier :) if I understood correctly they did something like literally VNC in to the desktop, open the system preferences > security & privacy > privacy > Full Disk Acces, unlock, and then add /bin/sh
<oliver85>
ah gotcha, Yah VNC into desktop will be hard to incorporate with the AMI creation process. (i.e pause the process, VNC in, then continue)
<oliver85>
when you say `iirc others reported being able to skirt this issue` which issue you referring to? dyld?
<oliver85>
wait a second, when I created buildkite-user on my machine I had bin/bash as my default shell `sudo sysadminctl -addUser buildkite-agent -shell /bin/bash`
<oliver85>
if I have the buildkite daemon run job `echo $SHELL` I get bin/bash as well.
<oliver85>
I should switch to bin/sh?
<oliver85>
so they are compatable and bin/sh have access to the entire disk (somehow?)
<abathur>
worth a try
<abathur>
it does seem like the difference between works/doesn't is just some permission grant that happens in a normal gui login session, and apparently happens in a ssh session
<abathur>
but which isn't happening for these launchd jobs
<abathur>
oh huh, that reminds me...
<abathur>
ah; nm; was thinking of a launchd setting, but it looks like it should be enabled by default in this case
<oliver85>
cool cool.
<oliver85>
changed default shell to `chsh -s /bin/sh`
<oliver85>
if I run `echo $SHELL` I see /bin/sh
<oliver85>
but if I run nix --version......I see its using /bin/bash trapping?! (huh) I am digging to see where this trap comming from
<oliver85>
```
<oliver85>
$ trap 'kill -- $$' INT TERM QUIT; nix --version
<oliver85>
dyld: Library not loaded: /nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib
<oliver85>
Reason: no suitable image found. Did find:
<oliver85>
file system sandbox blocked open() of '/nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib'
<oliver85>
/nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib: stat() failed with errno=1
<oliver85>
file system sandbox blocked open() of '/nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib'
<oliver85>
```
<abathur>
not sure I follow, but chsh alone won't change the shell for your current session; you might need to run /bin/sh or start a new one
<oliver85>
yeah I started a new one.
<oliver85>
sorry the above log wasn't complete
<oliver85>
if the buildkite-agent daemon echo SHELL we get bin/sh but if the daemon runs `nix --version` I seee it erorr out with indication using bin/bash. strange..
<oliver85>
file system sandbox blocked open() of '/nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib'
<oliver85>
🚨 Error: The command exited with status 134
<oliver85>
in/bash: line 1: 1426 Abort trap: 6 nix --version
<oliver85>
uh ignore the above, not sure why my copy pasting is been weird
<abathur>
I wonder if that's buildkite
<oliver85>
see below =============
<oliver85>
$ trap 'kill -- $$' INT TERM QUIT; nix --version
<oliver85>
dyld: Library not loaded: /nix/store/i1cg0wfns9j4lmfzvx5dz6rc436vs6ms-libsodium-1.0.18/lib/libsodium.23.dylib
<oliver85>
the paste keeps ignoring this line `/bin/bash: line 1: 1426 Abort trap: 6 nix --version` <---- that should appear above the line of red siren
<abathur>
so I think that's probably a red herring here
<abathur>
but still probably worth making it use /bin/sh at least once, just in case
<oliver85>
hmm not sure whats the best way of forcing it to. probably should change the plist?