<ixxie>
do you use helm with your NixOS Kube clusters?
<johanot>
my colleagues do, I don't.. afaik helm/tiller works fine in "kubernixos"
<ixxie>
so you just package whatever you need in Nix?
<johanot>
oh wait I don't anything about any nix'y way of managing helm charts. I think my colleagues keep helm packages and nix separated. I can ask in the office tomorrow
<ixxie>
johanot: well, no stress about that; I was just curious. I could start with my own simple deployment patterns, and try out your addon-manager
<ixxie>
first thing first is learning basics of the cluster mgmt
<johanot>
The official kube-addon-manager is part of the NixOS module now, but it is kinda limited, because it only allows for addons in the kube-system namespace
<johanot>
btw. if you look at it, it is nothing more than a simple bash loop that runs "kubectl apply -f <manifest>; sleep XX;"
<ixxie>
so you can just drop manifests somewhere and they get deployed
<ixxie>
why not put them all in a big directory and do "kubectl apply -R -f dir/"?
<ixxie>
along with your own manifests that is?
<johanot>
we have our own internal option-set (not part of the upstream module) like this: "k8s.manifests = { my-deploy = { apiVersion = "v1"; kind = "Deployment; ... }; }; .. and then our "addon-manager" runs pkgs.writeText builtins.toJSON .. so the generated json manifests ends up in the store. Then we generate one systemd unit per manifest, actually. To have flexibility in "stopping/starting"
<johanot>
individual addons.
<johanot>
big disclaimer: this is far from optimal, which is why we haven't upstreamed it (at least yet) :P
<ixxie>
so the service does something like "kubectl apply -f manifest.json" and "kubectl delete -f manifest.json"?
<johanot>
ixxie: yes
<ixxie>
nice
<johanot>
AND you get systemd depedency chains for free, i.e. manifest A requires manifest B etc.
<ixxie>
and I guess the main upshot is being able to wrap kubernetes objects in nixos services?
<ixxie>
because the manifests could already be quite declarative
<ixxie>
but of course normally without a language as expressive as Nix
<ixxie>
johanot: for your branch, I can just do `services.kubernetes.roles = [ "master" "node" ];` and I should be good to go?
<johanot>
right.. but i'm still not 100% satisfied :P it does not prevent someone imperatively modifying your cluster object, even though you created them in a declarative manner (for example).
<ixxie>
yeah
<johanot>
ixxie: Yes.. You'll need "services.kubernetes.masterAddress" as well
<johanot>
1 sec
<ixxie>
I can see your motivation, and your approach seems cool