Hi all,
finally I had some time to try and get the 389ds server running in Kubernetes.
As I am also still learning Kubernetes (and will be for the next few years, always something new to learn), I tried to get this into a helm chart. And: It works!
https://github.com/johanneskastl/389server-helm-chart/tree/main/charts/389se...
This means, I can create a Kubernetes secret containing the DS_DM_PASSWORD and SUFFIX_NAME variables, and the pod running inside Kubernetes gets those values injected from the secret. I could connect to the server using the "cn=Directory Manager", i.e. I get an "No such object" response but no longer a "ldap_bind: Invalid credentials (49)" warning.
There are three questions that I could not answer myself:
1. Does the docker container have any kind of bootstrapping mechanism included, i.e. I put some LDIF files somewhere and those get imported automatically? Or is the idea to just setup a working server, that the user can then put things into. And that state is being preserved in /data/? I guess it is the latter.
2. The choice of certificate/key naming is unfortunate, as Kubernetes secrets automatically contain a tls.key and tls.crt file. I can easily inject such a secret, but as 389ds is expecting server.key and server.crt, I guess those files would just be ignored. Right? Could this maybe be enhanced (not changed, to keep backwards compatibility)? That would make life in Kubernetes land easier.
3. The Dockerhub page say "Instances now support running dirsrv as non-root...". However I could not get it to run without root permissions. That might have been due to the "wrong" user I picked. So, before spending more time on debugging this, is this still supported? Which user/group should I pick to do that?
Thanks in advance, and have a nice day everyone!
Johannes
On 22 Mar 2022, at 19:22, Johannes Kastl kastl@b1-systems.de wrote:
Hi all,
finally I had some time to try and get the 389ds server running in Kubernetes.
As I am also still learning Kubernetes (and will be for the next few years, always something new to learn), I tried to get this into a helm chart. And: It works!
https://github.com/johanneskastl/389server-helm-chart/tree/main/charts/389se...
This means, I can create a Kubernetes secret containing the DS_DM_PASSWORD and SUFFIX_NAME variables, and the pod running inside Kubernetes gets those values injected from the secret. I could connect to the server using the "cn=Directory Manager", i.e. I get an "No such object" response but no longer a "ldap_bind: Invalid credentials (49)" warning.
There are three questions that I could not answer myself:
- Does the docker container have any kind of bootstrapping mechanism included, i.e. I put some LDIF files somewhere and those get imported automatically? Or is the idea to just setup a working server, that the user can then put things into. And that state is being preserved in /data/? I guess it is the latter.
At the moment we can't import an ldif like this. There are lots of issues with state management in this kind of setup, that LDAP is not well suited to cope with.
- The choice of certificate/key naming is unfortunate, as Kubernetes secrets automatically contain a tls.key and tls.crt file. I can easily inject such a secret, but as 389ds is expecting server.key and server.crt, I guess those files would just be ignored. Right?
Could this maybe be enhanced (not changed, to keep backwards compatibility)? That would make life in Kubernetes land easier.
It's annoying that kubernetes is so inflexible in how it emits the certs like this. Can you explain more about how it puts them into the volume or similar? That way we can work out what the right answer is.
- The Dockerhub page say "Instances now support running dirsrv as non-root...". However I could not get it to run without root permissions. That might have been due to the "wrong" user I picked. So, before spending more time on debugging this, is this still supported? Which user/group should I pick to do that?
Any! What version of the container are you running? You'll need the upstream version which has the correct nsswitch components in place.
Thanks in advance, and have a nice day everyone!
Thanks!
Johannes
-- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de
B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Sincerely,
William Brown
Senior Software Engineer, Identity and Access Management SUSE Labs, Australia
Hi William,
thanks for the reply.
On 22.03.22 at 23:22 William Brown wrote:
On 22 Mar 2022, at 19:22, Johannes Kastl kastl@b1-systems.de wrote:
- Does the docker container have any kind of bootstrapping mechanism included, i.e. I put some LDIF files somewhere and those get imported automatically? Or is the idea to just setup a working server, that the user can then put things into. And that state is being preserved in /data/? I guess it is the latter.
At the moment we can't import an ldif like this. There are lots of issues with state management in this kind of setup, that LDAP is not well suited to cope with.
No problem, then I know that I do not need to build anything like that into the helm chart.
- The choice of certificate/key naming is unfortunate, as Kubernetes secrets automatically contain a tls.key and tls.crt file. I can easily inject such a secret, but as 389ds is expecting server.key and server.crt, I guess those files would just be ignored. Right?
Could this maybe be enhanced (not changed, to keep backwards compatibility)? That would make life in Kubernetes land easier.
It's annoying that kubernetes is so inflexible in how it emits the certs like this. Can you explain more about how it puts them into the volume or similar? That way we can work out what the right answer is.
Here is an example from another pod, where the Kubernetes secret is mounted into the pod into /certificate:
~ $ ls -lh /certificate/ total 0 lrwxrwxrwx 1 root root 14 Mar 23 03:32 tls.crt -> ..data/tls.crt lrwxrwxrwx 1 root root 14 Mar 23 03:32 tls.key -> ..data/tls.key ~ $
As you can see, there are two "files", that are being generated from the two key-values-pairs inside the secret. Those are actually linked into that directory.
But the default names are tls.crt and tls.key.
I will try to do subpath mounting, i.e. I will specify where each of the keys is "mounted" to, aka which file name it gets. But AFAIK this has some drawbacks, but currently it seems like the only option.
Alternative approach: I would need to have a look if I can somehow use a different "CMD" in Kubernetes, I am thinking of a small script that just copies tls.key and tls.crt from $LOCATION to /data/... somehwere, where 389ds expects them. And then runs "/usr/lib/dirsrv/dscontainer -r"
- The Dockerhub page say "Instances now support running dirsrv as non-root...". However I could not get it to run without root permissions. That might have been due to the "wrong" user I picked. So, before spending more time on debugging this, is this still supported? Which user/group should I pick to do that?
Any! What version of the container are you running? You'll need the upstream version which has the correct nsswitch components in place.
I tried running without root permissions, but immediately got CrashLoopBackOff errors in Kubernetes. But if you say it should work then I will dig deeper into that and try to get it working. I am using the latest image from dockerhub, tag 2.0.
Kind Regards, Johannes
On 23.03.22 at 09:43 Johannes Kastl wrote:
I will try to do subpath mounting, i.e. I will specify where each of the keys is "mounted" to, aka which file name it gets. But AFAIK this has some drawbacks, but currently it seems like the only option.
This approach did work, putting the following into the deployment specification:
- name: 389server-certs secret: secretName: my-tls-secret items: - key: tls.key path: /data/tls/server.key - key: tls.crt path: /data/tls/server.crt
Not sure what happens on certificate renewal, it might be that the old certificate stays mounted (from what I read). We'll see.
Johannes
On 23.03.22 at 10:50 Johannes Kastl wrote:
This approach did work, putting the following into the deployment specification:
- name: 389server-certs secret: secretName: my-tls-secret items: - key: tls.key path: /data/tls/server.key - key: tls.crt path: /data/tls/server.crt
Copied the wrong (non-working) version of the code... :-)
This one works, even when mounting the secret on top of the volume that is needed to preserve the persistent data:
volumeMounts: - name: 389server-data mountPath: '/data/' - name: 389server-certs mountPath: '/data/tls/' readOnly: true
[...]
volumes: - name: 389server-data persistentVolumeClaim: claimName: 389server-data - name: 389server-certs secret: secretName: my-tls-secret items: - key: tls.key path: server.key - key: tls.crt path: server.crt
Hi again,
On 23.03.22 at 11:23 Johannes Kastl wrote:
seems like just having proper server.key and tls.key files is not enough. I still get a warning that TLS could not be enabled:
INFO: Checking for PEM TLS files ... INFO: Found -> [] INFO: Have /data/tls/server.key -> True INFO: Have /data/tls/server.crt -> True INFO: Have /data/tls/ca -> False INFO: Have /data/config/pwdfile.txt -> True INFO: Unable to configure TLS from PEM, missing a required file.
The only thing that is apparently not found is /data/tls/ca, which is existing (but is an empty directory).
389server-5945c6677f-c856s:/ # ll /data/tls/ total 0 drwxr-xr-x. 2 root root 80 Mar 23 10:40 ..2022_03_23_10_40_09.094925764 lrwxrwxrwx. 1 root root 31 Mar 23 10:40 ..data -> ..2022_03_23_10_40_09.094925764 drwxrwxrwx. 1 root root 0 Mar 23 10:40 ca lrwxrwxrwx. 1 root root 17 Mar 23 10:40 server.crt -> ..data/server.crt lrwxrwxrwx. 1 root root 17 Mar 23 10:40 server.key -> ..data/server.key 389server-5945c6677f-c856s:/ # ll /data/tls/ca/ total 0 389server-5945c6677f-c856s:/ #
From the docker image description:
ca is a folder with PEM format CA's that should be added to the trust root for this instance.
I understood this to be optional? Is this actually required?
Kind Regards, Johannes
On 23.03.22 at 11:46 Johannes Kastl wrote:
ca is a folder with PEM format CA's that should be added to the trust root for this instance.
I understood this to be optional? Is this actually required?
Found it: https://www.port389.org/docs/389ds/howto/howto-ssl.html
ca.crt # The issuers CA certificate.
So I created yet another Kubernetes "secret" holding the Let's Encrypt CA certificate and mounted it to /data/tls/ca/ca.crt, and now the server is enabling TLS and I can connect using "ldapsearch -ZZ ...".
Johannes
Cool, so it sounds like you have it all working then?
On 23 Mar 2022, at 20:23, Johannes Kastl kastl@b1-systems.de wrote:
On 23.03.22 at 10:50 Johannes Kastl wrote:
This approach did work, putting the following into the deployment specification:
- name: 389server-certs secret: secretName: my-tls-secret items: - key: tls.key path: /data/tls/server.key - key: tls.crt path: /data/tls/server.crt
Copied the wrong (non-working) version of the code... :-)
This one works, even when mounting the secret on top of the volume that is needed to preserve the persistent data:
volumeMounts: - name: 389server-data mountPath: '/data/' - name: 389server-certs mountPath: '/data/tls/' readOnly: true
[...]
volumes: - name: 389server-data persistentVolumeClaim: claimName: 389server-data - name: 389server-certs secret: secretName: my-tls-secret items: - key: tls.key path: server.key - key: tls.crt path: server.crt
-- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de
B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Sincerely,
William Brown
Senior Software Engineer, Identity and Access Management SUSE Labs, Australia
Hi William,
On 23.03.22 at 22:48 William Brown wrote:
Cool, so it sounds like you have it all working then?
Yes, it seems like it. I.e. it is not erroring out and not crashlooping, I can get a working TLS connection.
I'll submit an SR for the docker file to include the tar executable, as this is needed within Kubernetes to have "kubectl cp" working (copying files to and from the container inside the pod, e.g. copying LDIF files or similar).
The next thing I wanna tackle is running without root permissions.
And then I need to figure out how to actually get data into that server. I'll open up another thread for that.
Kind Regards, Johannes
Sounds good, happy to have helped!
On 24 Mar 2022, at 17:48, Johannes Kastl kastl@b1-systems.de wrote:
Hi William,
On 23.03.22 at 22:48 William Brown wrote:
Cool, so it sounds like you have it all working then?
Yes, it seems like it. I.e. it is not erroring out and not crashlooping, I can get a working TLS connection.
I'll submit an SR for the docker file to include the tar executable, as this is needed within Kubernetes to have "kubectl cp" working (copying files to and from the container inside the pod, e.g. copying LDIF files or similar).
The next thing I wanna tackle is running without root permissions.
And then I need to figure out how to actually get data into that server. I'll open up another thread for that.
Kind Regards, Johannes
-- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de
B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Sincerely,
William Brown
Senior Software Engineer, Identity and Access Management SUSE Labs, Australia
389-users@lists.fedoraproject.org