Alexander and others who care about dnssec:
Given the ongoing problems with opendnssec/libp11 and the many freeipa routines and resources dedicated to working around it, has bind9's native dnssec implementation improved to the point we can greatly reduce the freeipa package count by just using bind's dnssec provisions? What's the roadmap on that?
My dnssec database was corrupted by a recent freeipa upgrade that reverted to upstream code I patched locally some time ago. The upstream doesn't really comprehend/value/test multi-concurrent thread machine speed key/context/pin updates as deployed by opendnssec / softhsm2. Even after I fixed the database by rolling keys and identifying corrupt files, etc, etc , the 'new and improved' upstream code corrupted it again within minutes. So I've disabled dnssec entirely as money to fund those minutes has run out.
I suspect if you only have one dnssec enabled domain, it may work for you as is. But beyond that... you are on borrowed time.
Gave it all I could, hope someone else fixes it (or provides some $ for me to do it...)
Best to all
Harry
On to, 16 joulu 2021, Harry G. Coin via FreeIPA-users wrote:
Alexander and others who care about dnssec:
Given the ongoing problems with opendnssec/libp11 and the many freeipa routines and resources dedicated to working around it, has bind9's native dnssec implementation improved to the point we can greatly reduce the freeipa package count by just using bind's dnssec provisions? What's the roadmap on that?
We did a research some time ago and a result was that native DNSSEC implementation in BIND required some tooling that would have been an equivalent of what is being done with SoftHSM/OpenDNSSEC:
The BIND / DNSSec guide is rather naive when it comes to storing private keys: https://bind9.readthedocs.io/en/latest/dnssec-guide.html#key-storage
Quoting the guide: "Ideally, private keys should be stored offline, in secure devices such as a smart card. Operationally, however, this creates certain challenges, since the private key is needed to create RRSIG resource records, and it is a hassle to bring the private key out of storage every time the zone file changes or signatures expire. A common approach to strike the balance between security and practicality is to have two sets of keys (...) For example, a KSK private key stored on a USB flash drive that is kept in a fireproof safe, only brought online once a year to sign a new pair of ZSKs, combined with a ZSK private key stored on the network file system and available for routine use, may be a good balance between operational flexibility and security."
Effectively, this is a repeatance of what OpenDNSSEC provides if you'd look into making HSM-backed keys.
My dnssec database was corrupted by a recent freeipa upgrade that reverted to upstream code I patched locally some time ago. The upstream doesn't really comprehend/value/test multi-concurrent thread machine speed key/context/pin updates as deployed by opendnssec / softhsm2. Even after I fixed the database by rolling keys and identifying corrupt files, etc, etc , the 'new and improved' upstream code corrupted it again within minutes. So I've disabled dnssec entirely as money to fund those minutes has run out.
Probably a good way going forward would be to have an isolated example with OpenDNSSEC/SoftHSMv2 alone that shows this set of problems and work with that to identify the issue. Right now, it seems, there is no a single place with end to end description of the problem.
I suspect if you only have one dnssec enabled domain, it may work for you as is. But beyond that... you are on borrowed time.
Gave it all I could, hope someone else fixes it (or provides some $ for me to do it...)
Best to all
Harry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 12/16/21 11:10 PM, Alexander Bokovoy wrote:
On to, 16 joulu 2021, Harry G. Coin via FreeIPA-users wrote:
Alexander and others who care about dnssec:
Given the ongoing problems with opendnssec/libp11 and the many freeipa routines and resources dedicated to working around it, has bind9's native dnssec implementation improved to the point we can greatly reduce the freeipa package count by just using bind's dnssec provisions? What's the roadmap on that?
We did a research some time ago and a result was that native DNSSEC implementation in BIND required some tooling that would have been an equivalent of what is being done with SoftHSM/OpenDNSSEC:
The BIND / DNSSec guide is rather naive when it comes to storing private keys: https://bind9.readthedocs.io/en/latest/dnssec-guide.html#key-storage
Quoting the guide: "Ideally, private keys should be stored offline, in secure devices such as a smart card. Operationally, however, this creates certain challenges, since the private key is needed to create RRSIG resource records, and it is a hassle to bring the private key out of storage every time the zone file changes or signatures expire. A common approach to strike the balance between security and practicality is to have two sets of keys (...) For example, a KSK private key stored on a USB flash drive that is kept in a fireproof safe, only brought online once a year to sign a new pair of ZSKs, combined with a ZSK private key stored on the network file system and available for routine use, may be a good balance between operational flexibility and security."
Effectively, this is a repeatance of what OpenDNSSEC provides if you'd look into making HSM-backed keys.
My dnssec database was corrupted by a recent freeipa upgrade that reverted to upstream code I patched locally some time ago. The upstream doesn't really comprehend/value/test multi-concurrent thread machine speed key/context/pin updates as deployed by opendnssec / softhsm2. Even after I fixed the database by rolling keys and identifying corrupt files, etc, etc , the 'new and improved' upstream code corrupted it again within minutes. So I've disabled dnssec entirely as money to fund those minutes has run out.
Probably a good way going forward would be to have an isolated example with OpenDNSSEC/SoftHSMv2 alone that shows this set of problems and work with that to identify the issue. Right now, it seems, there is no a single place with end to end description of the problem.
Thinking about it for a few days, given ICANN's call for full dnssec deployment, and the fact that dns-over-tls only provides improved results between end-user and server (nothing regarding confidence about server data integrity): I think it's fair to ask the freeipa community whether now is the time to re-assess the reasons why, after so many years, freeipa's caution 'dnssec support is experimental' remains (https://www.freeipa.org/page/Howto/DNSSEC).
Alexander correctly suggests that creating an isolated example that highlights an issue be done. Notice that becomes much easier to ask for than produce when the problem is a race condition among threads that manifests itself in a library that's, what, 4 or 5 packages deep from the launching daemons, each package maintained by different groups with different dev priorities abilities regarding multithreading. Remember 'softhsm' is an add-on to a 'smart card' technology most use not at machine speeds but as humans insert and remove physical security devices. And significantly -- almost always just one at a time per host. Multi-processing, multi-threading and re-entrancy (such as during bulk reconsideration of multiple zsk's, ksks and sqlite db contents while other threads are assessing master-slave sync issues) are not top of softhsm/libp11 dev minds and quite understandably. Just a quick look at libp11's eng_back.c will show even a casual reader externally accessible routines could find essential variables in an inconsistent state from a multi-threading perspective ( e.g. ctx->pin ctx->pin_length).
Moreover, since freeipa last reassessed the matter in the big picture, bind's dnssec shortcomings have been improved. From a maintainability perspective, there's obviously something to be said for the same results being delivered by fewer dependencies and fewer points of failure in sub packages. After all, in the present case, to cite just one example, we have freeipa's dnssec routines (call it 'top level') running code quite literally to 'fix' cert/key file permissions so that softhsm, opendnssec, libp11, openssl-pkcs11, and, bind, can interoperate with dyndb-ldap and, last, freeipa. While that's obviously necessary to deliver function, I don't think anybody would call it 'their first design choice'.
I respectfully ask freeipa folks to re-assess whether bind's implementation has improved to the point the, ah, 'unfortunate' documentation Alexander points at might be less important than the tenuous reliability we experience now owing to reliance on so many levels of differently maintained packages.
Whether this particular issues persuades the freeipa community or does not: At least to me, most significantly, it has been many years that freeipa has written 'dnssec is experimental'. After so long, it's fair to ask what is lacking that dnssec can't be considered 'production ready' and take steps to cause that to occur.
Best to all
Harry Coin
Hi Harry,
On ma, 20 joulu 2021, Harry G. Coin wrote:
On 12/16/21 11:10 PM, Alexander Bokovoy wrote:
On to, 16 joulu 2021, Harry G. Coin via FreeIPA-users wrote:
Alexander and others who care about dnssec:
Given the ongoing problems with opendnssec/libp11 and the many freeipa routines and resources dedicated to working around it, has bind9's native dnssec implementation improved to the point we can greatly reduce the freeipa package count by just using bind's dnssec provisions? What's the roadmap on that?
We did a research some time ago and a result was that native DNSSEC implementation in BIND required some tooling that would have been an equivalent of what is being done with SoftHSM/OpenDNSSEC:
The BIND / DNSSec guide is rather naive when it comes to storing private keys: https://bind9.readthedocs.io/en/latest/dnssec-guide.html#key-storage
Quoting the guide: "Ideally, private keys should be stored offline, in secure devices such as a smart card. Operationally, however, this creates certain challenges, since the private key is needed to create RRSIG resource records, and it is a hassle to bring the private key out of storage every time the zone file changes or signatures expire. A common approach to strike the balance between security and practicality is to have two sets of keys (...) For example, a KSK private key stored on a USB flash drive that is kept in a fireproof safe, only brought online once a year to sign a new pair of ZSKs, combined with a ZSK private key stored on the network file system and available for routine use, may be a good balance between operational flexibility and security."
Effectively, this is a repeatance of what OpenDNSSEC provides if you'd look into making HSM-backed keys.
My dnssec database was corrupted by a recent freeipa upgrade that reverted to upstream code I patched locally some time ago. The upstream doesn't really comprehend/value/test multi-concurrent thread machine speed key/context/pin updates as deployed by opendnssec / softhsm2. Even after I fixed the database by rolling keys and identifying corrupt files, etc, etc , the 'new and improved' upstream code corrupted it again within minutes. So I've disabled dnssec entirely as money to fund those minutes has run out.
Probably a good way going forward would be to have an isolated example with OpenDNSSEC/SoftHSMv2 alone that shows this set of problems and work with that to identify the issue. Right now, it seems, there is no a single place with end to end description of the problem.
Thinking about it for a few days, given ICANN's call for full dnssec deployment, and the fact that dns-over-tls only provides improved results between end-user and server (nothing regarding confidence about server data integrity): I think it's fair to ask the freeipa community whether now is the time to re-assess the reasons why, after so many years, freeipa's caution 'dnssec support is experimental' remains (https://www.freeipa.org/page/Howto/DNSSEC).
Alexander correctly suggests that creating an isolated example that highlights an issue be done. Notice that becomes much easier to ask for than produce when the problem is a race condition among threads that manifests itself in a library that's, what, 4 or 5 packages deep from the launching daemons, each package maintained by different groups with different dev priorities abilities regarding multithreading. Remember 'softhsm' is an add-on to a 'smart card' technology most use not at machine speeds but as humans insert and remove physical security devices. And significantly -- almost always just one at a time per host. Multi-processing, multi-threading and re-entrancy (such as during bulk reconsideration of multiple zsk's, ksks and sqlite db contents while other threads are assessing master-slave sync issues) are not top of softhsm/libp11 dev minds and quite understandably. Just a quick look at libp11's eng_back.c will show even a casual reader externally accessible routines could find essential variables in an inconsistent state from a multi-threading perspective ( e.g. ctx->pin ctx->pin_length).
Moreover, since freeipa last reassessed the matter in the big picture, bind's dnssec shortcomings have been improved. From a maintainability perspective, there's obviously something to be said for the same results being delivered by fewer dependencies and fewer points of failure in sub packages. After all, in the present case, to cite just one example, we have freeipa's dnssec routines (call it 'top level') running code quite literally to 'fix' cert/key file permissions so that softhsm, opendnssec, libp11, openssl-pkcs11, and, bind, can interoperate with dyndb-ldap and, last, freeipa. While that's obviously necessary to deliver function, I don't think anybody would call it 'their first design choice'.
I respectfully ask freeipa folks to re-assess whether bind's implementation has improved to the point the, ah, 'unfortunate' documentation Alexander points at might be less important than the tenuous reliability we experience now owing to reliance on so many levels of differently maintained packages.
Whether this particular issues persuades the freeipa community or does not: At least to me, most significantly, it has been many years that freeipa has written 'dnssec is experimental'. After so long, it's fair to ask what is lacking that dnssec can't be considered 'production ready' and take steps to cause that to occur.
"FreeIPA folks" are community members. Some of them work for Red Hat, some aren't. If you want to focus on a certain area, you are free to do so. Even if Red Hat right now is not considering DNSSEC support as priority (hence keeping it in experimental state) and not adding more resources to solve that problem, community members can step up and contribute.
Traditionally there were not so many contributions in these core areas from outside, for various reasons. Our community has many more administrators than developers. However, our current BIND support is kept in a good shape also thanks to community contributors (Stas did a great part of that, thank you! A lot of Ubuntu users contributed with bug reports too). I certainly hope we can get DNSSEC support to a better shape but that needs more involvement from everyone who can help, not just core team.
Myself, I am deep in making FreeIPA to work with upcoming MIT Kerberos 1.20 which changes its KDB interface in a serious way to step up few very important security properties of Kerberos use in real life deployments -- this is a result of several years of improving interoperability with Active Directory but not only that. Latest Microsoft's security release for Windows Server which fixed at least four CVEs reported by Samba Team members made a number of changes that we hope will allow open source implementations to get much better semantics around authentication and authorization. And this needs our work 'now' (my team was busy with it for last half a year, at least). I am simply not able to look into DNSSEC in a serious way before this and other prioritised tasks are done.
So if you feel you can contribute to DNSSEC support improvements, please do so. We typically work by creating a design document by iterating through it in reviews, identifying the affected components and then step by step implementing the changes. This is a lengthy process but it doesn't really matter who'd initiate and contribute to it.
Next few weeks are special for Red Hatters, though. Traditionally, Red Hat encourages employees to spend time with their families around Christmas and new year celebrations across the world. Most of Red Hat contributors to FreeIPA and related open source projects are either already on their vacation or will be going for vacation soon. I myself will start that later this week and be back on the second week of January. I might spent some of my family time looking at Fedora and FreeIPA but this year was so pressing that a proper isolation of an online presence might be a good way to recharge.
freeipa-users@lists.fedorahosted.org