Okay, I double checked the timestamps and read out the certs from the pki-tomcat alias NSSDB and realised that this is not the case, it was last modified exactly when the new certs were issued and it is perfectly in sync with with the certs in LDAP.

Then I'm still stumped why my freeipa cert list looks like: https://i.imgur.com/JjgoWTy.png in the web frontend - anyone have an idea?






On Mon, 24 Jun 2024 at 16:07, Johannes Falke <johannesfalke@gmail.com> wrote:
Dear Alexander,

On Wednesday, 21 December 2022 10:10 a.m., Martin (Lists) wrote:
Hallo all
I have a strange issue with one of my ipa servers. after an upgrade from
fedora 35 to fedora 37 the ipa-server-upgrade failed on the pki-tomcat
part. The ipaupgrade.log says:
2022-12-21T15:27:52Z INFO Migrating profile 'caECFullCMCSharedTokenCert'
2022-12-21T15:27:52Z DEBUG request GET
https://ipa1.server.org:8443/ca/rest/account/login
2022-12-21T15:27:52Z DEBUG request body ''
2022-12-21T15:27:52Z DEBUG response status 404
2022-12-21T15:27:52Z DEBUG response headers Content-Type:
text/html;charset=utf-8
Content-Language: de
Content-Length: 795
Date: Wed, 21 Dec 2022 15:27:52 GMT
2022-12-21T15:27:52Z DEBUG response body (decoded): b'<!doctype
html><html lang="de"><head><title>HTTP Status 404
\xe2\x80\x93 nicht
gefunden</title><style
type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3,
b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
{font-size:16px;
} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line
{height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
Status 40
4 \xe2\x80\x93 nicht gefunden</h1><hr class="line"
/><p><b>Type</b>
Status Report</p><p><b>Message</b> The requested resource
[&#47;ca&#47;rest&#47;account
&#47;login] is not available</p><p><b>Beschreibung</b> The
origin server
did not find a current representation for the target resource or is not
willing to
disclose that one exists.</p><hr class="line" /><h3>Apache
Tomcat/9.0.68</h3></body></html>'
2022-12-21T15:27:52Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2022-12-21T15:27:52Z DEBUG   File
"/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in
execute
    return_value = self.run()
                   ^^^^^^^^^^
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run    server.upgrade()
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",
line 2061, in upgrade    upgrade_configuration()
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",
line 1914, in upgrade_configuration    ca_enable_ldap_profile_subsystem(ca)
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",
line 458, in ca_enable_ldap_profile_subsystem    
cainstance.migrate_profiles_to_ldap()
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py",
line 2155, in migrate_profiles_to_ldap    
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File
"/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py",
line 2209, in _create_dogtag_profile    with api.Backend.ra_certprofile
as profile_api:
  File "/usr/lib/python3.11/site-packages/ipaserver/plugins/dogtag.py",
line 1211, in __enter__    raise
errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST
API'))
2022-12-21T15:27:52Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
The catalina logfile says:
21-Dec-2022 16:27:26.946 SCHWERWIEGEND [main]
org.apache.catalina.core.StandardContext.startInternal One or more
listeners failed to start. Full details will be found in the appropriate
container log file
21-Dec-2022 16:27:26.948 SCHWERWIEGEND [main]
org.apache.catalina.core.StandardContext.startInternal Context [/ca]
startup failed due to previous errors
the CA debug log file says:
2022-12-21 16:27:26 [main] FINE: LdapBoundConnection: Connecting to
ipa1.server.org:636 with client cert auth
2022-12-21 16:27:26 [main] FINE:
ldapconn/PKISocketFactory.makeSSLSocket: begins
2022-12-21 16:27:26 [main] FINE: SignedAuditLogger: event
CLIENT_ACCESS_SESSION_ESTABLISH
2022-12-21 16:27:26 [main] SEVERE: Unable to create socket:
java.net.ConnectException: Verbindungsaufbau abgelehnt
with many java traceback errors following. directory server is running
at this time and there is no connection reported at the given time.
ipa-healthceck does not give anny errors or warnings. Re-starting the
pki-tomcat server manually afterwards ist working fine and does not give
any errors. starting ipa in force mode gives no errors as well. What can
I do?
Regards
Martin

FWIW, I used the hyperkitty web reply link, which gives a direct mailto link and no option to add anything, and unfortunately didn't realize the context would be missing. Sorry about that (and for the double reply spam).
I was replying to https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/message/5PC566RTR2XDCSV5MYLM7QJZBXART535/ (also quoted at the start of this e-mail this time).

I see the exact same error as above upgrading either F35->F36 or F35->F37 (and I also tried 37->38 on top in case it is somehow resolved by some later patch).
I am currently on F35/freeipa-server 4.9.11. And ipa-healthcheck finds no issues (besides missing URI records). It's a single master server.

The second error mentioned later in the thread (55-pbacmemberof.update) for me does in fact show up in an earlier upgrade to reach Fedora 35/freeipa-server 4.9.11. However, the upgrade returned exit 0 anyway on F34->F35. And the error does not occur again on the later attempted updates F35->F36 or F35->F37. So I think it's probably unrelated.

On F35 everything *appears* to be working fine, no expired certs, except for the fact that the freeipa web interface seems to be showing 
1) a number of expired certs (old ones)
2) a number of greyed-out certs without name (and higher serial # - the actual renewed ones)
getcert list shows all certs as normal (I assume it grabs them from LDAP) - but I see that /etc/pki/pki-tomcat/alias NSSDB last modification date is from before their issuing - so it seems it was not updated properly for some reason.

I suspect this might be related to the upgrade failure I am seeing. Is there some easy way to refresh the pki-tomcat alias NSSDB without reissuing certs from those stored in LDAP?


Best wishes,
Johannes

On Mon, 24 Jun 2024 at 08:44, Alexander Bokovoy <abokovoy@redhat.com> wrote:
On Няд, 23 чэр 2024, Johannes Falke via FreeIPA-users wrote:
>   How did you actually manage to resolve this issue? I'm seeing the same       
>   thing trying to upgrade either f35->f37 or f35->f36 (and NO ldap errors).   
>   On f35, freeipa says it's healthy.                                           

Just a reminder: this is a mailing list, not a forum. If you are using
lists.fedorahosted.org web interface, make sure to check the box to
quote the email you are answering to because otherwise this message
appears without a reference to the thread you are trying to reply to.

As such, this email contains no reference to existing discussion of the
same topic and it is literally impossible to identify what do you mean
by 'resolve this issue'.

Also, lists.fedorahosted.org web interface gives you a way to search the
list archives and see the whole discussion there. It should help in
searching for successful resolutions of many problems. If you still
cannot find a solution, please make sure to include enough details in
your new email thread to provide information that will allow list
members to help you.




--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland