RHN-ORG-TRUSTED-SSL-CERT empty in /usr/share/susemanager/salt/certs

[MaxHerrmannSVA]

Problem description

When migrating Uyuni from 2024.08 to 2024.12 I ran into an error where the RHN-ORG-TRUSTED-SSL-CERT was not migrated to /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT.

Due to this, the states under /usr/share/susemanager/salt/certs did not fail, but emptied the local CA File on all clients and no client was able to get updates with a "SSL certificate problem: unable to get local issuer certifiacte" error.

After copying the cert to the /usr/share/susemanager/salt/certs directory and executing the highstate again, the certs got rolled out again, and updates are working
cp /etc/pki/trust/anchors/LOCAL-RHN-ORG-TRUSTED-SSL-CERT /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT

The Main Problem is, that this "fix" is not reboot persistant, so everytime we restart the Uyuni Server the /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT is emtpy again and we have to manually copy the Filecontent

Steps to reproduce

1. check filesize in /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT
2. reboot
3. check filesize in /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT again

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : @System
Name           : Uyuni-Server-release
Version        : 2024.12-241000.225.1.uyuni5
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : unknown
Installed Size : 1.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : Uyuni-Server-release-2024.12-241000.225.1.uyuni5.src
Upstream URL   : https://www.uyuni-project.org/
Summary        : Uyuni Server
Description    :
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

Useful logs

I found nothing in the logs...

Additional information

I installed a fresh Testserver to see if this is a generall problem.
There i discovered that the Cert is copied there while uyuni is starting and emtpy on default...

uyuni-server:~ # reboot
...

uyuni-server:~ # mgrctl term
...
uyuni-server:/ # date && ls -la /usr/share/susemanager/salt/certs
Fri Feb  7 10:54:16 AM CET 2025
total 24
drwxr-xr-x. 1 root root 186 Dec 19 14:07 .
drwxr-xr-x. 1 root root 702 Dec 19 14:07 ..
-rw-r--r--. 1 root root 335 Dec 13 15:12 debian.sls
-rw-r--r--. 1 root root 430 Dec 13 15:12 init.sls
lrwxrwxrwx. 1 root root  10 Dec 13 15:12 openeuler.sls -> redhat.sls
-rw-r--r--. 1 root root 623 Dec 13 15:12 redhat.sls
-rw-r--r--. 1 root root   0 Dec 19 14:07 RHN-ORG-TRUSTED-SSL-CERT
-rw-r--r--. 1 root root 835 Dec 13 15:12 suse.sls
-rw-r--r--. 1 root root 395 Dec 13 15:12 update-multi-cert.sh
uyuni-server:/ # date && ls -la /usr/share/susemanager/salt/certs
Fri Feb  7 10:54:17 AM CET 2025
total 28
drwxr-xr-x. 1 root root   48 Dec 19 14:07 .
drwxr-xr-x. 1 root root   10 Dec 19 14:07 ..
-rw-r--r--. 1 root root  335 Dec 13 15:12 debian.sls
-rw-r--r--. 1 root root  430 Dec 13 15:12 init.sls
lrwxrwxrwx. 1 root root   10 Dec 13 15:12 openeuler.sls -> redhat.sls
-rw-r--r--. 1 root root  623 Dec 13 15:12 redhat.sls
-rw-r--r--. 1 root root 2602 Feb  7 10:54 RHN-ORG-TRUSTED-SSL-CERT
-rw-r--r--. 1 root root  835 Dec 13 15:12 suse.sls
-rw-r--r--. 1 root root  395 Dec 13 15:12 update-multi-cert.sh
uyuni-server:/ #

I guess that on my migrated installation we have a problem with the mechanism that copies the certfile to the salt states...

Any ideas how to fix this?

Thanks for your help!
Max

[mcalmer]

When the container startup a service is executed called "uyuni-update-config.service".
This is preparing the environment and one task is to copy the CA to that place with the exact command you pasted:

https://github.com/uyuni-project/uyuni/blob/master/spacewalk/admin/uyuni-update-config#L71-L76

Please check the logs (journal) if you see an error message when the container starts.

[agraul]

@MaxHerrmannSVA in case it helps: you can use mgrctl exec 'journalctl -u uyuni-update-config.service' to print the logs of this service while the container is running.

[MaxHerrmannSVA]

Hi,
first thanks for your responses!

Sadly the Unit dont print anything to journald...

uyuni-server:# journalctl -u uyuni-update-config.service
-- No entries --

I digged a bit deeper into the Unit target wants, but this seems fine to me...

uyuni-server:/ # ls -la /etc/systemd/system/multi-user.target.wants/
total 148
drwxr-xr-x. 1 root root 1306 Jan 17 16:03 .
drwxr-xr-x. 1 root root   40 Feb  7 11:06 ..
lrwxrwxrwx. 1 root root   40 Jan 17 16:03 apache2.service -> /usr/lib/systemd/systemd/apache2.service
lrwxrwxrwx. 1 root root   41 Jan 17 16:03 apparmor.service -> /usr/lib/systemd/systemd/apparmor.service
lrwxrwxrwx. 1 root root   39 Jan 17 16:03 auditd.service -> /usr/lib/systemd/systemd/auditd.service
lrwxrwxrwx. 1 root root   52 Dec 19 14:07 billing-data-service.service -> /usr/lib/systemd/system/billing-data-service.service
lrwxrwxrwx. 1 root root   54 Jan 17 16:03 btrfsmaintenance-refresh.path -> /usr/lib/systemd/systemd/btrfsmaintenance-refresh.path
-rw-r--r--. 1 root root  330 Nov 19 10:01 check-mk-agent-async.service
lrwxrwxrwx. 1 root root   40 Jan 17 16:03 chronyd.service -> /usr/lib/systemd/systemd/chronyd.service
-rw-r--r--. 1 root root  797 Nov 19 10:01 cmk-agent-ctl-daemon.service
lrwxrwxrwx. 1 root root   41 Jan 17 16:03 cobblerd.service -> /usr/lib/systemd/systemd/cobblerd.service
lrwxrwxrwx. 1 root root   37 Jan 17 16:03 cron.service -> /usr/lib/systemd/systemd/cron.service
lrwxrwxrwx. 1 root root   34 Jan 17 16:03 cups.path -> /usr/lib/systemd/systemd/cups.path
lrwxrwxrwx. 1 root root   42 Jan 17 16:03 firewalld.service -> /usr/lib/systemd/systemd/firewalld.service
lrwxrwxrwx. 1 root root   43 Jan 17 16:03 irqbalance.service -> /usr/lib/systemd/systemd/irqbalance.service
-rw-r--r--. 1 root root  419 Nov 21  2022 jabberd.service
lrwxrwxrwx. 1 root root   44 Jan 17 16:03 kbdsettings.service -> /usr/lib/systemd/systemd/kbdsettings.service
lrwxrwxrwx. 1 root root   39 Jan 17 16:03 mcelog.service -> /usr/lib/systemd/systemd/mcelog.service
lrwxrwxrwx. 1 root root   37 Jan 17 16:03 nscd.service -> /usr/lib/systemd/systemd/nscd.service
-rw-r--r--. 1 root root  419 Sep 28  2022 osa-dispatcher.service
lrwxrwxrwx. 1 root root   40 Jan 17 16:03 postfix.service -> /usr/lib/systemd/systemd/postfix.service
lrwxrwxrwx. 1 root root   43 Jan 17 16:03 postgresql.service -> /usr/lib/systemd/systemd/postgresql.service
lrwxrwxrwx. 1 root root   56 Dec 19 14:07 prometheus-node_exporter.service -> /usr/lib/systemd/system/prometheus-node_exporter.service
lrwxrwxrwx. 1 root root   46 Jan 17 16:03 purge-kernels.service -> /usr/lib/systemd/systemd/purge-kernels.service
lrwxrwxrwx. 1 root root   41 Jan 17 16:03 remote-fs.target -> /usr/lib/systemd/systemd/remote-fs.target
lrwxrwxrwx. 1 root root   43 Jan 17 16:03 rhn-search.service -> /usr/lib/systemd/systemd/rhn-search.service
lrwxrwxrwx. 1 root root   40 Jan 17 16:03 rsyslog.service -> /usr/lib/systemd/systemd/rsyslog.service
lrwxrwxrwx. 1 root root   41 Jan 17 16:03 salt-api.service -> /usr/lib/systemd/systemd/salt-api.service
lrwxrwxrwx. 1 root root   44 Jan 17 16:03 salt-master.service -> /usr/lib/systemd/systemd/salt-master.service
lrwxrwxrwx. 1 root root   39 Jan 17 16:03 smartd.service -> /usr/lib/systemd/systemd/smartd.service
-rw-r--r--. 1 root root  571 Aug  1  2024 spacewalk.target
lrwxrwxrwx. 1 root root   37 Jan 17 16:03 sshd.service -> /usr/lib/systemd/systemd/sshd.service
lrwxrwxrwx. 1 root root   36 Dec 19 14:07 sssd.service -> /usr/lib/systemd/system/sssd.service
lrwxrwxrwx. 1 root root   43 Jan 17 16:03 taskomatic.service -> /usr/lib/systemd/systemd/taskomatic.service
lrwxrwxrwx. 1 root root   50 Dec 19 14:07 timezone_alignment.service -> /usr/lib/systemd/system/timezone_alignment.service
lrwxrwxrwx. 1 root root   39 Jan 17 16:03 tomcat.service -> /usr/lib/systemd/systemd/tomcat.service
lrwxrwxrwx. 1 root root   41 Jan 17 16:03 vmtoolsd.service -> /usr/lib/systemd/systemd/vmtoolsd.service
lrwxrwxrwx. 1 root root   39 Jan 17 16:03 wicked.service -> /usr/lib/systemd/systemd/wicked.service
-rw-r--r--. 1 root root  351 May 17  2023 xinetd.service
uyuni-server:/ #

uyuni-server:/ # cat /etc/systemd/system/multi-user.target.wants/spacewalk.target'
[Unit]
Description=Spacewalk
Requires=uyuni-update-config.service
Requires=uyuni-check-database.service
Requires=tomcat.service
Requires=spacewalk-wait-for-tomcat.service
Requires=salt-master.service
Requires=salt-api.service
Requires=spacewalk-wait-for-salt.service
Requires=apache2.service
Requires=rhn-search.service
Requires=cobblerd.service
Requires=taskomatic.service
Requires=spacewalk-wait-for-taskomatic.service
Requires=salt-secrets-config.service
Requires=mgr-websockify.service
Requires=cobbler-refresh-mkloaders.service
 
[Install]
WantedBy=multi-user.target
uyuni-server:/ #

uyuni-server:/ # cat /usr/lib/systemd/system/spacewalk.target
[Unit]
Description=Spacewalk
Requires=uyuni-update-config.service
Requires=uyuni-check-database.service
Requires=tomcat.service
Requires=spacewalk-wait-for-tomcat.service
Requires=salt-master.service
Requires=salt-api.service
Requires=spacewalk-wait-for-salt.service
Requires=apache2.service
Requires=rhn-search.service
Requires=cobblerd.service
Requires=taskomatic.service
Requires=spacewalk-wait-for-taskomatic.service
Requires=salt-secrets-config.service
Requires=mgr-websockify.service
Requires=cobbler-refresh-mkloaders.service
 
[Install]
WantedBy=multi-user.target

The only difference in those file to my fresh test installation is that /etc/systemd/system/multi-user.target.wants/spacewalk.target is no symlink but the file content is the same...

Maybe my colleague ran into #9531 while migrating the system, so maybe this has something to do with it...
We executed the following commands to fix 9531:

# mgrctl term
uyuni-server# cd /etc/systemd/system/multi-user.target.wants
# for i in apache2.service apparmor.service auditd.service btrfsmaintenance-refresh.path chronyd.service
cobblerd.service cron.service cups.path firewalld.service irqbalance.service kbdsettings.service
mcelog.service nscd.service postfix.service postgresql.service purge-kernels.service remote-fs.target
rhn-search.service rsyslog.service salt-api.service salt-master.service smartd.service sshd.service
taskomatic.service tomcat.service vmtoolsd.service wicked.service ; do rm $i ; ln -s
/usr/lib/systemd/systemd/$i $i ; done
# exit
# mgradm stop ; mgradm start