dustymabe reported a new issue against the project: `atomic-wg` that you are following: `` Started the CentOS Atomic Host vagrant box I have lying around on my system. It was the `7.1711` version. It starts fine but after I upgrade docker service doesn't come up. I see two issues:
- docker-storage-setup fails
``` -bash-4.2# systemctl status docker-storage-setup -o cat | tee ● docker-storage-setup.service - Docker Storage Setup Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2018-05-16 01:30:26 UTC; 10min ago Process: 719 ExecStart=/usr/bin/container-storage-setup (code=exited, status=1/FAILURE) Main PID: 719 (code=exited, status=1/FAILURE)
Starting Docker Storage Setup... ERROR: Storage is already configured with devicemapper driver. Can't configure it with overlay2 driver. To override, remove /etc/sysconfig/docker-storage and retry. docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE Failed to start Docker Storage Setup. Unit docker-storage-setup.service entered failed state. docker-storage-setup.service failed. ```
- docker service itself doesn't come up
``` -bash-4.2# systemctl status docker -o cat | tee ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d └─flannel.conf Active: failed (Result: exit-code) since Wed 2018-05-16 01:30:27 UTC; 9min ago Docs: http://docs.docker.com Process: 795 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE) Main PID: 795 (code=exited, status=1/FAILURE)
Starting Docker Application Container Engine... time="2018-05-16T01:30:26.496375725Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" time="2018-05-16T01:30:26.499243981Z" level=info msg="libcontainerd: new containerd process, pid: 925" Error starting daemon: error initializing graphdriver: devicemapper: Non existing device atomicos-docker--pool docker.service: main process exited, code=exited, status=1/FAILURE Failed to start Docker Application Container Engine. Unit docker.service entered failed state. docker.service failed. ```
We probably need to address these issues. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
The issue: `upgrading CAH `7.1711` -> `7.1803` docker daemon fails to start` of project: `atomic-wg` has been assigned to `jasonbrooks` by dustymabe.
dustymabe added a new comment to an issue you are following: `` @jasonbrooks or @miabbott - can you take a look? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
dustymabe added a new comment to an issue you are following: `` related: https://github.com/projectatomic/container-storage-setup/issues/267 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
walters added a new comment to an issue you are following: `` In general, I'm not sure about filing atomic-wg issues that relate to CentOS because we can't generally fix them without going through RHEL, right?
Anyways https://bodhi.fedoraproject.org/updates/FEDORA-2018-03bdc0733a is the Fedora version of the first one. This fix needs to go through the whole downstream thing.
For the pool issue...I think that deserves a bug against docker in RHEL? Though of course that sort of wants to have reproduced it against RHELAH first.
That said it's definitely possible with some of these things that it's CentOS-AH specific as some of the metadata gets important. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
dustymabe added a new comment to an issue you are following: ``
In general, I'm not sure about filing atomic-wg issues that relate to CentOS because we can't generally fix them without going through RHEL, right?
You are right, but I think it's mostly a philosophical question. Filing issues here helps bring awareness to the issues I believe. For me things get too easily lost in bugzilla.
Anyways https://bodhi.fedoraproject.org/updates/FEDORA-2018-03bdc0733a is the Fedora version of the first one. This fix needs to go through the whole downstream thing.
+1, thanks
For the pool issue...I think that deserves a bug against docker in RHEL? Though of course that sort of wants to have reproduced it against RHELAH first.
probably
That said it's definitely possible with some of these things that it's CentOS-AH specific as some of the metadata gets important.
Right. To me this is as good a place as any to "route" issues and find the most effective way to get them resolved. Whether they are specific to Fedora, specific to CentOS, or specific to RHELAH and need their own BZ. Maybe I'm created too much overhead. Maybe not.. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
vgoyal added a new comment to an issue you are following: `` I understand that why docker-storage-setup error message is coming and how upgrade will fix that. What I don't understand is that why docker is complaining that "atomicos-docker--pool" does not exist.
If storage is already setup, then this thin pool should have come up automatically after reboot. If its not there, then we are looking at a different issue.
What's the output of "lvs", "vgs", "pvs" and "lsblk" command on the system. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
dustymabe added a new comment to an issue you are following: ``
I understand that why docker-storage-setup error message is coming and how upgrade will fix that. What I don't understand is that why docker is complaining that "atomicos-docker--pool" does not exist. If storage is already setup, then this thin pool should have come up automatically after reboot. If its not there, then we are looking at a different issue. What's the output of "lvs", "vgs", "pvs" and "lsblk" command on the system.
just tried to reproduce and I only see the first error this time and the docker service itself seems to come up ok.. Maybe a race condition? I can try to play around and see if I can repro. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
dustymabe added a new comment to an issue you are following: `` and just rebooting that same host allows me to reproduce:
``` -bash-4.2# reboot Connection to 192.168.121.193 closed by remote host. Connection to 192.168.121.193 closed. $ $ vagrant ssh Last login: Mon May 21 15:22:08 2018 from 192.168.121.1 [vagrant@vanilla-c7atomic ~]$ [vagrant@vanilla-c7atomic ~]$ sudo su - Last login: Mon May 21 15:22:10 UTC 2018 on pts/0 -bash-4.2# systemctl status docker -o cat | tee ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d └─flannel.conf Active: failed (Result: exit-code) since Mon 2018-05-21 15:25:53 UTC; 9min ago Docs: http://docs.docker.com Process: 806 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE) Main PID: 806 (code=exited, status=1/FAILURE)
Starting Docker Application Container Engine... time="2018-05-21T15:25:52.960602241Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" time="2018-05-21T15:25:52.965747392Z" level=info msg="libcontainerd: new containerd process, pid: 834" Error starting daemon: error initializing graphdriver: devicemapper: Non existing device atomicos-docker--pool docker.service: main process exited, code=exited, status=1/FAILURE Failed to start Docker Application Container Engine. Unit docker.service entered failed state. docker.service failed. -bash-4.2# -bash-4.2# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool atomicos twi-a-t--- 2.68g 0.71 0.33 root atomicos -wi-ao---- <2.93g -bash-4.2# vgs VG #PV #LV #SN Attr VSize VFree atomicos 1 2 0 wz--n- 9.70g <4.07g -bash-4.2# pvs PV VG Fmt Attr PSize PFree /dev/vda2 atomicos lvm2 a-- 9.70g <4.07g -bash-4.2# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 11G 0 disk ├─vda1 252:1 0 300M 0 part /boot └─vda2 252:2 0 9.7G 0 part ├─atomicos-root 253:0 0 3G 0 lvm /sysroot ├─atomicos-docker--pool_tmeta 253:1 0 12M 0 lvm │ └─atomicos-docker--pool 253:3 0 2.7G 0 lvm └─atomicos-docker--pool_tdata 253:2 0 2.7G 0 lvm └─atomicos-docker--pool 253:3 0 2.7G 0 lvm vdb 252:16 0 20G 0 disk -bash-4.2#
``` ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
dustymabe added a new comment to an issue you are following: `` upstream discussion about this race condition: https://github.com/projectatomic/container-storage-setup/issues/267#issuecom... ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/498
atomic@lists.fedoraproject.org