People,
Date: Mon, 21 Mar 2016 15:02:02 -0600 From: Chris Murphy lists@colorremedies.com To: phil@pricom.com.au, Community support for Fedora users users@lists.fedoraproject.org Subject: Re: Discourse - DeviceMapper causing corruption? Message-ID: CAJCQCtSCPQRQsecq0+dzEKdNjEDfBzaDavOOV_e_ARL6LjgkDg@mail.gmail.com Content-Type: text/plain; charset=UTF-8
On Mon, Mar 21, 2016 at 1:03 PM, Philip Rhoades phil@pricom.com.au wrote:
People,
I had a couple of issues to sort out with installing the Docker Discourse app and while that was being done people made these comments:
"Devicemapper is non starter, fails spectacularly under load and causes corruption. We block setup if we detect devicemapper. You need aufs or another better supported docker filesystem."
- which was not true - it did install without resorting to aufs.
also:
"Redhat team get very upset when we mention that it just does not work for us, but release after release they say there are no bugs left, and each time we keep seeing Discourse users complain about corruption due to device mapper."
Any comments?
On the one hand, I'd say, talk is cheap, show me the bug.
On the other hand, having blown up thinly provisioned pools quite a few times early on, I basically couldn't handle how non-deterministic the blow ups were, and how difficult the LVM and devicemapper tools are to understand the state of the pool, and fix it. So I gave up and went down the Btrfs path just because I understand it better, and there are only so many rabbit holes one has time and interest to go down.
For a good bug report you either need a way to debug it, i.e. you need familiarity with that thing you are debugging. Or you need a concise set of reproduce steps (and a list of versions) so someone else, like a dev or more experienced user, can reproduce that environment and steps and debug it. Without one of those two things at least, I think it's unreasonable to say "hi this is busted" and then just walk away.
aufs is just not workable in my opinion because it's not in the mainline kernel. So any sense of portability can't be that important if you're going to depend on aufs. There are plenty of bugs to go around at this point, between overlay(fs), btrfs, and devicemapper although I think by now most of the devicemapper stuff is probably the more stable of the three (?) just a guess really. I had 500+ container states as Btrfs subvolumes and never ran into a Btrfs related problem though. What are the CoreOS folks using? Overlay or dm?
I had another response on the Discourse forum:
"It's probably this bug, devicemapper seems to be unable to free deleted files until you delete the container:
https://github.com/docker/docker/issues/188672
Which is catastrophic for any long-lived server that uses temporary files in any capacity at all."
Does that also make sense to more clued-up tech people than me?
Regards,
Phil.
On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades phil@pricom.com.au wrote:
I had another response on the Discourse forum:
"It's probably this bug, devicemapper seems to be unable to free deleted files until you delete the container:
https://github.com/docker/docker/issues/188672
Which is catastrophic for any long-lived server that uses temporary files in any capacity at all."
Does that also make sense to more clued-up tech people than me?
That link doesn't work.
There was a thinp trim bug, but is been squashed on Fedora for over a year.
Chris Murphy
Chris,
On 2016-04-04 03:25, Chris Murphy wrote:
On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades phil@pricom.com.au wrote:
I had another response on the Discourse forum:
"It's probably this bug, devicemapper seems to be unable to free deleted files until you delete the container:
https://github.com/docker/docker/issues/188672
Which is catastrophic for any long-lived server that uses temporary files in any capacity at all."
Does that also make sense to more clued-up tech people than me?
That link doesn't work.
Sorry, it looks like copying and pasting added a "2" to the link (the number of clicks on the link on the forum) - it should be:
https://github.com/docker/docker/issues/18867
There was a thinp trim bug, but is been squashed on Fedora for over a year.
No mention of "thinp" on the page . .
Thanks,
Phil.
On Sun, Apr 3, 2016 at 5:09 PM, Philip Rhoades phil@pricom.com.au wrote:
Chris,
On 2016-04-04 03:25, Chris Murphy wrote:
On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades phil@pricom.com.au wrote:
I had another response on the Discourse forum:
"It's probably this bug, devicemapper seems to be unable to free deleted files until you delete the container:
https://github.com/docker/docker/issues/188672
Which is catastrophic for any long-lived server that uses temporary files in any capacity at all."
Does that also make sense to more clued-up tech people than me?
That link doesn't work.
Sorry, it looks like copying and pasting added a "2" to the link (the number of clicks on the link on the forum) - it should be:
https://github.com/docker/docker/issues/18867
There was a thinp trim bug, but is been squashed on Fedora for over a year.
No mention of "thinp" on the page . .
It's using loop devices which is only the default in Docker because it's easy for them to setup out of the box, where the recommended (at least on RHEL, CentOS, Fedora) XFS on dmthin requires a.) an LVM VG with free extents available, and b.) systemctl enable/start docker-storage-setup to allocate bit more or less than half of those free extents to the creation of a dmthin pool in which the containers exist.
What I can't tell you off hand is whether there's something in Docker land that automatically does discard or fstrim periodically (like on container deletion) to actually revert the logical extents that are freed up, back into the dmthin pool. https://github.com/docker/docker/issues/12902
Looks like something is happening, I'm just not sure what.
On Sun, Apr 3, 2016 at 8:20 PM, Chris Murphy lists@colorremedies.com wrote:
On Sun, Apr 3, 2016 at 5:09 PM, Philip Rhoades phil@pricom.com.au wrote:
Chris,
On 2016-04-04 03:25, Chris Murphy wrote:
On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades phil@pricom.com.au wrote:
I had another response on the Discourse forum:
"It's probably this bug, devicemapper seems to be unable to free deleted files until you delete the container:
https://github.com/docker/docker/issues/188672
Which is catastrophic for any long-lived server that uses temporary files in any capacity at all."
Does that also make sense to more clued-up tech people than me?
That link doesn't work.
Sorry, it looks like copying and pasting added a "2" to the link (the number of clicks on the link on the forum) - it should be:
https://github.com/docker/docker/issues/18867
There was a thinp trim bug, but is been squashed on Fedora for over a year.
No mention of "thinp" on the page . .
It's using loop devices which is only the default in Docker because it's easy for them to setup out of the box, where the recommended
The other thing is Docker on loop device is slow. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-s...
cat /usr/bin/docker-storage-setup
The gist is that you want a Volume Group with a decent chunk of free extents. If you do that, then 'systemctl enable docker-storage-setup' and then 'systemctl start docker-storage-setup' should do the trick. You can check it with 'systemctl status docker-storage-setup' to see if it worked or if there are errors listed.
If you can make semi-decent sense out of that bash script, and you aren't already familiar with Btrfs, then you should use dmthin and the docker-storage-setup script. The biggest gotcha is that lvs/lvdisplay will show the resulting thin pool, but you can't use LVM tools to manage it. It's strictly a device mapper thin pool. Fortunately, you shouldn't have to directly interact with it though.
The main reason why I prefer Btrfs is simply because I know where the bodies are buried, and I don't know where the landmines are in the device mapper black box. But I'd only recommend Btrfs instead if you are already somewhat familiar with it, and are comfortable with either baby sitting the occasional need to balance, or adapting the openSUSE scripts that can be set on a systemd timer for this purpose.
-
Chris Murphy -
Philip Rhoades