On 07/28/14 at 08:56am, Vivek Goyal wrote:
On Mon, Jul 28, 2014 at 05:13:52PM +0800, WANG Chao wrote:
> On 07/25/14 at 08:01am, Vivek Goyal wrote:
> > On Fri, Jul 25, 2014 at 01:44:40PM +0800, WANG Chao wrote:
> > > On 07/24/14 at 01:35pm, Vivek Goyal wrote:
> > > > On Fri, Jul 25, 2014 at 12:13:45AM +0800, WANG Chao wrote:
> > > > > Now when mount in /etc/fstab fails, systemd would not consider
it as
> > > > > critical and it would continue to boot. In fact, emergency
service is
> > > > > triggered, but not in a isolation mode, and it results in the
emergency
> > > > > service getting shutdown at some point later of the boot
process. We
> > > > > need isolation otherwise we won't see any emergency
service.
> > > > >
> > > > > That is because in kdump initramfs, mount units specified in
/etc/fstab
> > > > > are required by "local-fs.target". When any of these
mounts fails,
> > > > > local-fs.target fails.
> > > > >
> > > > > For kdump initramfs, we need to isolate to emergency service on
any of
> > > > > the mount failure, that said, every service should be stopped
and onlu
> > > > > emergency service would run. But local-fs.target won't
trigger that on
> > > > > its failure. That means in case of mount failure,
local-fs.target also
> > > > > enters failure state, but all the service will continue without
any
> > > > > interruption.
> > > > >
> > > > > After digging looking into source code of
systemd-fstab-generator. I
> > > > > find "x-initrd.mount" using in initramfs mount, will
make the mount
> > > > > units required by "initrd-root-fs.target" rather than
it's used to be
> > > > > "local-fs.target".
> > > > >
> > > > > "initrd-root-fs.target" is suitable to us because if
it fails, it will
> > > > > isolate to emergency service. That means in case of any mount
failure,
> > > > > the emergeny service will start and everything else will stop.
We want
> > > > > this effect because we need to take kdump fail-safe action when
there's
> > > > > a mount failure.
> > > > >
> > > > > >From systemd unit point of view,
"initrd-root-fs.target" has
> > > > > OnFailureIsolate=yes, but "local-fs.target"
doesn't. From
> > > > > systemd.unit(5):
> > > > >
> > > > > OnFailureIsolate=
> > > > > Takes a boolean argument. If true, the unit listed in
OnFailure=
> > > > > will be enqueued in isolation mode, i.e. all units that are
not its
> > > > > dependency will be stopped. If this is set, only a single
unit may
> > > > > be listed in OnFailure=. Defaults to false.
> > > > >
> > > > > NOTE: Harald who contributed "x-initrd.mount" in
systemd, confirmed that
> > > > > this feature will stay.
> > > > >
> > > > > Signed-off-by: WANG Chao <chaowang(a)redhat.com>
> > > > > ---
> > > > > mkdumprd | 6 ++++++
> > > > > 1 file changed, 6 insertions(+)
> > > > >
> > > > > diff --git a/mkdumprd b/mkdumprd
> > > > > index ba35800..9ee2f0b 100644
> > > > > --- a/mkdumprd
> > > > > +++ b/mkdumprd
> > > > > @@ -106,6 +106,12 @@ to_mount() {
> > > > > _fstype=$(findmnt -k -f -n -r -o FSTYPE $_dev)
> > > > > _options=$(findmnt -k -f -n -r -o OPTIONS $_dev)
> > > > > _options=${_options/#ro/rw} #mount fs target as rw in 2nd
kernel
> > > > > + # "x-initrd.mount" mount failure will trigger
isolate emergency service
> > > > > + # W/o this, systemd won't isolate, thus we won't
get to emergency
> > > > > + # This only applicable for local fs mount
> > > > > + if ! is_nfs_target; then
> > > > > + _options="$_options,x-initrd.mount"
> > > > > + fi
> > > >
> > > > So why is it applicable to local fs mounts only. What will happen in
> > > > case of remote mount. Do we already isolate to emergency service in
that
> > > > case?
> > >
> > > If we use "x-initrd.mount", remote mount will become required
by
> > > "initrd-root-fs.target", instead of
"remote-fs.target". That's how it is
> > > handled within systemd internal.
> > >
> > > We need remote mount to be required "remote-fs.target", because
we need
> > > to bring up network before any remote mount and
"remote-fs.target" can
> > > be a checkpoint of that.
> >
> > So first of all this explanation should be part of the comment near the
> > code.
>
> Sure. Will do.
> >
> > Secondly, so what will happen if nfs mount fails and remote-fs.target
> > does not reach. Will we isolate to kdump error handler or not? So handling
> > of remote-fs.target is different from local-fs.target when it comes
> > to emergency shell? I think all this needs to be part of changelog/comment
> > so that reader can understand why are we treating remote mounts and
> > local mounts differently.
>
> In case of nfs mount failure, we won't isolate kdump error handler
> right away. But will do in the later point in kdump.sh fails because
> the dump target isn't mounted.
And why do we treat those two cases differently? One problem with local
mounts was that we will never hit the kdump.sh because some intermediate
targets have not reached. That's why we had to isolate to emergency
shell immediately.
So in case of remote mounts we don't have the same issue? We will still
hit kdump.sh, no matter what.
In fact, "remote-fs-pre.target" plays a role as a checkpoint for
network is brought up.
Say we use "initrd-root-fs.target" for nfs mount, systemd could start to
mount nfs before network is OK. nfs mount would fail.
Now we use "remote-fs.target", systemd will start mount nfs after
"remote-fs-pre.target". The call graph looks like:
remote-fs-pre.target -> bring up network -> nfs.mount -> remote-fs.target
The reason we treat nfs and local mount diffferent because nfs mount
must be started after network is ok but local mount doesn't need
network.
>
> >
> > >
> > > >
> > > > Also what happens in case of raw disk? What happens in case of ssh
dump
> > > > when network interface does not show up?
> > >
> > > In dracut-initqueue hook, both network and disk will be brought up, if
> > > needed. And kdump.sh runs after dracut-initqueue, so network and disk
> > > will be there before running kdump.sh.
> >
> > So if dracut-initqueue fails/timesout, will we isolate to kdump emergency
> > shell?
>
> No. dracut-initqueue is configured by dracut and it's emergency handler
> is dracut-emergency.service.
But I thought we disabled dracut-emergency.service. So are you saying
that if failure happens early, then dracut emergency handler will run.
But that will put us on shell instead of rebooting?
We've disabled dracut-emergency.service, so failure of dracut-initqueue
is ignored and systemd would just continue to run kdump.sh
Rethinking another round, I think it's better to enable dracut-emergency
service but replace it with our own error handler. So that we can fail
early and run kdump error handler, instead of ignoring the failure.
Thanks
WANG Chao
>
> In case of dracut-initqueue failure, I think we can still proceed
> kdump.sh and gets failed there.
Not sure what does this mean.
Thanks
Vivek