Change in vdsm[master]: hsm: prepareForShutdown - operations order

nsoffer at redhat.com nsoffer at redhat.com
Sun Mar 15 06:28:49 UTC 2015


Nir Soffer has posted comments on this change.

Change subject: hsm: prepareForShutdown - operations order
......................................................................


Patch Set 4: Code-Review-1

(7 comments)

https://gerrit.ovirt.org/#/c/36162/4//COMMIT_MSG
Commit Message:

Line 8: 
Line 9: Currently cleanupMasterMount() is executed before
Line 10: taskMgr.prepareForShutdown() that causes to errors on updating the tasks
Line 11: status during the tasks abortion (becasue of failure to access the path
Line 12: of the tasks dir).
Fixing this requires separation of stopping a other threads and waiting for the thread to complete.

This is implemented partially in the domain monitor (cutting down shutdown time from 120 seconds to 10 with 50 threads).

All entities should implement:
- stop - signal need to stop, non-blocking
- wait - wait until entity is stopped

During shutdown, we should ask all entities to stop, then wait for all of them.

So fixing this require much larger change, best leave code as is for now.
Line 13: 
Line 14: The solution introduced in that patch is to change the order of the
Line 15: operations - so that the cleanupMasterMount() will be executed only
Line 16: after the tasks abortion is executed.


Line 12: of the tasks dir).
Line 13: 
Line 14: The solution introduced in that patch is to change the order of the
Line 15: operations - so that the cleanupMasterMount() will be executed only
Line 16: after the tasks abortion is executed.
This may leave behind stale master mounts, worth then tasks with incorrect status.
Line 17: 
Line 18: Change-Id: I9edd84317b08a17db80e265053edaf69582c2a51
Line 19: Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1161934


https://gerrit.ovirt.org/#/c/36162/4/vdsm/storage/hsm.py
File vdsm/storage/hsm.py:

Line 3418
Line 3419
Line 3420
Line 3421
Line 3422
Typically we have 10 seconds to complete this, before vdsm is killed (SIGKILL).


Line 3433
Line 3434
Line 3435
Line 3436
Line 3437
After this I would not touch shared storage.


Line 3439
Line 3440
Line 3441
Line 3442
Line 3443
This may block for long time if a task is blocked.


Line 3447
Line 3448
Line 3449
Line 3450
Line 3451
This may take several seconds (10 seconds for 50 storage domains, caused by sanlock), so there will be no time left to cleanup master mount after this.


Line 3452:                 self.log.warning("Failed to stop RepoStats thread",
Line 3453:                                  exc_info=True)
Line 3454: 
Line 3455:             self.taskMng.prepareForShutdown()
Line 3456:             sp.StoragePool.cleanupMasterMount()
This may never run or vdsm may be killed in the middle.
Line 3457:         except:
Line 3458:             pass
Line 3459: 
Line 3460:     @classmethod


-- 
To view, visit https://gerrit.ovirt.org/36162
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I9edd84317b08a17db80e265053edaf69582c2a51
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Liron Aravot <laravot at redhat.com>
Gerrit-Reviewer: Adam Litke <alitke at redhat.com>
Gerrit-Reviewer: Allon Mureinik <amureini at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: Federico Simoncelli <fsimonce at redhat.com>
Gerrit-Reviewer: Liron Aravot <laravot at redhat.com>
Gerrit-Reviewer: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: automation at ovirt.org
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes


More information about the vdsm-patches mailing list