[iwhd patch] Document inner workings

Pete Zaitcev zaitcev at redhat.com
Wed Feb 8 18:48:15 UTC 2012


As it happens, some of the inner workings of iwhd are less than obvious,
at least until explained. In particular sysadmins would appreciate knowing
what data is stored where and what to remove if something overflows.

This addresses bz#782872. Wes reported the bug but the complaints came
from more than just QA. Generally iwhd seems opaque to users.

---
 doc/admin_guide.md |   68 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 64 insertions(+), 4 deletions(-)

diff --git a/doc/admin_guide.md b/doc/admin_guide.md
index e81b557..9f07ff7 100644
--- a/doc/admin_guide.md
+++ b/doc/admin_guide.md
@@ -125,7 +125,67 @@ The iwhd writes a log by itself, without syslog. However, when installed
 from a package, the log is properly rotated using logrotate(8). The default
 location is /var/log/iwhd.log.
 
-The log format is somewhat arbitrary, but it is being improved.
-
-When troubleshooting, it is advised to add "-v" to IWHD_ARGS in
-/etc/sysconfig/iwhd and restart iwhd.
+The log format includes 2 messages for each API operation. This is done
+because often it is necessary to be aware of operations that take a long
+time to complete. Therefore, one message is logged when the API call
+is received, and another when the operation completes. The two messages
+are correlated using the IP address and port number of the client.
+The following example corresponds to the curl command above:
+
+Dec 06 15:33:43 iwhd[3363]: 192.168.128.11.37750 "GET /" -
+Dec 06 15:33:43 iwhd[3363]: 192.168.128.11.37750 200 302
+
+In the log above, the HTTP GET transfer to the API root completed
+with a code "200" (success) and returned 302 bytes of data.
+
+For additional troubleshooting, it may be helpful to add "-v" to IWHD_ARGS
+in /etc/sysconfig/iwhd and restart iwhd. However, the verbose log is noisy.
+It is generally used only to debug iwhd in a development environment.
+
+Recovery from lack of space
+---------------------------
+
+Lack of space is the most common reason why the error 500 is seen
+returned. It can happen on both PUT and POST transfers, and it is
+most insidious in smaller private clouds that are short on space,
+because iwhd has to use temporary files, which are cleaned up in
+case of error.
+
+In addition, certain kinds of operational errors cause "leaks" of
+resources, at least in the present version, in order to help with
+the troubleshooting by preserving the temporary data which would
+otherwise be deleted. This is the case in particular for registrations
+to RHEV-M or oVirt clouds. Once the root cause of the failing
+registrations is resolved, the accumulated templates have to be
+removed manually.
+
+For these reasons, it is helpful to know where iwhd stores its data
+(not counting logs in /var/log, which are rotated by logrotate).
+Two places are used for permanent data:
+
+ * Attributes and metadata are kept in MongoDB database. Its location
+   is set by the configuration of mongod. The default is /var/lib/mongodb,
+   but it is usually overridden.
+
+ * Bulk data is either stored in the cloud under management, where it
+   is a subject to the controls of the cloud, or in a filesystem-like
+   type of back-end, such as "fs" or "fs-rhev-m". In that case, its
+   location is set as described in chapter "Step by step configuration"
+   above.
+
+Registration operations require additional temporary space to succeed.
+Small control files may be created in /tmp. In addition, registration
+in RHEV-M requires space in the cluster.
+
+ * The image is uploaded to the "Export Domain", seen as an NFS-mounted
+   directory. Location of this directory is set outside of iwhd, by
+   the Aeolus Conductor usually. System crashes may cause half-written
+   templates to accumulate there. Their file names start with "iwhd.*".
+
+ * Once template is uploaded, it has to be imported, according to the
+   RHEV-M terminology. This process may also fail, leaving a template
+   which is visible to RHEV-M. The safest way to remove this template
+   is from the UI of RHEV-M.
+
+In a busy cloud it is unsafe to remove everything temporary, so look for
+temporary files and templates that are unduly old.


More information about the iwhd-devel mailing list