On Wed, Apr 16, 2014 at 12:50:53PM -0400, Martin Langhoff wrote:
On Mon, Apr 14, 2014 at 9:07 AM, Jaroslav Reznik
<jreznik(a)redhat.com> wrote:
> The communication between the two daemons is done over standard HTTPS,
I hear you holler "OMG you have to build full redundancy in your
logging backend"; and... I have not seen a single operation where the
logging backed was fully redundant.
Hi,
the upload client is like any other journal client — it is fully asynchronous
wrt. to journald writing log entries. (It's something like
'journalctl -o export|curl -X POST
https://some.where/upload'.)
So it is assumed that local storage of journal logs is enabled (either
in /var/log/journal or in /run/log/journal), and the uploader simply
reads data from there at its leisure. In fact, the uploader is only
started late during boot, after the network is up, while journald
writes logs from very early in the boot.
There's no guarantee that the logs will ever be uploaded, but in
practice they will only be lost if enough new logs come in to force
the old entries out. Even with the journal capped at a few megabytes,
this shouldn't happen but rarely.
And in fact it may be too much to ask -- in most setups log entries
are not _that_ precious. I know I can reconfigure a syslog server and
restart it without the 1K VMs that talk to it glitching. A recent
"loop" in our syslog configuration was a relatively minor problem
because it just dropped the traffic it couldn't handle for a brief
time while we fixed things.
This is the reality of system configs I know. Fully redundant,
"perfect" log servers are very hard to run, might not be worth it, and
anyway such a change won't happen overnight.
To avoid gridlocking operations, IMO this logging will need a local
queue, (for later retry), with a "drop anything older than X" escape
hatch...
I think that this describes my current implementation to a large extent.
The client remembers the position ("cursor") of the last uploaded entry
and restarts from there. "drop anything older than X" feature sounds
useful, and should be easy to add.
Zbyszek