On Tue, 25 Jun 2013 15:16:04 -0400
seth vidal <skvidal(a)fedoraproject.org> wrote:
Last week when we were talking about spawning rdiff-backup to backup
our systems, we diverged into discussing app/apache logs and the
somewhat complicated system we currently have for grabbing those logs.
Right now we have a list of hosts on log02 that it should grab logs
from. Those hosts need to have rsyncd running on them to allow access
from log02 to fetch the /var/log/httpd/ path from them.
That requires 2 things to be coupled and it is a bit awkward if you
set up a host that is tricky to access from log02 or isn't on the vpn.
In general I also am not in love with having to have rsyncd listening
on systems - even if it is ip-restricted.
So the thought was we could do something like this on log02:
1. setup an ssh key on log02 that can run rsync to /var/log/httpd on
2. make any host that needs to have its logs retrieved be marked in
the ansible inventory host/group vars
3. git clone public-ansible-repo onto log02
4. use group_by to construct a group of the hosts which can then be
retrieved using rsync.
The sole reason for using ansible here is so we can keep the log sync
info in our inventory and to parallelize the retrieval of logs.
This is more or less identical to what we talked about for backups
Yeah, I think it's worth trying out... we may need to wait until we
have more stuff moved over to ansible just to try and minimize
When we were discussing this Luke mentioned then using
to search the resulting
files and compile a set of tracebacks our apps are dumping out.
Sounds good to me. We might also look at any stats we want to pull out
or other errors we want to note. We could do these similar to the
epylog runs? generate some kind of report link and mail it to
If we have all the logs on log02 generating a report like this would
be pleasantly kept away from the rest of our hosts and could give us
reasonably useful reports of brokenness.
Yeah, I agree...
I'd love some feed back on if this is all crazy or not :)
I don't think it's crazy at all.