monitoring open http sessions on a jboss server
by Attila Heidrich
Hi,
is there a way to monitor the open sessions of a jboss server?
I see I have some metrics for the http connector, and the requests per
second value is really useful, but we need some more detailed statistics.
It would be the best to have some browser to select the parameters... ;)
for webapp dveelopers it would be such a great benefit!
Afaik, there're some internal values in the jboss server, should I extend
the JBoss plugin to get a value, which isn't got now?
Attila
6 years, 3 months
Load factor
by Elmar Knipp
Hi all,
Is there a metric which is similar to the unix load factor, which you see e.g.
with "uptime" or "top"?
I see "user load" and "system load", but not a combined metric. I want to be
alerted when a system is sweating ;-)
Thanks,
Elmar
10 years, 1 month
false alerts on process resources
by Attila Heidrich
Hi!
RHQ 4.9
Sometimes we have got "status down" alerts of process resources. The
processes haven't got any problems meanwhile, and we don't have the false
positives for each and every process on the same platform.
The processes are mainly "vboxheadless something"-like processes - at lease
we only have problems with these. The alerts are cleared very soon, but
sthey are till really annoying.
Anyone having the same problem, or preferably a solution?
Regards,
Attila
10 years, 2 months
jboss resource: no alert clearing
by Attila Heidrich
Please take a look at the screenshots here:
https://www.dropbox.com/sh/heyowe3rc145ivd/_tlWMpMG3F
The resource is available, as it is shown in the timeline, but still there
are live alerts.
any idea, what is wrong? This is exactly the same as two days before, even
the TOD is the same.
I have found something at the given time in the Cassandra log:
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 MessagingService.java
(line 860) 1 READ messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 StatusLogger.java (line
53) Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java (line
68) ReadStage 0 0 23386785
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java (line
68) RequestResponseStage 0 0 43148496
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) ReadRepairStage 0 0 1722951
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) MutationStage 0 0 76756647
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) ReplicateOnWriteStage 0 0 0
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) GossipStage 0 0 11364876
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) AntiEntropyStage 0 0 69120
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) MigrationStage 0 0 0
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) MemtablePostFlusher 0 0 84294
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) FlushWriter 0 0 1277
0 3
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) MiscStage 0 0 201
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) commitlog_archiver 0 0 0
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) InternalResponseStage 0 0 0
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
68) AntiEntropySessions 0 0 2048
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
68) HintedHandoff 0 0 5
0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
78) CompactionManager 0 0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
90) MessagingService n/a 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
100) Cache Type Size
Capacity
KeysToSave Provider
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
101) KeyCache 38797256
38797312 all
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
107) RowCache 0
0 all
org.apache.cassandra.cache.SerializingCacheProvider
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
114) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
117) system.local 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
117) system.peers 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.batchlog 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.LocationInfo 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.Schema 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.Migrations 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.schema_columns 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.range_xfers 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.peer_events 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.hints 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) system.HintsColumnFamily 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.twenty_four_hour_metrics 139188,2097152
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.raw_metrics 870881,14680064
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.six_hour_metrics 487836,13631488
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.metrics_index 660267,4194304
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.schema_version 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.one_hour_metrics 163941,5242880
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.users 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.credentials 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.permissions 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_traces.events 0,0
Similar happens sometimes, but doesn't look to be stronglyrelated to my
problem, since there is no such info from 28-01.
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-02 04:09:21,020 MessagingService.java (line 860) 3 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-06 04:04:16,964 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-07 04:11:18,930 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-14 04:11:49,388 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-14 04:15:19,999 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-21 04:07:32,958 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-23 04:06:56,011 MessagingService.java (line 860) 3 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-23 04:10:53,667 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-30 04:10:19,381 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
Any idea? Last time I had to restart the agent. Then I have received the
clearing message two times.
Regards,
Attila
10 years, 2 months
no Alert clearing mail
by Attila Heidrich
Hi!
RHQ 4.9...
Sometimes the "clearing alert" doesn't happen, but the alert status is
cleared.
I have an uncleared alert, while the resource status is OK.
In fact, the timeline shows the jboss server unknown, while the status is
OK, and the agent log shows the periodic updates even for the recent
minutes. Cannot find any anomaly. Other resources on the same Platform
seems to work well.
Where to look for some traces of "WTF happened"? :)
Attila
10 years, 2 months
RHQ CLI - Obfuscating user/password data
by Andy Overton
Hi,
Does anyone know if it's possible to use obfuscated username/password values
to connect to RHQ via the CLI?
Seems like an obvious thing to do but from the docs it looks like you can
either pass in the username/pass in plain text or get it to prompt you for
the password but no option to use obfuscated values.
So, I can do:
rhq-cli.sh -u rhqadmin -p rhqadmin
rhq-cli.sh -u rhqadmin -P
but I need to do
rhq-cli.sh -u H*%@SDAFSDF -p H*%@SDAFSDF
or similar.
The problem is this needs to run as a cron job so I can't prompt for the
password but don't want it stored as plain text.
Thanks,
Andy
10 years, 2 months
RHQ state of the union January 2014
by Heiko W.Rupp
[ This is the first of a series of postings describing where we stand with RHQ and where we may want to go to. ]
As many of you already know, RHQ is the upstream of JBoss Operations Network (JBoss ON) and so RHQ is following the requirements of JBoss ON.
RHQ has released 4.9 in September 2013 and is due for another release. That next release will include some performance enhancments on inventory syncing and agent footprint as well as a lot of fixes since RHQ 4.9. This next release is supposed to happen within the next month.
The RHQ team recently had a face to face meeting in Brno at the same time that the WildFly team was there as well. There are a few photos available ([1] and [2]) but we missed to create an all-team photo.
Some of the outcomes of that meeting are:
• RHQ source code will definitively be moved from FedoraHosted to GitHub in the near future.
• RHQ will in the future use the JIRA instance at jboss.org instead of Bugzilla (timing unknown at this point).
Those two actions will make it easier to contribute in the future when you already have an account at jboss.org
From a technical standpoint, we will tackle the following in the near to mid term - please comment on the respective wiki pages:
• Audit Subsystem [3]
• "Canned DynaGroup expressions" [4]
• Add support for https connections of the as7-plugin
• Add support Bundles and Drift for the Domain mode of as7/WildFly
• Provide a mechanism for out-of-band signaling of state and events from the agent to the server [7] (the current proposal on the wiki is a first draft to illustrate the idea and needs much more work).
• Implement signaling of as7/WildFly "dirty" state over the mechanism in the previous item
Of course we were also talking about the longer term and here mostly about two items:
• UI: The current dependency on the SmartGWT widget set becomes more and more problematic as SmartGWT is not able to keep up with all the browser changes. In addition, the end of life of GWT DevMode is also reducing our ability to further improve the UI.
We have thus decided to investigate other options like using Angular.JS for further UI work. We will create a separate UI project on the RHQ GitHub account [5] and will create a UI that can be dropped into RHQ to get more insight how such a Angular.JS driven UI could work for RHQ. This Angular UI will talk via the existing REST-interface to the RHQ server.
• Agent: Our current agent is a powerful workhorse, but has its shortcomings. One of those is its runtime footprint that is currently addressed. But even afterwards we want to try to reduce its impact, so that it is easier to run the agent on managed systems.
The other part is that the agent currently is only extensible in Java, while operations people are often much more versed in languages like Python. Also the possibility to ad-hoc define some metrics to be evaluated is rather limited (especially when it comes to arbitrary MBean attributes).
This means that we will evaluate alternatives to the current agent and also to evaluate how the current agent can be enhanced to support polyglot and smaller runtime environments.
In addition to the above we also were talking about enhancing the Alerting subsystem[8] and to introcude a 2-phase-discovery to improve on some of the shortcomings of the current resource discovery [6].
Last but not least we want to thank all contributors for their suggestions, bug reports, enhancement requests and code submissions.
Heiko on behalf of the RHQ team
Links:
[1] https://plus.google.com/114249341487134308671/posts/5nC9PXCDHT8
[2] https://www.facebook.com/media/set/?set=a.250786711757107.1073741829.2505...
[3] https://docs.jboss.org/author/display/RHQ/Audit+Subsystem
[4] https://docs.jboss.org/author/display/RHQ/Canned+DynaGroup+expressions
[5] https://github.com/rhq-project/angular-ui
[6] https://docs.jboss.org/author/display/RHQ/2-phase+discovery
[7] https://docs.jboss.org/author/display/RHQ/Agent-Server-Backchannel
[8] https://docs.jboss.org/author/display/RHQ/Wintermute
This one was posted to
https://community.jboss.org/en/rhq/blog/2014/01/23/rhq-state-of-the-union...
10 years, 3 months
dependency of resources
by Attila Heidrich
Hi!
I use 4.9... is it possible to define some kind of dependency among the
resources? I get all the alerts (processes, network interfaces... one by
one) when the server stops. I guess there should be only a single one.
Attila
10 years, 3 months
agent lost for a few minutes
by Attila Heidrich
Hi!
Running on a virtualized Linux, one agent has lost for several minutes. No
measurements are present in the servers database, while theres no any sign
of problem in the virtualization log, or even in the agent's own log. There
is en exception, but the very same happens many time.
According the log, it sent the reports regularly for the server:
2014-01-14 10:17:41,788 INFO [MeasurementManager.sender-1]
(rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection
for [14] metrics took 94ms - sending report to Server...
2014-01-14 10:18:09,712 INFO [InventoryManager.availability-1]
(rhq.core.pc.inventory.AvailabilityExecutor)- Scan Starting: Tue Jan 14
10:18:09 CET 2014
2014-01-14 10:18:09,733 INFO [InventoryManager.availability-1]
(rhq.core.pc.inventory.AvailabilityExecutor)- Scan Ended : Tue Jan 14
10:18:09 CET 2014 : Scan [startTime=1389691089712, endTime=1389691089732,
runtime=20, isFull=false, isForced=false, numResources=179,
numGetAvailabilityCalls=7, numScheduledRandomly=0, numPushedByInterval=6,
numAvailabilityChanges=0, numDeferToParent=0]
2014-01-14 10:18:11,788 INFO [MeasurementManager.sender-1]
(rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection
for [14] metrics took 67ms - sending report to Server...
2014-01-14 10:18:11,812 INFO [ConfigurationManager.threadpool-1]
(rhq.core.pc.configuration.ConfigurationCheckExecutor)- Starting
configuration update check...
2014-01-14 10:18:11,814 INFO [ResourceContainer.invoker.daemon-56351]
(org.rhq.plugins.jmx.MBeanResourceComponent)- Configuration management is
currently disabled for the ResourceType[id=0, name=java.util.logging,
plugin=RHQAgent, category=Service] Resource with key
[java.util.logging:type=Logging] - it can be enabled in the Resource's
Inventory > Connection tab.
2014-01-14 10:18:12,793 WARN [ConfigurationManager.threadpool-1]
(rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred
while checking for an updated Resource configuration for Resource[id=11763,
uuid=74154a3e-07fc-4b0e-88be-f14e95059b8a, type={OpenSSH}SSHD, key=sshd,
name=OpenSSH sshd, parent=amweb-se, version=1.0].
net.augeas.AugeasException: get failed
at net.augeas.Augeas.processLastCall(Unknown Source)
at net.augeas.Augeas.get(Unknown Source)
at
org.rhq.plugins.sshd.OpenSSHDComponent.getConfig(OpenSSHDComponent.java:189)
at
org.rhq.plugins.sshd.OpenSSHDComponent.loadResourceConfiguration(OpenSSHDComponent.java:168)
at
org.rhq.plugins.sshd.OpenSSHDComponent.loadResourceConfiguration(OpenSSHDComponent.java:119)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2014-01-14 10:18:12,793 WARN [ConfigurationManager.threadpool-1]
(rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred
while checking for an updated Resource configuration for Resource[id=11765,
uuid=d9704c88-052e-47e2-8b9e-3495a57bb5c5, type={Apache}Apache HTTP Server,
key=/etc/apache2||/etc/apache2/apache2.conf, name=127.0.1.1:80,
parent=amweb-se, version=2.2.22].
java.lang.IllegalStateException: Configuration and child resource
creation/deletion support for Apache is optional. If you switched it on by
enabling Augeas support in the connection settings of the Apache server
resource and still get this message, it means that either your Apache
version is not supported (only Apache 2.x is supported) or Augeas is not
available on your platform.
at
org.rhq.plugins.apache.ApacheServerComponent.loadResourceConfiguration(ApacheServerComponent.java:369)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2014-01-14 10:18:12,794 WARN [ConfigurationManager.threadpool-1]
(rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred
while checking for an updated Resource configuration for Resource[id=11799,
uuid=63150549-2454-4218-aaad-ddcffc071718, type={Apache}Apache Virtual
Host, key=MainServer, name=Main, parent=127.0.1.1:80].
java.lang.IllegalStateException: Configuration and child resource
creation/deletion support for Apache is optional. If you switched it on by
enabling Augeas support in the connection settings of the Apache server
resource and still get this message, it means that either your Apache
version is not supported (only Apache 2.x is supported) or Augeas is not
available on your platform.
at
org.rhq.plugins.apache.ApacheVirtualHostServiceComponent.loadResourceConfiguration(ApacheVirtualHostServiceComponent.java:165)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2014-01-14 10:18:12,795 INFO [ConfigurationManager.threadpool-1]
(rhq.core.pc.configuration.ConfigurationCheckExecutor)- Configuration
update check completed in 982ms
2014-01-14 10:18:39,733 INFO [InventoryManager.availability-1]
(rhq.core.pc.inventory.AvailabilityExecutor)- Scan Starting: Tue Jan 14
10:18:39 CET 2014
2014-01-14 10:18:39,735 INFO [InventoryManager.availability-1]
(rhq.core.pc.inventory.AvailabilityExecutor)- Scan Ended : Tue Jan 14
10:18:39 CET 2014 : Scan [startTime=1389691119733, endTime=1389691119735,
runtime=2, isFull=false, isForced=false, numResources=179,
numGetAvailabilityCalls=1, numScheduledRandomly=0, numPushedByInterval=0,
numAvailabilityChanges=0, numDeferToParent=0]
2014-01-14 10:18:41,789 INFO [MeasurementManager.sender-1]
(rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection
for [9] metrics took 73ms - sending report to Server...
The Exception above occurs in every hour in the same minute.
And this is the servers log:
10:20:08,530 INFO [org.rhq.enterprise.server.core.AgentManagerBean]
(RHQScheduler_Worker-5) Have not heard from agent [amweb] since [Tue Jan 14
10:14:17 UTC 2014]. Will be backfilled since we suspect it is down
Any idea how to prevent such events in the future?
10 years, 3 months
reset pw of rhqadmin
by Elmar Knipp
Hi,
It seems that I forgot my password for the rhqadmin user. Is there a way
to retrieve or reset it? I have searched in the docu / forum but could
not find a solution.
Thanks, Elmar
10 years, 3 months