On Tue, 2006-11-21 at 12:55 -0600, olga@urbantimes.net wrote:
El Jueves, 16 de Noviembre de 2006 22:56, olga@urbantimes.net escribió:
On Thu, 2006-11-16 at 10:26 -0600, olga@urbantimes.net wrote:
Hi,
I wrote about kernel errors which somebody pointed out was because
the
server was running out of memory.
Now I found the following which makes me think that that server may
have
been compromized.
Here's what I get when I issued: netstat -nap
tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED 5226/ps x tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED 5365/ps x
About a hundred instances of that program 'ps x' running.
Also here's what ps -ef produced:
apache 6323 1 0 10:30 ? 00:00:00 ps x apache 6324 1 0 10:30 ? 00:00:00 ps x apache 6326 1 0 10:30 ? 00:00:00 ps x apache 6328 1 0 10:30 ? 00:00:00 ps x apache 6330 1 0 10:30 ? 00:00:00 ps x
What does ls -l /proc/6323/exe say? That would be a symlink to the executable for that process. Normal ps lives in /bin so the link
should
point at /bin/ps. If it is connecting out to a remote host, it's
likely
not the normal ps, just something that's masking itself to make it
less
likely to get picked up.
-- David Hollis dhollis@davehollis.com
apache 3102 1 0 15:53 ? 00:00:00 httpd apache 3104 1 0 15:53 ? 00:00:00 httpd apache 3106 1 0 15:53 ? 00:00:00 httpd apache 3108 1 0 15:53 ? 00:00:00 httpd apache 3110 1 0 15:53 ? 00:00:00 httpd apache 3112 1 0 15:53 ? 00:00:00 httpd apache 3114 1 0 15:53 ? 00:00:00 httpd apache 3116 1 0 15:53 ? 00:00:00 httpd apache 3118 1 0 15:53 ? 00:00:00 httpd apache 3120 1 0 15:53 ? 00:00:00 httpd apache 3122 1 0 15:53 ? 00:00:00 httpd apache 3125 1 0 15:54 ? 00:00:00 httpd apache 3127 1 0 15:54 ? 00:00:00 httpd apache 3129 1 0 15:54 ? 00:00:00 httpd apache 3131 1 0 15:54 ? 00:00:00 httpd apache 3133 1 0 15:54 ? 00:00:00 httpd apache 3135 1 0 15:54 ? 00:00:00 httpd apache 3137 1 0 15:54 ? 00:00:00 httpd apache 3139 1 0 15:54 ? 00:00:00 httpd apache 3141 1 0 15:54 ? 00:00:00 httpd apache 3143 1 0 15:54 ? 00:00:00 httpd apache 3145 1 0 15:54 ? 00:00:00 httpd apache 3639 1 0 15:57 ? 00:00:00 ps x apache 3642 1 0 15:57 ? 00:00:00 ps x apache 3645 1 0 15:58 ? 00:00:00 ps x apache 3647 1 0 15:58 ? 00:00:00 ps x
I am getting a ton of these... Here's what ls -l /proc/3147/exe says lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe -> /usr/bin/perl
When I do netstat -nap I get: tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED -
The ip points to google...
And these appeared in the /tmp folder:
drwxrwxrwt 8 root root 4096 Nov 16 16:00 . drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix drwx------ 2 root root 4096 Nov 16 14:59 mc-root drwx------ 2 root root 12288 Nov 16 15:16 orbit-root -rw-r--r-- 1 apache apache 0 Nov 16 15:58 sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 sess_rdav631df3a1ddfaa34s1x1wwo521459 -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix
What is going on?
Finally...did they break into your system? Did you find something strange on the logs? I wonder what happened, give us some information this thread is quite interesting and will help other folks in a near future ;-) One way or another, if they got shell access (even remote text shell, you know...) you should think about reinstalling your system, as far as i know, if the left a rootkit you must not trust your system anymore.
By the way, let me give you and advice, installing Babel Enterprise could be a nice idea, ( http://babel.sourceforge.net/en/ ), yeah yeah, it's GPL ;-)
Babel is an enterprise-grade auditing system to manage a consistency on security policy between different systems in a non-homogeneus architecture. Babel allows to manage very different operating systems, like AIX, Solaris, Windows 2000, Windows XP, Linux, *BSD or HPUX.
Babel allows administrator team to monitor the hardening level of their systems and keep constantly monitored, using periodic policy polling, and of course, a WEB Based, graphical reporting, and of course, a centralized management for all systems
There's a demo online, try it.
Hope this helps.
It does appear that there has been a break-in. Some kind of script was running that was consuming all system resourses. At the time it was running, it was also deleting log entries, so if I looked at the log and searched for the time we brough the server up on the network, logs would show no activity at that time. And that 72.x.x.x IP was probably bogus as well.
Here's what I found in the httpd error log:
--06:31:56-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain]
0K .......... . 100% 169.99KB/s
06:31:56 (169.99 KB/s) - `borek.txt' saved [11666/11666]
Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. rm: cannot remove `borek.txt*': No such file or directory % Total % Received % Xferd Average Speed Time Curr. Dload Upload Total Current Left Speed 100 11666 100 11666 0 0 23100 0 0:00:00 0:00:00 0:00:00 156k Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. rm: cannot remove `borek.txt*': No such file or directory Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. sh: line 1: lynx: command not found sh: line 1: fetch: command not found Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. --06:32:02-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain]
0K .......... . 100% 166.39KB/s
A bunch of these with other file names instead of borek.txt and other ips as well.
Someone else has already suggested it, but I second the suggestion. Wipe the disk clean and reformat with a new install.
You have no idea what garbage may be laying around to bite you later if you just try to clean it up. A new install with a formatted disk will at least make sure no surprises are waiting for you.