[Aide] RE: Weird AIDE problem

John Farmer jfarmer at iirenergy.com
Tue Apr 5 23:36:12 EEST 2005 


>-----Original Message-----
>From: Lue-Fook-Sang, Andre
>Sent: Friday, April 01, 2005 9:25 PM
>To: 'aide at cs.tut.fi'
>Subject: Re: [Aide] Weird AIDE problem
>
>
>Is this a server reachable from the web? There was asecurity hole in openssl
>not so long ago.

Server is not reachable from the web.  Its purpose is just to backup 
servers.  The only service that runs on that machine is sshd.


>If you have a backup before the problem started try doing a strings -a on
>some of the binaries and do a diff vs the one there now on the strings
>output but do it on different system. You are looking for added function
>calls or some call not present in other reference binaries.
>
>Transfer it the suspect binaries using a non-standard system tool(in case
>you were rooted) ie nc and do it on a trusted system.
>
>Honedtly this doesn't sound like aide. If you were not compromised, I'd
>guess some probs with the hard drive on it's way out and lastly ext3 under
>severe I/O having issue (doubtful and testable on another system)


I dont think it is a failing hard drive because we have a 3ware raid card 
that will send me a message if it even hickups.  The reason I thought it 
might be ext3 under heavy I/O is because the files change everynight after 
the server runs the backups.  And the fact that the files change 4 times in 
a cylce (meaning it changes back to its original form)  kind of leads me to 
believe it may have something to do with the journaling.  I ran the diff 
and cmp commands to show the difference see the output below.  Someone has 
mentioned that it might be caused by a prelink command but I do not have a 
prelink command on this box.  I really appreciate anyone and everyones help 
on this problem so far.











>Hope this helps
>Andre
>Andre' Lue-Fook-Sang
>Thomson One Security Engineer
>Technical Operations - Production Support
>Thomson Financial
>Tel: 212-510-3943
>Fax: 212-510-4498
>
>
>-----Original Message-----
>From: aide-bounces at cs.tut.fi <aide-bounces at cs.tut.fi>
>To: aide at cs.tut.fi <aide at cs.tut.fi>
>Sent: Fri Apr 01 09:57:56 2005
>Subject: Re: [Aide] Weird AIDE problem
>
>here is the output for cmp -b
>
>thanks for you help again.
>
>/usr/local/ssl/bin/openssl /usr/local/ssl/bin/opensslbackup differ: byte
>766582, line 1303 is 376 M-~ 377 M-^?
>
>
>
>At 03:00 AM 4/1/2005, you wrote:
> >Send Aide mailing list submissions to
> >         aide at cs.tut.fi
> >
> >To subscribe or unsubscribe via the World Wide Web, visit
> >         https://mailman.cs.tut.fi/mailman/listinfo/aide
> >or, via email, send a message with subject or body 'help' to
> >         aide-request at cs.tut.fi
> >
> >You can reach the person managing the list at
> >         aide-owner at cs.tut.fi
> >
> >When replying, please edit your Subject line so it is more specific
> >than "Re: Contents of Aide digest..."
> >
> >
> >Today's Topics:
> >
> >    1. Re: Weird AIDE problem (John Farmer)
> >    2. Re: Weird AIDE problem (Richard van den Berg)
> >
> >
> >----------------------------------------------------------------------
> >
> >Message: 1
> >Date: Thu, 31 Mar 2005 09:41:00 -0600
> >From: John Farmer <jfarmer at iirenergy.com>
> >Subject: Re: [Aide] Weird AIDE problem
> >To: aide at cs.tut.fi
> >Message-ID:
> >         <6.2.1.2.0.20050331093307.04d2b080 at mail.industrialinfo.com>
> >Content-Type: text/plain; charset="iso-8859-1"; format=flowed
> >
> >The file does change I made a backup but its so small I guess it doesnt
> >really make a difference.  Here is a diff -a of a file before and after
> >the heavy io.
> >
> >*** opensslbackup       Tue Aug 24 09:15:32 2004
> >--- openssl     Tue Aug 24 09:15:32 2004
> >*************** X[^Ͷ*** 458,464 ****
> >    ‰äžèïêÿÿ돃ìPèôvƒìSèôòÿÿƒÄëЃìPè& 
> èÒåÿÿƒÄ 빍¶!     1Û9óŒÓ
>1Û9óŒÔ---
> >458,464 ----
> >    ‰äžèïêÿÿ돃ìPèôvƒìSèôòÿÿƒÄëЃìPè& 
> èÒåÿÿƒÄ
> >빍¶!     1Û9óŒÓ  1Û9óŒÔ*************** 
> W‹rU‹zSƒìl‹‹h‰$‰l$‹X‹h‰\$‰l$‹X
> >*** 1300,1306 ****
> >    ‹œø1Ò÷4™ƒúv4Cûÿ  PVèoÁùÿƒÄƒøvCûÿ 
> P‹D$,PèÛ¿ùÿƒÄ…Àt4ƒì‹ !
> >P‹D$Pè¿¿ùÿƒÄ…ÀtCûÿ$H‰D$D‰D$@ƒÄ,[^_]é*Áùÿv 
> ¿‹D$Õ‹WƒÓÑ0RUUè[­ÿÿƒÄ…À„
> >ÿÿÿƒìjW蕬ùÿƒÄ…Àu´é 
> ÿÿÿƒìhÁ´&…Àº$(90tTƒìhˆ  ÇÐU
> >èI}ûÿƒÄëАƒì¡0V…ÀuƒÄÃìhà UjÇ0V--- 1300,1306 ----
> >    ‹œø1Ò÷4™ƒúv4Cûÿ  PVèoÁùÿƒÄƒøvCûÿ 
> P‹D$,PèÛ¿ùÿƒÄ…Àt4ƒì‹ !
> >P‹D$Pè¿¿ùÿƒÄ…ÀtCûÿ$H‰D$D‰D$@ƒÄ,[^_]é*Áùÿv 
> ¿‹D$Õ‹WƒÓÑ0RUUè[­ÿÿƒÄ…À„
> >ÿÿÿƒìjW蕬ùÿƒÄ…Àu´é 
> ÿÿÿƒìhÁ´&…Àº$(90tTƒìhˆ  ÇÐU
> >èI}ûÿƒÄëАƒì¡0V…ÀuƒÄÃìhà UjÇ0V
> >
> >
> >The file still works if you run it so I dont know really what is going
> >on.  This isnt the only file that is changing.  A bunch of other
> >binaries are also changing in the same way that the openssl binary is
> >changing.
> >
> >ssh,ssh-keyscan,h2xs,libnetcfg,sshd,pine,autoexpect,makemap,debugfs
> >
> >
> >Has anyone else seen anything like this?  I'm completely stumped.
> >
> >
> >
> >At 12:39 AM 3/31/2005, you wrote:
> > >On Wed, 23 Mar 2005, John Farmer wrote:
> > >
> > > > I'm noticing some strange behavior on our server and I wondered if
> > > > anyone had seen anything like this before. Here is how it started.
> > > > On this day:
> > > >
> > > > Start timestamp: 2005-03-15 15:00:01
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : WJvJGt/2UCv5nHph2RqTpQ== , 0HH05buevntg0SmoSlavvA==
> > > >
> > > >
> > > > So I updated the aide database and then the next day.
> > > >
> > > > Start timestamp: 2005-03-16 02:00:02
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : 0HH05buevntg0SmoSlavvA== , WPOUrghNI3gE9TDt4DNqXA==
> > > >
> > > > So again I updated the aide database:
> > > > Start timestamp: 2005-03-17 02:00:03
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : WPOUrghNI3gE9TDt4DNqXA== , WJvJGt/2UCv5nHph2RqTpQ==
> > > >
> > > >
> > > > So I reloaded it one more time.
> > > > Start timestamp: 2005-03-17 19:00:01
> > > > File: /usr/local/ssl/bin/openssl
> > > >    MD5      : WJvJGt/2UCv5nHph2RqTpQ==          ,
> > > > 0HH05buevntg0SmoSlavvA==
> > > >
> > > >
> > > >
> > > > Around 2am and 2pm is when this server is under very heaving IO
> > > > from
> > doing
> > > > backups.  The partition with the "changing" files is an EXT3
> > > > partition. Anyone have any ideas on why this is happening?
> > >
> > >If the file doesn't change in reality, there must be a bug somewhere.
> > >Might want to try configure switch "--without-mmap".
> > >
> > >Duke NEMO / C.O.M.A
> > >alias pablo the pallo virolainen
> >
> >
> >
> >
> >
> >
> >------------------------------
> >
> >Message: 2
> >Date: Thu, 31 Mar 2005 18:12:58 +0200
> >From: Richard van den Berg <richard at vdberg.org>
> >Subject: Re: [Aide] Weird AIDE problem
> >To: Aide user mailinglist <aide at cs.tut.fi>
> >Message-ID: <424C218A.5010105 at vdberg.org>
> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> >
> >John Farmer wrote:
> > > The file does change I made a backup but its so small I guess it
> > > doesnt really make a difference.  Here is a diff -a of a file before
> > > and after the heavy io.
> >
> >Try cmp -b for seeing the changes in binary files. It looks like you
> >have a serious problem on your system. Aide is right to report a
> >difference if you can even spot it with diff.
> >
> >Sincerely,
> >
> >Richard van den Berg
> >
> >
> >------------------------------
> >
> >_______________________________________________
> >Aide mailing list
> >Aide at cs.tut.fi
> >https://mailman.cs.tut.fi/mailman/listinfo/aide
> >
> >
> >End of Aide Digest, Vol 9, Issue 1
> >**********************************
>
>------------------------------
>
>Message: 2
>Date: Mon, 4 Apr 2005 11:59:03 -0400
>From: "Lue-Fook-Sang, Andre" <andre.lue-fook-sang at thomson.com>
>Subject: RE: [Aide] Problems with S (check for growing size)
>To: "'Montserrat.Calero at Sun.COM'" <Montserrat.Calero at Sun.COM>,  "'Aide
>         user mailinglist'" <aide at cs.tut.fi>
>Message-ID:
>         <ED33F4A38B567C44951054C9CF35AA6A343BB0 at CS-MAIL.cust.ilx.com>
>Content-Type: text/plain
>
>Hmm, not sure but S is defined as a check for growing size, not sure if it
>does decreasing.
>
>Try the > check, for growing logfile = p+u+g+i+n+S
>Something like
>LOG = >
>
>Then try
>
>/var/adm/sulog     LOG
>
>Hope that helps
>Andre'
>
>-----Original Message-----
>From: aide-bounces at cs.tut.fi [mailto:aide-bounces at cs.tut.fi] On Behalf Of
>Montserrat Calero Marchante
>Sent: Monday, April 04, 2005 4:16 AM
>To: aide at cs.tut.fi
>Subject: [Aide] Problems with S (check for growing size)
>
>
>Hello
>
>I've installed AIDE  (aide-0.10-sol9-sparc-local.gz) on a Solaris SPARC
>version 9 platform.
>
>I've some problems with the following default group: S (check for growing
>size). i've been making tests to know the behaviour of this pamaremter over
>the /var/adm/sulog file. The problem is that #aide --check don't detect any
>change on this file, even if it's increassing or decreassing.
>
>Bellow the aide.conf file I've used:
>
>
>@@define BINDIR /usr/local/bin
>@@define CONFDIR /usr/local/etc
>@@define DBDIR /usr/local/etc
>@@define LOGDIR /var/adm
>
># the database
>database=file:@@{DBDIR}/aide.db database_out=file:@@{DBDIR}/aide.db.new
># reporting options
>#report_url=file:@@{LOGDIR}/aide.log
>report_url=file:/var/log/aide.log
>verbose=20
>#warn_dead_symlinks=yes
>
>
># main configuration
>/var/adm/sulog$         S
>
>There, the aide.db contents:
>
>@@begin_db
># This file was generated by Aide, version 0.10
># Time of generation was 2005-04-04 09:56:06
>@@db_spec name lname attr size
>/var/adm/sulog 0 4194305 70
>@@end_db
>
>TEST1: Increase size /var/adm/sulog
>RESULT1:  /usr/local/bin/aide --check
>
>AIDE, version 0.10
>
>### All files match AIDE database.  Looks okay!
>
>TEST2: Decrease size /var/adm/sulog
>RESULT2:  /usr/local/bin/aide --check
>
>AIDE, version 0.10
>
>### All files match AIDE database.  Looks okay!
>
>Any idea or suggestion about AIDE don't say anything about de sulog growing
>or decreasse size?
>
>Thanks in advance
>
>Montse
>
>
>
>--
>_________________________________________________________________
>Montse Calero Marchante
>Project Engineer / Client Solutions
>
>Sun Microsystems Iberica
>Centro Empresarial Parque Norte
>C/ Serrano Galvache, 56  -  28033 Madrid - SPAIN
>Oficina: +34 91 767 6587
>Movil: 696927125
>_________________________________________________________________
>
>
>_______________________________________________
>Aide mailing list
>Aide at cs.tut.fi
>https://mailman.cs.tut.fi/mailman/listinfo/aide
>
>
>------------------------------
>
>Message: 3
>Date: Mon, 4 Apr 2005 13:16:48 -0400
>From: "Miner, Jonathan W (CSC) (US SSA)"
>         <jonathan.w.miner at baesystems.com>
>Subject: RE: [Aide] Weird AIDE problem
>To: "Aide user mailinglist" <aide at cs.tut.fi>
>Message-ID: <7FCB0E206880084DB3D57CBAA2119F12D7B596 at blums0010>
>Content-Type: text/plain; charset="windows-1252"
>
>This may be a result of the "prelink" command.  I had not really been 
>following this thread, but happened upon thus page:
>
>http://www.fedoraforum.org/forum/archive/index.php/t-23067.html
>
>while researching something else.  Hope this helps.
>
>
>-----Original Message-----
>From:   aide-bounces at cs.tut.fi on behalf of Lue-Fook-Sang, Andre
>Sent:   Mon 04/04/2005 11:33 AM
>To:     'aide at cs.tut.fi'
>Cc:
>Subject:        RE: [Aide] Weird AIDE problem
>
>
>-----Original Message-----
>From: Lue-Fook-Sang, Andre
>Sent: Friday, April 01, 2005 9:25 PM
>To: 'aide at cs.tut.fi'
>Subject: Re: [Aide] Weird AIDE problem
>
>
>Is this a server reachable from the web? There was asecurity hole in openssl
>not so long ago.
>
>If you have a backup before the problem started try doing a strings -a on
>some of the binaries and do a diff vs the one there now on the strings
>output but do it on different system. You are looking for added function
>calls or some call not present in other reference binaries.
>
>Transfer it the suspect binaries using a non-standard system tool(in case
>you were rooted) ie nc and do it on a trusted system.
>
>Honedtly this doesn't sound like aide. If you were not compromised, I'd
>guess some probs with the hard drive on it's way out and lastly ext3 under
>severe I/O having issue (doubtful and testable on another system)
>
>Hope this helps
>Andre
>Andre' Lue-Fook-Sang
>Thomson One Security Engineer
>Technical Operations - Production Support
>Thomson Financial
>Tel: 212-510-3943
>Fax: 212-510-4498
>
>
>-----Original Message-----
>From: aide-bounces at cs.tut.fi <aide-bounces at cs.tut.fi>
>To: aide at cs.tut.fi <aide at cs.tut.fi>
>Sent: Fri Apr 01 09:57:56 2005
>Subject: Re: [Aide] Weird AIDE problem
>
>
>here is the output for cmp -b
>
>thanks for you help again.
>
>/usr/local/ssl/bin/openssl /usr/local/ssl/bin/opensslbackup differ: byte
>766582, line 1303 is 376 M-~ 377 M-^?
>
>
>
>At 03:00 AM 4/1/2005, you wrote:
> >Send Aide mailing list submissions to
> >         aide at cs.tut.fi
> >
> >To subscribe or unsubscribe via the World Wide Web, visit
> >         https://mailman.cs.tut.fi/mailman/listinfo/aide
> >or, via email, send a message with subject or body 'help' to
> >         aide-request at cs.tut.fi
> >
> >You can reach the person managing the list at
> >         aide-owner at cs.tut.fi
> >
> >When replying, please edit your Subject line so it is more specific
> >than "Re: Contents of Aide digest..."
> >
> >
> >Today's Topics:
> >
> >    1. Re: Weird AIDE problem (John Farmer)
> >    2. Re: Weird AIDE problem (Richard van den Berg)
> >
> >
> >----------------------------------------------------------------------
> >
> >Message: 1
> >Date: Thu, 31 Mar 2005 09:41:00 -0600
> >From: John Farmer <jfarmer at iirenergy.com>
> >Subject: Re: [Aide] Weird AIDE problem
> >To: aide at cs.tut.fi
> >Message-ID:
> >         <6.2.1.2.0.20050331093307.04d2b080 at mail.industrialinfo.com>
> >Content-Type: text/plain; charset="iso-8859-1"; format=flowed
> >
> >The file does change I made a backup but its so small I guess it doesnt
> >really make a difference.  Here is a diff -a of a file before and after
> >the heavy io.
> >
> >*** opensslbackup       Tue Aug 24 09:15:32 2004
> >--- openssl     Tue Aug 24 09:15:32 2004
> >*************** X[^Ͷ*** 458,464 ****
> >    ‰äžèïêÿÿ돃ìPèôvƒìSèôòÿÿƒÄëЃìPè&    èÒåÿÿƒÄ 빍¶!     1Û9óŒÓ
>1Û9óŒÔ---
> >458,464 ----
> >    ‰äžèïêÿÿ돃ìPèôvƒìSèôòÿÿƒÄëЃìPè&    èÒåÿÿƒÄ
> >빍¶!     1Û9óŒÓ  1Û9óŒÔ*************** W‹rU‹zSƒìl‹‹h‰$‰l$‹X‹h‰\$‰l$‹X
> >*** 1300,1306 ****
> >    ‹œø1Ò÷4™ƒúv4Cûÿ  PVèoÁùÿƒÄƒøvCûÿ  P‹D$,PèÛ¿ùÿƒÄ
Àt4ƒì‹ !
> >P‹D$Pè¿¿ùÿƒÄ
ÀtCûÿ$H‰D$D‰D$@ƒÄ,[^_]é*Áùÿv  ¿‹D$Õ‹WƒÓÑ0RUUè[­ÿÿƒÄ
À„
> >ÿÿÿƒìjW蕬ùÿƒÄ
Àu´é     ÿÿÿƒìhÁ´&
Àº$(90tTƒìhˆ  ÇÐU
> >èI}ûÿƒÄëАƒì¡0V
ÀuƒÄÃìhàUjÇ0V--- 1300,1306 ----
> >    ‹œø1Ò÷4™ƒúv4Cûÿ  PVèoÁùÿƒÄƒøvCûÿ  P‹D$,PèÛ¿ùÿƒÄ
Àt4ƒì‹ !
> >P‹D$Pè¿¿ùÿƒÄ
ÀtCûÿ$H‰D$D‰D$@ƒÄ,[^_]é*Áùÿv  ¿‹D$Õ‹WƒÓÑ0RUUè[­ÿÿƒÄ
À„
> >ÿÿÿƒìjW蕬ùÿƒÄ
Àu´é     ÿÿÿƒìhÁ´&
Àº$(90tTƒìhˆ  ÇÐU
> >èI}ûÿƒÄëАƒì¡0V
ÀuƒÄÃìhàUjÇ0V
> >
> >
> >The file still works if you run it so I dont know really what is going
> >on.  This isnt the only file that is changing.  A bunch of other
> >binaries are also changing in the same way that the openssl binary is
> >changing.
> >
> >ssh,ssh-keyscan,h2xs,libnetcfg,sshd,pine,autoexpect,makemap,debugfs
> >
> >
> >Has anyone else seen anything like this?  I'm completely stumped.
> >
> >
> >
> >At 12:39 AM 3/31/2005, you wrote:
> > >On Wed, 23 Mar 2005, John Farmer wrote:
> > >
> > > > I'm noticing some strange behavior on our server and I wondered if
> > > > anyone had seen anything like this before. Here is how it started.
> > > > On this day:
> > > >
> > > > Start timestamp: 2005-03-15 15:00:01
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : WJvJGt/2UCv5nHph2RqTpQ== , 0HH05buevntg0SmoSlavvA==
> > > >
> > > >
> > > > So I updated the aide database and then the next day.
> > > >
> > > > Start timestamp: 2005-03-16 02:00:02
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : 0HH05buevntg0SmoSlavvA== , WPOUrghNI3gE9TDt4DNqXA==
> > > >
> > > > So again I updated the aide database:
> > > > Start timestamp: 2005-03-17 02:00:03
> > > >
> > > > File: /usr/local/ssl/bin/openssl
> > > > MD5 : WPOUrghNI3gE9TDt4DNqXA== , WJvJGt/2UCv5nHph2RqTpQ==
> > > >
> > > >
> > > > So I reloaded it one more time.
> > > > Start timestamp: 2005-03-17 19:00:01
> > > > File: /usr/local/ssl/bin/openssl
> > > >    MD5      : WJvJGt/2UCv5nHph2RqTpQ==          ,
> > > > 0HH05buevntg0SmoSlavvA==
> > > >
> > > >
> > > >
> > > > Around 2am and 2pm is when this server is under very heaving IO
> > > > from
> > doing
> > > > backups.  The partition with the "changing" files is an EXT3
> > > > partition. Anyone have any ideas on why this is happening?
> > >
> > >If the file doesn't change in reality, there must be a bug somewhere.
> > >Might want to try configure switch "--without-mmap".
> > >
> > >Duke NEMO / C.O.M.A
> > >alias pablo the pallo virolainen
> >
> >
> >
> >
> >
> >
> >------------------------------
> >
> >Message: 2
> >Date: Thu, 31 Mar 2005 18:12:58 +0200
> >From: Richard van den Berg <richard at vdberg.org>
> >Subject: Re: [Aide] Weird AIDE problem
> >To: Aide user mailinglist <aide at cs.tut.fi>
> >Message-ID: <424C218A.5010105 at vdberg.org>
> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> >
> >John Farmer wrote:
> > > The file does change I made a backup but its so small I guess it
> > > doesnt really make a difference.  Here is a diff -a of a file before
> > > and after the heavy io.
> >
> >Try cmp -b for seeing the changes in binary files. It looks like you
> >have a serious problem on your system. Aide is right to report a
> >difference if you can even spot it with diff.
> >
> >Sincerely,
> >
> >Richard van den Berg
> >
> >
> >------------------------------
> >
> >_______________________________________________
> >Aide mailing list
> >Aide at cs.tut.fi
> >https://mailman.cs.tut.fi/mailman/listinfo/aide
> >
> >
> >End of Aide Digest, Vol 9, Issue 1
> >**********************************

More information about the Aide mailing list