[Aide] Binary MD5 hashes differ on identical RH Linux installations

[Goltz, Jim (NIH/NLM/LHC)]

[Apologies in advance if this has come up before.]

We're in the process of implementing AIDE (similar to Tripwire) for all of
our servers, and I've run into a very odd problem.

One of the things we want to do is have a "reference" AIDE database on a
central secure server, to compare MD5 hashes of binaries to known good
values.  For Solaris systems, I came up with a list of binaries commonly
replaced by rootkits, extracted the MD5 hashes, verified them against the
Solaris Fingerprint Database, and created a truncated AIDE database.  I then
use a Perl program to show me binaries whose hashes differ from the known
good values.

When I try to extend this to Red Hat Linux, however, I find that many
binaries that are identical in size, timestamps, and permissions come up
with completely different hashes.  

A good example is /bin/ls, commonly replaced by rootkits to hide certain
files.  Comparing /bin/ls binaries from three different systems with
identical installations (installed from the same kickstart server), all
three are exactly 85,232 bytes in size.  However, "cmp -l" shows differences
in selected bytes from 3241-3615 and 79,206-79,679.

The really weird thing is, a lot of binaries *do* have matching hashes, such
as /bin/su, /bin/ping, and some (but not all) of the binaries in
/usr/X11R6/bin.

I've searched on the web for any mention of how kickstart installs binaries,
whether some are linked on-the-fly, etc.  So far, no luck.

I know we can use "rpm" for some checks, but a clever attacker could
manipulate such a database or just replace the rpm program itself.

Does anyone know something that would account for this behavior?


-----
James P. Goltz (LHC/NLM/NIH)
38A/B1N-28K
Email: jgoltz at mail.nih.gov
Phone: 301-594-7523