Imagine if, at the very time you are in discussions with a security services company about beefing up your defenses, you find that your company has been compromised. But the intruder, rather than attacking your company's network, instead uses your servers as a launching pad for attacks on other companies, making your firm an unwitting accomplice.

This is the scenario that faced IT managers at a company we'll call BlueLeaf. Although we've changed its name for obvious reasons, along with the names of other relevant companies and locations, the story about the attack - and, more importantly, its resolution - is true. The only name that has not been changed is that of Riptech, the security services firm that helped BlueLeaf get to the bottom of the attack.

BlueLeaf is a publicly traded company that is considered a market leader in the highly competitive, multi-billion dollar IT infrastructure market. At the time of the attack, BlueLeaf IT managers were in discussions with Riptech to plan an external penetration test. The goal of the penetration test was to reveal IT infrastructure weaknesses to BlueLeaf management. Armed with this information, management was to consider the benefits of further investment in security improvements versus the risk of inaction.

At the time of the incident, BlueLeaf used a Check Point firewall to protect its corporate network, but the system administrator rarely reviewed logs generated by the firewall. BlueLeaf had not implemented intrusion detection capabilities.

BlueLeaf learned of the incident when from an unrelated firm that contacted BlueLeaf corporate administrators in response to a network attack that originated from a server located at BlueLeaf headquarters. The company demanded that BlueLeaf take all necessary steps to terminate the attack.

In the absence of diligent firewall and intrusion detection monitoring, BlueLeaf was only able to determine retroactively that one of its systems was compromised. This is representative of the plight at many corporations. Often a compromise is only noticed if an intruder affects system performance to such an extent that a system administrator is forced to investigate the cause, such as when an intruder accidentally fills a hard drive with captured passwords.

In the BlueLeaf case, while the intruder launched his attack from a server within the BlueLeaf network, he happened to attack an outside systems that was being monitored for such activity. When the system administrator of the attacked network detected the hostile activity, he quickly notified BlueLeaf system administrators.

After further investigation, Riptech soon discovered that BlueLeaf was actually used as a launching point for numerous attacks, several of which involved US military systems. It is important to note that only one company contacted BlueLeaf to complain; therefore, it is probably safe assume to that several compromised organizations were unaware of the attacks. It is also safe to assume that if BlueLeaf had not been notified by the compromised organization, BlueLeaf's system may have remained compromised for months without notice.

Following is a day-by-day account of the security incident and the follow-up investigation at BlueLeaf.

Day 1: The initial call
BlueLeaf administrators spent the first day conducting an internal investigation of the potential compromise. Through their initial investigation, the administrators discovered that one FTP server was sending a large volume of traffic to various external IP addresses. BlueLeaf administrators reviewed the system in question, but were unable to detect any signs of compromise or identify suspicious programs that could be the source of the attacks.

At this point, Riptech was contacted to analyze the intrusion and help BlueLeaf recover from the compromise. After an initial telephone interview regarding the incident, a Riptech consultant was sent onsite to assist BlueLeaf. In addition, Riptech advised BlueLeaf administrators to disconnect the network connection of the compromised system to prevent further damage by the hacker to both internal and external systems.

Although BlueLeaf system administrators were highly competent, they did not possess the security experience required to identify the source and nature of the attack, as the hackers were particularly skilled at hiding their presence. In addition, by attempting to identify the hacker responsible for compromising the system, BlueLeaf administrators risked destroying potential evidence, as well as destroying the system itself by accidentally tripping on the equivalent of a hacker landmine.

Once onsite, the Riptech consultant was brought up to speed on the events that had transpired to date. Unfortunately, due to the fact that the system in question was a critical FTP server that was used to transmit customer data between partners, BlueLeaf was unable to comply with Riptech's recommendation to take the system offline.

Proceeding with the investigation, Riptech created backups of critical data. Making an initial backup, which captures a "low level image" of a compromised system, is critical for future forensics, especially if the intruder employed advanced tools or techniques. Because system availability was critical, the system was backed up online using system utilities provided by Riptech. Riptech opted not to use BlueLeaf's system utilities because intruders often modify these. It is essential to use as little of the compromised system as possible in the actual backup process.

Often, the hard drive is removed from the compromised system and mounted read-only on a Riptech incident response (IR) system, where it is subsequently imaged in a secure environment. Disk images are used by Riptech to determine the scope of the compromise and to identify the techniques used by the intruder. Riptech also uses disk images to help identify other potential victims, as well as provide an undisturbed copy of the compromised system for evidence if BlueLeaf decided to charge the attacker in a criminal or civil case. Due to the critical nature of this particular server, removal of the hard drive was simply not possible in this case.

Riptech evaluated the compromised system in search of "fingerprints" left by the intruder and searched for files that had been created or modified recently. After a thorough review, Riptech discovered several files that were known components of a commonly available distribution of "rootkit," a popular hacker toolkit. Typically, intruders install a combination of custom and publicly available tools, such as rootkit, that allow them to disguise their presence and increase access to the compromised system. Additional MD5 checksum comparisons and examination of the system executables confirmed that a variant of rootkit was installed on the system.

The investigation revealed that the intruder's toolkit included several trojanized system executables, as noted in Figure 1.

In addition, the intruder's toolkit added a network sniffer to capture user passwords, a log cleaner to clean out any references in the system's logs, and a program to fix time/date stamps and CRC checksums (but not MD5 checksums) on the above trojanized executables. This is intended to prevent an administrator from easily detecting recent changes to files.

While inspecting these executables, Riptech uncovered a hidden directory that contained the toolkit used to attack remote systems. This tool was still running on the compromised BlueLeaf system when it was discovered. Riptech also learned that in addition to the rootkit supplied backdoor, the intruder installed an encrypted backdoor attached to a high TCP port. This technique essentially allowed the attacker to access the system with virtually no risk of detection.

Finally, upon further investigation, Riptech discovered that the intruder's password sniffer had a log file containing over 1400 username and passwords stolen from BlueLeaf employees, customers and partners.

In the next installment, we conclude the forensics effort by developing an action plan and tracking the intruder back to his home base.
Registered users were able to access the complete version earlier than unregistered users. Click here to register and see exclusive content, and to sign up for the SecurityGram newsletter.

Tim Belcher is chief technology officer of Riptech, Inc., a security service provider based in Alexandria, Va. that offers real-time security monitoring as well as professional security services including forensics, assessment, auditing and policy development. www.Riptech.com