This is the scenario that faced IT managers at a company we'll call BlueLeaf. Although we've changed its name for obvious reasons, along with the names of other relevant companies and locations, the story about the attack - and, more importantly, its resolution - is true. The only name that has not been changed is that of Riptech, the security services firm that helped BlueLeaf get to the bottom of the attack.

BlueLeaf is a publicly traded company that is considered a market leader in the highly competitive, multi-billion dollar IT infrastructure market. At the time of the attack, BlueLeaf IT managers were in discussions with Riptech to plan an external penetration test. The goal of the penetration test was to reveal IT infrastructure weaknesses to BlueLeaf management. Armed with this information, management was to consider the benefits of further investment in security improvements versus the risk of inaction.

At the time of the incident, BlueLeaf used a Check Point firewall to protect its corporate network, but the system administrator rarely reviewed logs generated by the firewall. BlueLeaf had not implemented intrusion detection capabilities.

BlueLeaf learned of the incident when from an unrelated firm that contacted BlueLeaf corporate administrators in response to a network attack that originated from a server located at BlueLeaf headquarters. The company demanded that BlueLeaf take all necessary steps to terminate the attack.

In the absence of diligent firewall and intrusion detection monitoring, BlueLeaf was only able to determine retroactively that one of its systems was compromised. This is representative of the plight at many corporations. Often a compromise is only noticed if an intruder affects system performance to such an extent that a system administrator is forced to investigate the cause, such as when an intruder accidentally fills a hard drive with captured passwords.

In the BlueLeaf case, while the intruder launched his attack from a server within the BlueLeaf network, he happened to attack an outside systems that was being monitored for such activity. When the system administrator of the attacked network detected the hostile activity, he quickly notified BlueLeaf system administrators.

After further investigation, Riptech soon discovered that BlueLeaf was actually used as a launching point for numerous attacks, several of which involved US military systems. It is important to note that only one company contacted BlueLeaf to complain; therefore, it is probably safe assume to that several compromised organizations were unaware of the attacks. It is also safe to assume that if BlueLeaf had not been notified by the compromised organization, BlueLeaf's system may have remained compromised for months without notice.

Following is a day-by-day account of the security incident and the follow-up investigation at BlueLeaf.

Day 1: The initial call

BlueLeaf administrators spent the first day conducting an internal investigation of the potential compromise. Through their initial investigation, the administrators discovered that one FTP server was sending a large volume of traffic to various external IP addresses. BlueLeaf administrators reviewed the system in question, but were unable to detect any signs of compromise or identify suspicious programs that could be the source of the attacks.

At this point, Riptech was contacted to analyze the intrusion and help BlueLeaf recover from the compromise. After an initial telephone interview regarding the incident, a Riptech consultant was sent onsite to assist BlueLeaf. In addition, Riptech advised BlueLeaf administrators to disconnect the network connection of the compromised system to prevent further damage by the hacker to both internal and external systems.

Although BlueLeaf system administrators were highly competent, they did not possess the security experience required to identify the source and nature of the attack, as the hackers were particularly skilled at hiding their presence. In addition, by attempting to identify the hacker responsible for compromising the system, BlueLeaf administrators risked destroying potential evidence, as well as destroying the system itself by accidentally tripping on the equivalent of a hacker landmine.

Once onsite, the Riptech consultant was brought up to speed on the events that had transpired to date. Unfortunately, due to the fact that the system in question was a critical FTP server that was used to transmit customer data between partners, BlueLeaf was unable to comply with Riptech's recommendation to take the system offline.

Proceeding with the investigation, Riptech created backups of critical data. Making an initial backup, which captures a "low level image" of a compromised system, is critical for future forensics, especially if the intruder employed advanced tools or techniques. Because system availability was critical, the system was backed up online using system utilities provided by Riptech. Riptech opted not to use BlueLeaf's system utilities because intruders often modify these. It is essential to use as little of the compromised system as possible in the actual backup process.

Often, the hard drive is removed from the compromised system and mounted read-only on a Riptech incident response (IR) system, where it is subsequently imaged in a secure environment. Disk images are used by Riptech to determine the scope of the compromise and to identify the techniques used by the intruder. Riptech also uses disk images to help identify other potential victims, as well as provide an undisturbed copy of the compromised system for evidence if BlueLeaf decided to charge the attacker in a criminal or civil case. Due to the critical nature of this particular server, removal of the hard drive was simply not possible in this case.

Riptech evaluated the compromised system in search of "fingerprints" left by the intruder and searched for files that had been created or modified recently. After a thorough review, Riptech discovered several files that were known components of a commonly available distribution of "rootkit," a popular hacker toolkit. Typically, intruders install a combination of custom and publicly available tools, such as rootkit, that allow them to disguise their presence and increase access to the compromised system. Additional MD5 checksum comparisons and examination of the system executables confirmed that a variant of rootkit was installed on the system.

The investigation revealed that the intruder's toolkit included several trojanized system executables, as noted in Figure 1.

In addition, the intruder's toolkit added a network sniffer to capture user passwords, a log cleaner to clean out any references in the system's logs, and a program to fix time/date stamps and CRC checksums (but not MD5 checksums) on the above trojanized executables. This is intended to prevent an administrator from easily detecting recent changes to files.

While inspecting these executables, Riptech uncovered a hidden directory that contained the toolkit used to attack remote systems. This tool was still running on the compromised BlueLeaf system when it was discovered. Riptech also learned that in addition to the rootkit supplied backdoor, the intruder installed an encrypted backdoor attached to a high TCP port. This technique essentially allowed the attacker to access the system with virtually no risk of detection.

Finally, upon further investigation, Riptech discovered that the intruder's password sniffer had a log file containing over 1400 username and passwords stolen from BlueLeaf employees, customers and partners.

Day 2: Developing an Action Plan

Upon completion of the initial incident forensics, BlueLeaf management expressed concern that the attacks may be a sign of corporate espionage. Management feared that the timing of the extranet compromise, which occurred at the same time as the acquisition talks, was more than a mere coincidence. Riptech believed that espionage was highly unlikely-typically compromised systems that are used by an attacker as a launching point for additional intrusions are indicative of a "random" attack. Despite Riptech's recommendation, the fear of corporate espionage led BlueLeaf to make the decision that further investigation was warranted in this case.

In order to determine the motive driving the attack, Riptech first employed a "fishbowl" tactic, which allows Riptech to monitor the attacker's activities without being observed. In order to limit further damage to BlueLeaf's internal and external systems, the compromised system was left on-line, but moved to an isolated network segment, thus minimizing the negative effects of the intruder's packet sniffer. Prior to this move, the server resided in a prime location, thus allowing the intruder to sniff all inbound and outbound connections into the corporate network.

While the intruder's network scanner was left running, the corporate firewall was modified to block outgoing traffic in order to limit BlueLeaf's potential downstream liability. While case law has not yet clarified the legal issues surrounding downstream liability, BlueLeaf's legal counsel was not willing to risk the legal liability involved in knowingly allowing the intruder to harm non-BlueLeaf systems.

Finally, Riptech provided a monitoring system to view future intruder activity remotely via an encrypted tunnel. Riptech also formulated an emergency action plan, which was to be engaged if the hacker became more destructive or attempted to gain additional access to BlueLeaf systems. If necessary, Riptech could immediately terminate the intruder's access remotely.

Days 3-5: Monitoring and Tracking the Intruder

Over the next 72 hours, Riptech observed three external sites connect to the compromised BlueLeaf server using the intruder's backdoors. None of these sites had any official business with BlueLeaf.

With BlueLeaf's permission, Riptech contacted the sites to inform them that they were involved in a hacking incident and that they were likely compromised as well. Riptech initiated these calls to the involved ISPs without identifying BlueLeaf as a victim, thus protecting BlueLeaf's identity while working to resolve the incident. Using a non-invasive approach from a remote location, Riptech was able to determine that all three systems had a backdoor similar to the one on BlueLeaf's server. Riptech observed that the compromised system was running the Linux operating system, a consistent trend up to this point in time.

Using the source code for the encrypted backdoor that was previously found on the BlueLeaf system, Riptech decrypted the intruder's connections, which allowed a keystroke-by-keystroke replay of the hacker's activity. This was a key piece of evidence for several reasons. First, it heightened suspicions that this intruder was extremely skilled. Second, it demonstrated clearly that damage was inflicted on the client infrastructure.

Analysis of the intruder's sessions also revealed that immediately after connecting to the system through the backdoor, the intruder checked recent system activity for possible detection. The intruder then immediately transferred the collection of passwords from the system, thoroughly cleaned the activity from the system logs, and exited. The entire session lasted little more than two minutes. The methods and speed of the intruder's search would have impressed even the most skilled Unix systems administrator. Because Riptech had taken the precaution of clearing the system and logs of their own activity, the intruder was completely unaware he had been discovered despite an exhaustive search for signs of detection.

Figure 2:
Anatomy of a forensic effort:
Riptech discoveries in the BlueLeaf case.

After contacting the three ISPs, only one site, GreenRoot, was willing to cooperate with the Riptech investigation. With permission, Riptech isolated and monitored GreenRoot's compromised system, employing the same methodologies used at BlueLeaf. Riptech investigated GreenRoot's servers cooperatively with GreenRoot system administrators and identified intruder fingerprints and attack tools on one of the servers. After closer analysis, Riptech determined that the tools were very similar to those found in the BlueLeaf system.

In this case, the intruder also left behind an Internet Relay Chat (IRC) client and a hidden web page with the message "KSE falcons rule!!!." Intruders often communicate with peers via IRC both socially and as a source of mentoring. Riptech suspected that the hidden web page referred to an obscure hacker group and was left by the intruder as a trophy.

On both the BlueLeaf and GreenRoot systems, the intruder was meticulous about cleaning system logs to thwart any efforts to track him. However, the intruder made one log cleaning mistake and missed two connections in GreenRoot's web access logs. According to the web acess logs, an IP address from an ISP in Kalamazoo, North Dakota visited the hacker's hidden web pages twice. Further investigation using this passive monitoring system revealed that IP addresses at an ISP in Chicago, IL and a high school in Kalamazoo, North Dakota were using the backdoors into GreenRoot's system.

Riptech contacted the ISP in Chicago, but, unfortunately, it did not log any of its users and claimed no responsibility for the actions of its users, hostile or benign. Open source searching revealed that quite a bit of known hacker activity originated from the Chicago ISP due to its liberal usage policies. This site had classic characteristics of an intruder "jump-point," not the source of the attack.

The connection from the North Dakota high school was more promising. Riptech fingerprinted the high school system, revealing that it was a Windows 98 system. Windows 98 systems are less often used as jump points as are other, more sophisticated operating systems. This find, combined with the two web access logs from the Kalamazoo ISP, indicated the high school system was a possible source rather than a "jump-point".

The IRC server to which the GreenRoot IRC client connected revealed that a hacker with the handle of "mojo" had last used it. Riptech was able to confirm that the hacker handle "mojo" had used the North Dakota ISP in the past. In addition, the "mojo" handle was used to publish several articles in 2600: The Hacker Quarterly. Riptech did more open source searching, this time targeting Kalamazoo, North Dakota. The search revealed a high school in whose mascot was a falcon and whose initials were KSE (Kalamazoo South East).

It's a wrap

These findings were sufficient to convince both Riptech and BlueLeaf that the intruder was most likely a student at Kalamazoo South East High School. With regard to a motive, Riptech's monitoring did not suggest that BlueLeaf's intellectual property was a target for the attack, nor did it reveal a pattern of attacks against BlueLeaf's partners.

With the possibility of corporate espionage ruled out, and no desire to publicly acknowledge that a teenager had compromised their corporate security, BlueLeaf and GreenRoot agreed to cease pursuing the intruder. At the direction of the client, Riptech served as an independent third party to notify all affected organizations of the incident and to provide them with several system recovery recommendations.

Riptech then helped BlueLeaf and GreenRoot recover from the intrusion by assisting with the transitioning of data on the compromised server to a secured system. Riptech also performed a limited security assessment of the BlueLeaf network to confirm proper system and firewall configuration as well as verify that BlueLeaf successfully changed all passwords compromised by the intruder. Riptech continued to monitor the affected systems for one month following the compromise. No signs of further activity from mojo were detected.

As this case study illustrates, tracking an intruder back to point of origin is a complex and time intensive process. The cost of pursuing attackers as well as the potential damage to corporate image deters most companies from pursuing attackers or involving legal authorities. For these reasons, rather than attempting to pursue the intruder, Riptech typically assists clients with recovery from the attack and take steps to minimize the future risk of system compromises.

Security Incident Checklist

Riptech encourages companies to take the following steps if they believe they have been compromised. This checklist was extracted from Steps for Recovering from a UNIX or NT System Compromise co-authored by the Computer Emergency Response Center/Coordination Center (CERT/CC) and the Australian CERT (AUSCERT).

1. Consult your security policy. A site's security policy should address the proper internal procedures for handling a computer security incident, including whom to contact, whether to involve law enforcement, public relations preparation, documentation requirements, etc. It is is ineffective to create policy during an incident, so do this now.
2. Regain control. If possible, disconnect the suspected system(s) from the network and perform a low level back up using as little of the compromised system in the process. For example, backup utilities on the compromised system should not be used to perform the backup, since the intruder may have modified them. In addition, deploy an incident response monitoring system to immediately begin monitoring the victim's network for additional intruder activity.
3. Analyze the intrusion. Review the compromised system(s) for any modifications, sniffer files, intruder tools, etc. Logs from network firewalls, intrusion detection systems and the like should also be reviewed to gain insight into the intruder's activity. Intruder's techniques are constantly evolving and fully analyzing such an incident is often better left for an experienced security response team. A majority of network incidents involve more than one compromised system; sites risk a failure to identify other compromised systems if security professionals do not handle this step.
4. Contact other sites involved. If possible, contact owners of other compromised systems so they can go through this same checklist. Many sites only discover they have been compromised when they are notified by another company that they have, in turn, compromised. A trusted third party incident response team can perform this step discretely to minimize potential adverse public response to the incident.
5. Recover from the intrusion. Install a clean version of the operating system and follow industry guidelines for securing the system, including the installation of vendor patches and disabling of all unnecessary services. Be cautious of old backups, since they may be compromised as well. Finally, ensure that all potentially compromised passwords are immediately changed.
6. Solidify the security of your network and system. Take steps to keep the intruder from re-compromising the network immediately and take final steps to identify other potentially compromised systems. Use the lessons learned from past security breaches and apply these lessons to remaining corporate assets. Review and improve firewall configuration and upgrade monitoring/logging capability based on deficiencies discovered in the incident. Consider subscribing to a service that provides real-time security monitoring and around-the-clock analysis.
7. Reconnect to the Internet. Restore connectivity, with diligent monitoring in place. Be alert for additional attempts to compromise the system.
8. Update your security policy. Take the lessons learned from the incident and apply it to your site security document to ensure that costly lessons are not wasted. Consider using the total cost of the incident to justify changes to the security posture, as well as additional security resources.

Tim Belcher is chief technology officer of Riptech, Inc., a security service provider based in Alexandria, Va. that offers real-time security monitoring as well as professional security services including forensics, assessment, auditing and policy development. www.Riptech.com

Copyright Riptech, Inc. 2000. All Rights Reserved.