I was called by a client to investigate a strange behaviour in their network infrastructure. It turned out to be a major network security breach. Of course, I cannot disclose the full details of the incident as Non Disclosure Agreement rules apply. But I wanted to share some lessons and tips learned from that event to help other cyber security professionals and network administrators.
How Strange Is Weird?
It all started with a strange behaviour on the core network infrastructure. Why should a core network device attempt to make regular outbound DNS requests to random home computers in Asia? Why should a core network device make constant HTTP requests to unknown home computers in Russia and then flood the local network with DDoS and bring down critical network services? These are some of the symptoms that caught my attention.
Further investigation using sys-admin and network tracing tools revealed that one of the core network devices had been compromised. The attacker installed a kernel rootkit that took over the network stack of the device allowing it to join a global botnet that was using several command and control centres spread around the globe. Further ‘Cyber Crime Scene Investigation’ (CCSI) was required to get to the bottom of what this rootkit was up to.
Having established the root cause of the strange network behaviour, the challenge was now to gather and analyse all the information in order to better understand what else could have been infected in the core network. The malware seemed to have some sort of awareness about its local network surroundings. With full control of the infected device network stack, the malware was able to monitor local activities and decide when, how, and what to connect to in order obtain further commands. By simulating the right conditions the malware was able to replicate several assaults allowing the collection and analysis of traffic patterns that provided the definitive answer of the inner workings of the rootkit.
Killing The Zombie Device Dead
Following a detailed analysis of network traces and system processes behaviour on the compromised device it was clear that this device was now a fully grown ‘zombie’ and had to be given a new life. Having collected all the evidence, a fresh and clean install was required to ensure no traces remained from the earlier breach. Cleaning up the offending device was just the start. Ensuring that the rootkit had not infected any other network components was crucial. A thorough analysis and monitoring of the whole network infrastructure and locked down was required to ensure that the malware was completely removed and cleanse. Of course, the lessons learned highlighted below had to be applied in order to strengthen the security of the overall infrastructure and reduce the likelihood of such breach occurring again in the future.
From this event I have drawn 7 lessons – most of which are common sense approaches to security – but nevertheless must serve as a reminder that security is an on-going battle between technology, humans, and processes. The three must work together to ensure that network infrastructures and applications remain secure and always available.
- Tightly control remote access: Never allow access to a network device from the Internet without strong authentication and ideally 2-factor authentication with a strict Access Control List (ACL) to restrict what, where, when, and who can manage that device.
- Defend in many layers: A defence-in-depth architecture should be followed to strengthen the security of the overall network infrastructure. By doing so this will greatly reduce the risk of a single component infecting the entire network infrastructure.
- Monitor, detect, and re-mediate: A robust monitoring, detection, and re-mediation system and processes should be in place to establish a baseline of normal network traffic behaviour. From that baseline, anomalies can be detected quickly and re-mediation applied promptly. Security analytics must play a key role here.
- Plan and prepare for re-mediation: The question is not if a network will be hacked but when. Given enough time and will any network can be hacked therefore it is necessary to have regular drills and a clear response plan to prepare for a major breach.
- The cyber battle never stops: Cyber security is an on-going battle between humans, processes, and technology. Technology alone cannot guarantee cyber security but a mix of the three will provide the strongest network and application security defence.
- Tightly control all inbound and outbound flows: Most companies strongly control inbound access to their network infrastructure and services but outbound access is often left unchecked. By putting in place strict rules for outbound connectivity a single device breach will remain isolated and reduce the chance of a malware infection spreading. For a rootkit this means starving it from connecting to its control and command centre and stopping it from downloading further commands as a stepping-stone to amplify its attacks.
- Establish a secure baseline of device build: All network devices should be hardened and added to the network with a secure baseline to ensure consistency and to avoid basic mistakes. This procedure should be much stricter for all Internet facing devices.