The BotProbe project

Big data challenge

Threat intelligence is a big data challenge. Threat intelligence data comes from multiple feeds such as network traffic data, system logs and Security Information and Event Management (SIEM).

Data volumes from traffic capture alone can be huge. A single sensor on a 10 gigabit per second network backbone has the potential to capture over 100 terabytes of PCAP data per day. Flow export protocols can reduce volumes, but NetFlow is an old protocol designed for capturing network management statistics; lacking the expected functionality of a modern flow export protocol.

Our University has developed a technique using IPFIX export protocol to turn big data capture into manageable data capture, by focusing on the data that matters in threat intelligence. This capture method takes advantage of IPFIX templates to capture any data from within a traffic packet - at a network or application level. This includes the ability to capture data from protocols that are misused for nefarious means such as HTTP, SMTP, DNS, etc.

Who needs this?

A security operation centre that captures big data volumes of network traffic as part of an established incident response policy can:

  • tune traffic data capture of networks and applications, for pre-attack forensics or on-going incident analysis
  • capture highly structured data to ease analysis and querying
  • reduce overall volumes of captured traffic that require storage for Investigatory Power Act requirements or network forensics.


As a proof of concept case study, our Department of Computing and Technology developed BotProbe to capture botnet traffic in cloud service providers. Our 11-field IPFIX template reduces data capture volumes by 8000% over PCAP. Compared to NetFlow, on average, our IPFIX template is 27% quicker with 14% less storage requirements.

BICEN: Botnet in the Cloud Eco-system for Neutralisation

BotProbe is one element of BICEN, an eco-system for botnet mitigation, being developed by our University’s Informatics, Computing and Electronics (ICE) Research Group.

The four elements of BICEN:

  • COLLECT: IPFIX allows capture of manageable data volumes of network traffic (BotProbe)
  • COMPARE: Threat profile intelligence is captured from distributed sensors (BotSense)
  • CAPTURE: Traffic profiles are compared with threat profiles to detect botnets (BotLearn)
  • CONTAIN: The AI takes appropriate action to contain the threat (BotFix)

Contact us

BotProbe and BICEN are being developed at Anglia Rusk University.

We need your help to validate the BICEN concept and shape its constituent elements. We would like to talk with you if you are:

  • interested in investment, collaboration or funding future research of BICEN
  • a cloud provider, internet service provider, or SmartCity implementer who is serious about addressing botnets
  • required to capture network data by the Investigatory Powers Act
  • any organisation with a Security Operations Centre who wants to reduce the amount to threat intelligence data in network forensics
  • an organisation running virtual networks that require protection from malware.

If you would like more information please contact Dr Mark Graham.

Research papers

Dinita, R.I., Winckles, A. and Wilson, G., 2016, July. A software approach to improving cloud computing datacenter energy efficiency and enhancing security through Botnet detection. In: Industrial Informatics (INDIN), 2016 IEEE 14th International Conference on (pp. 816-819). IEEE.

Graham, M., Winckles, A. and Sanchez, E., 2015. Practical Experiences of Building an IPFIX Based Open Source Botnet Detector. The Journal on Cybercrime and Digital Investigations. 1(1) (2015). ISSN: 2494-2715.

Graham, M., Winckles, A. and Sanchez-Velazquez, E., 2015, July. Botnet detection within cloud service provider networks using flow protocols. In: Industrial Informatics (INDIN), 2015 IEEE 13th International Conference on (pp. 1614-1619). IEEE.

Graham, M., Winckles, A. and Moore, A., 2014. Botnet Detection in Virtual Environments using NetFlow. In: CFET, 7th International Conference on Cybercrime Forensics Education & Training, Canterbury, UK, 10-11 May 2014. ISBN: 97801909067158.

Dinita, R.I., Wilson, G., Winckles, A., Cirstea, M. and Rowsell, T., 2013, November. A novel autonomous management distributed system for cloud computing environments. In: Industrial Electronics Society, IECON 2013-39th Annual Conference of the IEEE (pp. 5620-5625). IEEE.

Dinita, R.I., Wilson, G., Winckles, A., Cirstea, M. and Jones, A., 2013, February. Hardware loads and power consumption in cloud computing environments. In: Industrial Technology (ICIT), 2013 IEEE International Conference on (pp. 1291-1296). IEEE.

Dinita, R.I., Wilson, G., Winckles, A., Cirstea, M. and Jones, A., 2012, May. A cloud-based virtual computing laboratory for teaching computer networks. In: Optimization of Electrical and Electronic Equipment (OPTIM), 2012 13th International Conference on (pp. 1314-1318). IEEE.