To identify and prevent outages, but also to protect against cyberattacks and to design a more efficient IT infrastructure, MIT has developed a supercomputer capable of displaying real-time global traffic on the Internet.
The Internet is constantly evolving, with a growing number of connected devices in growing global traffic. The analysis of these data transfers in their entirety represents a real challenge for researchers who could learn a lot. With this information, they could help create a more efficient network, prevent outages and improve cyberattack defenses.
Until recently, the tools were lacking, but a team of researchers at the Massachusetts Institute of Technology (MIT) managed to create a global Internet traffic analysis tool based on a supercomputer. They analyzed nearly 50 billion packets of data collected since 2015 in Japan and California.
A measure of the background noise of our virtual universe
To do this, they were first forced to process these data “hypercreuses”, namely data that have more empty spaces that values, with a technique called Dynamic Distributed Dimensional Data Model (D4M). They then created a neural network that runs on the 10,000 processors of MIT SuperCloud, and that was used to analyze the data to find relationships, both to the giants of the web, such as Google or Facebook, than to rare links that have yet an impact on web traffic.
The researchers have assimilated the results to a measure of the background noise of the internet, which makes it possible to identify the anomalies and to obtain information on the file-sharing, the malicious IP addresses, and the spam, the distribution of the attacks, as well as data traffic jams.