Analyze PCAP API & Open-source SDK

We have talked about an analysis API since PacketTotal started over two years ago, and today we are incredibly excited to announce we now support submissions through our REST API.

Currently, you can analyze .pcap/pcapng files up to 6MB, and they must first be encoded to base64. These limits will likely be raised as we scale our infrastructure and data-model.

If REST APIs aren't your thing, you can also use the open-source SDK.

SDK Analyze Implementation

Researchers can also leverage the packettotal command-line utility that comes packaged with the SDK.

We are hoping that this added functionality will increase the usefulness of the tool, and open up exciting integration opportunities!

A Faster PacketTotal

Our Search API beta has dramatically increased the volume of concurrent requests to the engine, requiring us to re-think some of our initial infrastructure choices.

PacketTotal has previously performed noticeably slower when handling large volumes of traffic. To address this we have migrated our ElasticSearch cluster into a AWS high-availability environment, and made several optimizations to the backend data-model.

This update also makes major improvements to the speed and fidelity of the similar PCAPs view; reported timeouts have also been addressed.

Finally, we have taken this opportunity to add some small enhancements to the intel view, which now links to external intelligence sources.

More API updates right around the corner, stay tuned!

Search API Beta

Today we are incredibly excited to announce the beta preview of our Search API!

With it you can search, correlate, and download PCAP files, based on their behaviors and contents.

Exploring the API's Use-Cases

Understand how tactics, threats, and procedures (TTPs) of malicious adversaries evolve over time.

Network traffic is among some of the richest forms of evidence available to security professionals, it is also one of the hardest to analyze at scale. PacketTotal, has analyzed thousands of malicious packet captures, and derived millions of signatures, relationships, and correlations between them.
The Search API provides a powerful interface for finding network traffic that can help you understand, and anticipate the actions of malware in your own environment. Simple search for an indicator of compromise and pull back matching captures, use AND and OR aggregators to get as granular as you want with your queries.

Identifying top threats and their targeted sectors.

Understanding the goals of malicious adversaries is the first step in proactive security.
From malware detonated in a lab to evil found in the wild, PacketTotal is the most diverse set of malicious packet captures on the Internet.
Search by the latest malware campaigns, and see how it communicates, who they're targeting, and what methods they employ. 

Build or validate your network signatures through heuristic based approach.

PacketTotal does a good job of providing high-level categorizations of traffic found within packet captures. Through the API you can download captures that you could use to validate controls within your own environment.

Train your Machine Learning models!

If you're a data-scientist, and are lucky enough to be studying malicious network behaviors, you may find the bulk search and download functionality of the search api to be very useful in training your models. 

Interested in being a part of the Beta?

Ultimately, the search api is the first step in a much more comprehensive API that will offer full feature parity with the site (and more). If anything about this post excites you please consider giving our API a spin!

Faster Infrastructure & Beta API - Coming Soon!

Your data model can make or break your application, but it is secondary to your infrastructure. In theory your infrastructure would dictate your initial design decisions and influence your data model. However, as a developer you often find yourself tweaking both, in parallel. This often leads to bizarre and terrible design patterns.

PacketTotal 1.0 ran on a two node ElasticSearch cluster, with a local based retention backend for raw pcaps.

This was the case with the original version of PacketTotal, and it has lead to some scalability issues. Since the initial release we have addressed several of these by adding redundancy, load-balancing, and caching, but it doesn't solve the underlying issue - that our infrastructure was designed for a few concurrent users, not dozens.

PacketTotal 2.0 ran on a much more robust ElasticSearch cluster, and migrated much of it's raw PCAP processing and retention to AWS serverless infrastructure.

To date PacketTotal has focused very much on static based PCAP analysis. As we collect, categorize, and enrich this data it becomes obvious that there is a holistic value to this it as well. A few use-cases:

Malware Archive gives you insight into malicious traffic from a variety of sources

• Understanding how tactics, threats, and procedures (TTPs) of malicious adversaries evolve over time.
• Identifying top threats and their targeted sectors.
• Dynamically detecting IOCs through heuristic based approach.
• Dynamically creating new signatures based around "known bad" and "likely bad"
• Creating archives to categorize types of traffic interesting to students and researchers

To accomplish this we have begun the process of firstly migrating our existing data to a higher availability ElasticSearch cluster and removing some previous bottlenecks on our network. Secondly, we've re-indexed our data, and mapped it to field specific data-types. This dramatically increases search performance and accuracy as well as our ability to correlate across datasets, allowing us to start delivering on some of the use-cases above.

The new infrastructure is still undergoing testing, and will not be put into production until mid-march. In the meantime, stay tuned for the beta API release later this month which we will be making available to those interested!

The Malware Archive

As PacketTotal's database grows so do the challenges around categorizing that data. The main goals of 2018 were to improve the intelligence coming out of the tool and provide methods of increasing usage of the tool through improved search and community tagging. This meant providing the ability for users to tag packet captures, and re-designing the search-engine from scratch.

All of this work has positioned us well to start building out consulted intelligence views allowing us to group traffic, and understand malware trends over time.

Our first major release this year is the malware archive. The malware archive is the result of research done by the PacketTotal team and intelligence provided by the community. It is the first view that looks at data on PacketTotal at a holistic level, and we plan to create similar views to map out long term trends, and high-level metrics.

Initially, the malware view will be fairly bare-bones, containing some popular malware variants from notable categories. This list will be updated daily, and we always welcome community feedback if a listed entry is invalid.

Many exciting updates to come!