Friday, February 1, 2019

Faster Infrastructure & Beta API - Coming Soon!

Your data model can make or break your application, but it is secondary to your infrastructure. In theory your infrastructure would dictate your initial design decisions and influence your data model. However, as a developer you often find yourself tweaking both, in parallel. This often leads to bizarre and terrible design patterns.

PacketTotal 1.0 ran on a two node ElasticSearch cluster, with a local based retention backend for raw pcaps.
This was the case with the original version of PacketTotal, and it has lead to some scalability issues. Since the initial release we have addressed several of these by adding redundancy, load-balancing, and caching, but it doesn't solve the underlying issue - that our infrastructure was designed for a few concurrent users, not dozens.
PacketTotal 2.0 ran on a much more robust ElasticSearch cluster, and migrated much of it's raw PCAP processing and retention to AWS serverless infrastructure.

To date PacketTotal has focused very much on static based PCAP analysis. As we collect, categorize, and enrich this data it becomes obvious that there is a holistic value to this it as well. A few use-cases:
Malware Archive gives you insight into malicious traffic from a variety of sources
  • Understanding how tactics, threats, and procedures (TTPs) of malicious adversaries evolve over time.
  • Identifying top threats and their targeted sectors.
  • Dynamically detecting IOCs through heuristic based approach.
  • Dynamically creating new signatures based around "known bad" and "likely bad"
  • Creating archives to categorize types of traffic interesting to students and researchers
To accomplish this we have begun the process of firstly migrating our existing data to a higher availability ElasticSearch cluster and removing some previous bottlenecks on our network. Secondly, we've re-indexed our data, and mapped it to field specific data-types. This dramatically increases search performance and accuracy as well as our ability to correlate across datasets, allowing us to start delivering on some of the use-cases above.

The new infrastructure is still undergoing testing, and will not be put into production until mid-march. In the meantime, stay tuned for the beta API release later this month which we will be making available to those interested!

Tuesday, January 1, 2019

The Malware Archive

As PacketTotal's database grows so do the challenges around categorizing that data. The main goals of 2018 were to improve the intelligence coming out of the tool and provide methods of increasing usage of the tool through improved search and community tagging. This meant providing the ability for users to tag packet captures, and re-designing the search-engine from scratch.

All of this work has positioned us well to start building out consulted intelligence views allowing us to group traffic, and understand malware trends over time.

Our first major release this year is the malware archive. The malware archive is the result of research done by the PacketTotal team and intelligence provided by the community. It is the first view that looks at data on PacketTotal at a holistic level, and we plan to create similar views to map out long term trends, and high-level metrics.

Initially, the malware view will be fairly bare-bones, containing some popular malware variants from notable categories. This list will be updated daily, and we always welcome community feedback if a listed entry is invalid.

Many exciting updates to come!

Saturday, November 3, 2018

Bulk Upload to PacketTotal

We've recently released an open-source toolset that, among other things, allows you to analyze PCAPs in bulk on The toolset can be easily inserted into your network analysis workflow and can be used to capture network traffic for an arbitrary time, or upload multiple packet captures at once for analysis.

SnappyCap also tracks the status of each analysis submission, and will quickly return whether any malicious signatures fired.

In order to begin using this tool you must first fill out this form to be granted the ability to write to our public S3 repository.

Thursday, October 25, 2018

Capture Glyphs!

Captures Glyphs are a super high level representation of the traffic inside a PCAP. At a glance, you can determine roughly how many sessions occurred within the capture, how many of those sessions were TCP, UDP, or ICMP, and the rough duration.

Each unit, or square inside a Capture Glyph represents a TCP, UDP, or ICMP session. The pixel in the middle of each square represent the duration of that session. This glyph for example represents a PCAP that contains about 50% TCP traffic 50% UDP traffic. If we zoom in we can clearly see sessions that lasted over 60 seconds, and some that were very short as well.

We are incredibly excited about this update, as it provides a brand new (and kind of beautiful) way of visualizing packet captures. A glyph is automatically created when you upload a PCAP to If you have a cool glyph, be sure to tweet it at us!(@PacketTotal/@TheJaminBecker)

Monday, October 22, 2018

Bulk IOC Search

PacketTotal now contains almost 35,000 packet captures, over 70,000,000 records, and 20,000,000,000 data-points, this has lead to some challenges around searching and sorting the data.

Over the past few months search has gotten several improvements. We modified our algorithm to pre-process queries, and optimize them prior to search. We've doubled the number of nodes on our ElasticSearch backend, and added backend load-balancing. However, even with all these improvements making large queries, those with hundreds or even thousands of strings, poses technical issues with the current infrastructure.

Obviously, being able to search our dataset quickly is a huge priority, and to get around the technical barriers we created a separate query infrastructure on AWS. This query infrastructure provides the ability to run distributed searches asynchronously, which translates to lots of queries very fast.

To test this infrastructure we have created a tool called Bulk IOC Search Utility, which can take in a list of up to 100 line separated IPs, URLs, domains, hashes, etc. We are also planning additional projects that will leverage this infrastructure. These will be tracked here.

So give it a try! Like the rest of the site it's absolutely free. If you find the tool useful please feel free to shoot us a tweet! (@PacketTotal/@TheJaminBecker

Wednesday, October 3, 2018

Introducing PCAP Tagging

Up until this point, PacketTotal has functioned as a completely autonomous analysis engine. Meaning a capture is analyzed and results generated without any additional user-interaction. Strictly, relying on automatically generated analysis can result in users making false assumptions about the benignity of the traffic contained within. Our underlying IDSs won't catch everything, and there are often times that it makes sense for a user to be able to add additional information that was missed by analysis.

Enter tagging, which addresses this gap by allowing anyone to tag a missed insights in a capture.

Any user can add tags and references to an existing PCAP, which are periodically reviewed by the PacketTotal team to ensure efficacy. 

Tagged captures will contain a "Community Tags" tab. So next time you upload a PCAP be sure to add some helpful tags so other users can find it more easily.

Friday, July 27, 2018

Private Instance & Public API

A busy seven months since our last post, and now that we're past the half way mark an update is well overdue. With that being said we are super excited about the updates we have planned, and I wanted to share some of the highlights of what we have accomplished and what we are working on this year.

General Updates

  1. WhoIs/IP2Geo lookups are experiencing transient issues, as we are having to switch backends for both services. We expect to have these services at 100% by end of August.
  2. Our team is growing! As we are a free service we are always looking for contributors! Last month we welcomed @BigDataBryce to the team who is already making some massive improvements to the codebase.
  3. Due to popular demand we've shifted almost all our focus to a private instance of the tool.

Private Instance

A lot of discussion at the beginning of the year went into prioritizing feature additions. We received quite a bit of feedback around how people we're using the tool, and what is lacking. The request repeated most often was for a private version of the tool; understandable given the sensitive nature of packet-captures.

The first half of the year was spent re-writing the analysis engine, and laying the groundwork for a several major enhancements. Because of this I have little to show from a UI perspective, however the private version will include several UI improvements that were impractical to implement on

File Rendering/PE Details

Current PCAP analysis solutions do a poor job (in my opinion) of representing extracted content. We'd love to change this by providing a simple view of extracted content, where additional post-processing is conducted against executables. 

Below is an example of media extracted, identified, enriched, and rendered in new "files" view. This view will support rendering of all major video types and images, and provide additional metadata around these. 

Analyze 1, 10 or 10,000 PCAPs relies on multiple levels of analysis in order to extract metadata and artifacts from submitted PCAPs. This process is computationally expensive and we we're hesitant to allow this on the public site for this reason. However, we considered bulk analysis a must-have feature for a private instance.

Again, the UI is still very much in the works, but you will be able to upload practically any archive format containing the PCAPs you wish to analyze.

Other Features Additions
  1. Geo View for location based data both inferred and extracted from the capture
  2. Improved Timeline View will include timelines for every protocol, not just sessions
  3. VOIP Extraction & Replay
  4. A/V detection against extracted artifacts
  5. JavaScript de-obfuscation
  6. Integration - search for similar PCAPs across our public datastore.

Public API

Even with the shift in priorities to develop a private instance of this tool we have no intention of abandoning development of the public site. Indeed, our second most requested addition was to expose search capabilities through a public API. Much of the required groundwork has already been laid to facilitate this addition. At this point most of the changes that need to be made to make a public API a reality are infrastructure related. Users can expect a public search API by end of year, early next year.

I am working to share updates more frequently with our users, if you have any feature suggestions or questions please feel free to comment below!