Introducing Trending Pcaps!

Connecting users to the packet-captures most relevant to them has always been a major goal of this project. We have observed that often the most interesting packet-captures on the site are those which are repeatedly visited by our users.

December kicks off with a brand new view to make finding the packet-captures of most interest to our community even easier - Trending Pcaps. This view displays a list of packet-captures sorted by most viewed, and allows you to see which submissions are the most popular within daily, weekly, monthly, and yearly timeframes.

With the recent 2.0 release, the overhead of adding features like this has been greatly reduced, and we have lots of improvements in the pipeline.

Introducing PacketTotal 2.0

Introducing PacketTotal 2.0

PacketTotal 2.0 is here, and it is our biggest update to the site to date.

With this release comes a complete redesign of the user interface, and the introduction of several new features:

• Console view now includes a Similar Packet Captures view, allowing you to essentially "search-by-pcap."
• A completely redesigned graphs view now includes several new chart types including a time-based graphs.
• A much more intuitive search interface allows you to much more easily locate relevant packet-captures.

Our vision is to make PacketTotal the go-to resource for analyzing, downloading, and sharing packet captures. Browse a random packet capture now, or upload your own!

Coming Soon - PacketTotal 2.0

Normally I share these updates at the beginning of the month, but September has proven to be the busiest month since launch. Back in August the processing node engine saw a major re-write, resulting in a more modular programming interface,  allowing for new analysis engines to be added with substantially less overhead. This month has been about applying this modular paradigm to the web application itself, both to the backend and web-interface.

The changes go well beyond simple code-restructuring and engine optimizations. September has been very focused on re-thinking the UI and making it significantly more intuitive to use.

Upload and search will be accessible from the home page.

The updates to the UI extends to every aspect of the new site. Both the analysis and analytics section can be prone to bugs and slow render times during times of high-load. One of the major goals with the new interface has therefore been around improving stability and decreasing load time, especially with legacy browsers.

In a previous update I played with the idea of a static version of the site. I've since abandoned this concept as it seemed rather redundant, and instead simply changed the way the analysis console is rendered. These pages will now be generated almost completely server-side, and allow linking down to the log level, rather than just to a PCAP.

Another major component of the analysis console that is getting an update is CrossSearch. CrossSearch allows users to find similar PCAPs by using indicators in the currently open log to locate similar PCAPs. With the update, CrossSearch will be removed in favor of a Similar Packet Captures tab. Rather than only using the current log to locate similar PCAPs the new view will use all fields within the PCAP to seed the search, dramatically increasing the accuracy of the algorithm.

Similar Packet Captures: Uses all fields within the current packet capture to locate PCAPs with common attributes.

As you can imagine, this view is incredibly powerful, and effectively allows the user to "search by PCAP." In the context of malicious packet-captures the Similar Packet Captures view is also useful for intuiting which indicators would be most useful for building a signature.

Another major component of the site that is getting a face-lift is the analytics section. Like the analysis console, you will be able to link directly down to the log level within the analytics view. In addition to being able to toggle the chart which best represents your data, every log will contain a Transactions Over Time view. Clicking on any point of this graph will show transactions which occurred during that timeframe.

These updates make up about half the changes planned for the release of PacketTotal 2.0  I will be making a second post early next month to cover the updates to the new search builder and the search UI, followed later that month by the release of PacketTotal 2.0!

Processing Node 2.0 & Intel Analyzer

Back in early February I began working towards consolidating PacketTotal's three major components into the same codebase. The eventual goal being a turnkey virtual appliance that security researchers can install locally on their own network for quick PCAP analysis. Previously, the processing nodes, elastic-cluster, and front-end components could not be installed on the same host.  This was mostly because of the way multithreading was implemented in version 1.x processing nodes.

For those unfamiliar with the PacketTotal backend, processing nodes are responsible for receiving and replaying packet-captures through Bro and Suricata, parsing the logs, and delivering the results to the elastic-backend, via the Elastic document API. Besides solving issues with multithreading, version 2.0 introduces a much more modular programming interface, which allows new analyzers to be added quickly and with significantly less code. Expect more analysis engines this year! Version 2.0 also introduces the concept of "analysis stages" to track which engine is currently analyzing your PCAP.

New analysis status page fully implements analysis stages.

The first of these new analysis engines to be introduced to the processing nodes is the "Intel Analyzer." It uses high fidelity indicators found by Suricata and attempts to link them to relevant external content, such as blog posts or write-ups, using that extracted indicator. For example if your packet-capture contains an IP address that is known to be malicious, you may find additional information about that IP in the "Intel Community" tab within the analysis console.

August will be primarily focused on improving the front-end and merging the overlapping storage APIs into one codebase. Fixing search is also high on the list as it is still too fickle in my opinion.

More updates soon to come, in the mean time give the new engine a try!

Statistics New Look & Development Updates

This week PacketTotal got a much needed update to the statistics page. Along with the original metrics, the statistics page will now display upload counts spanning a week long period. As with the rest of the site the statistics view is a work and progress and will continue to be improved as the tool matures.

These past two months have been development heavy on multiple fronts. The continued work on a virtual appliance has been slow, as the entire interface needs to be re-worked. Ideas get added to the board, some implemented, others discarded as impractical or unscalable. The processing node itself has also experienced some hiccups in production. I have begun a complete re-write of the underlying agent, with the goal being to be running version 2.0 of the agent, with plug and play Bro scripts by the end of the year. Fortunately, most of the development work on the virtual appliance also benefits packettotal.com, so users can expect a better experience every time they visit the site. 

Another soon-to-be-added section of the site will be the archive. The archive will be a static version of packettotal.com, easily indexable and searchable on Google and other search-engines. A continued goal of this site is to make information found within malicious packet-captures easily accessible to the security community. While our built in search has been improved significantly since launch, having static content indexable by major search engines will improve people's ability to locate information within the tool.

The archive will be re-generated on a daily basis, and will also act as a front-end for additional post-processing found inside PCAPs! Initially, the tool will attempt to link high-fidelity malicious IOCs to relevant content on the web -- such as forums posts, recent news, or blog articles. Additional post-processing will leverage an improved version of the cross-search algorithm to link similar PCAPs and allow users to easily pivot between results.

As always I welcome any feature suggestions/improvements to the tool.

Introducing SearchBuilder!

Anyone who has searched PacketTotal.com has probably experienced frustration with finding relevant results. Up until this point creating specific queries was impossible because the field names displayed in the analysis console are not the same as those used on the backend. So running a search like Target IP: 8.8.8.8  simply would not work because "Target IP" exists as the field "id_resp_h" within ElasticSearch.

SearchBuilder bridges this gap, allowing you to build complex Lucene Queries to search our database. 

For example, you could craft a query to return results of PCAPs containing suspicious executables with the below query. 

_type:pe AND os:*Windows* AND (section_names:*UPX* OR section_names:*TLS*)) OR alert_signature:*exe*

This particular query checks for PCAPs containing at least one Windows executable which contains UPX (common packer) OR TLS (common anti-debugging technique) section. It will also return results if the Alert Signature from Malicious Activity (signature_alerts) contains the keyword .exe.

We could craft another signature to look for command and control traffic over IRC.

(_type:irc AND NOT id_resp_h:6667) OR (_type:signature_alerts AND alert_signature:*irc*)

Notice, in this example we use a NOT operator to look for IRC traffic on a non-standard port.

The SearchBuilder interface is fairly intuitive. To get started, click the dropdown arrow directly below the search bar located on the search page. 

Select a template from the templates dialog. This will populate all the fields available for search within the selected template. Due to the way the backend schema is designed you cannot AND fields from multiple templates together. For example combining fields for an HTTP specific URI AND a FTP specific target port will not work, as no one document will contain both of these fields. You could however, OR fields from multiple templates together without an issue.

The search terms dialog allows you to click any of the available terms, appending them to your search. By default the equals toggle is selected in the top right. This inserts quote characters around the placeholder, ensuring that only exact matches will be returned. You can toggle this option to use contains, which will insert asterisks on both sides of the placeholder. Toggling contains will look for any PCAP which has a field containing the text between the asterisks. 

Finally, the search operators dialog provides a quick way of adding aggregation and negation logic into your query. By default, when you select a new term, it will be AND'd with the previous term. This again can be toggled to use the OR operator instead.

SearchBuilder is yet another tool to improve the intelligence being gathered from this tool. Please feel free to email me with improvements or suggestions.

The CrossSearch Algorithm

When I first started building PacketTotal I wanted to give users the ability to find packet-captures similar to the one they uploaded.


Starting early on in the project I began writing a suite of algorithms to accomplish this in a reasonable amount of time. Unfortunately, at the time of the initial release, I did not have enough data to test the algorithm effectively, and was hesitant to release it to the community. However, with the number of submissions growing, I have been able to more effectively test this tool.

 The algorithms themselves are fairly straight forward, but in a nutshell they:

1. Use a simple heuristic to identify promising search terms within the metadata of a particular capture.
2. De-duplicate and aggregate these terms, constructing Lucene queries which can be used to search the Elastic backend.
3. Run the queries against Elasticsearch.
4. Group results by matched packet-captures, sorted by captures which contain the most matched terms.

The result - An incredibly powerful view, potentially allowing the users to pivot between captures that have similar attributes.

From a malware analysis standpoint the CrossSearch algorithm could give researchers the ability to tie together two or more seemingly unrelated campaigns - If, say, they shared a common C2 server or another IOC that could easily be missed in a manual analysis.

The first version of CrossSearch will be released later this week once final integration testing is completed, and a bit more tuning of the heuristic is done. The tool will be made available within the analysis console and will work against the currently selected view. Meaning, if the "Connections" tab is open,  running a CrossSearch will use the terms found within "Connections" to search the backend.

CrossSearch is the first of many tools that will greatly improve the value of the data being gathered on PacketTotal.com.