Monday, October 22, 2018

Bulk IOC Search

PacketTotal now contains almost 35,000 packet captures, over 70,000,000 records, and 20,000,000,000 data-points, this has lead to some challenges around searching and sorting the data.

Over the past few months search has gotten several improvements. We modified our algorithm to pre-process queries, and optimize them prior to search. We've doubled the number of nodes on our ElasticSearch backend, and added backend load-balancing. However, even with all these improvements making large queries, those with hundreds or even thousands of strings, poses technical issues with the current infrastructure.

Obviously, being able to search our dataset quickly is a huge priority, and to get around the technical barriers we created a separate query infrastructure on AWS. This query infrastructure provides the ability to run distributed searches asynchronously, which translates to lots of queries very fast.

To test this infrastructure we have created a tool called Bulk IOC Search Utility, which can take in a list of up to 100 line separated IPs, URLs, domains, hashes, etc. We are also planning additional projects that will leverage this infrastructure. These will be tracked here.

So give it a try! Like the rest of the site it's absolutely free. If you find the tool useful please feel free to shoot us a tweet! (@PacketTotal/@TheJaminBecker

Wednesday, October 3, 2018

Introducing PCAP Tagging

Up until this point, PacketTotal has functioned as a completely autonomous analysis engine. Meaning a capture is analyzed and results generated without any additional user-interaction. Strictly, relying on automatically generated analysis can result in users making false assumptions about the benignity of the traffic contained within. Our underlying IDSs won't catch everything, and there are often times that it makes sense for a user to be able to add additional information that was missed by analysis.

Enter tagging, which addresses this gap by allowing anyone to tag a missed insights in a capture.

Any user can add tags and references to an existing PCAP, which are periodically reviewed by the PacketTotal team to ensure efficacy. 

Tagged captures will contain a "Community Tags" tab. So next time you upload a PCAP be sure to add some helpful tags so other users can find it more easily.

Friday, July 27, 2018

Private Instance & Public API

A busy seven months since our last post, and now that we're past the half way mark an update is well overdue. With that being said we are super excited about the updates we have planned, and I wanted to share some of the highlights of what we have accomplished and what we are working on this year.

General Updates

  1. WhoIs/IP2Geo lookups are experiencing transient issues, as we are having to switch backends for both services. We expect to have these services at 100% by end of August.
  2. Our team is growing! As we are a free service we are always looking for contributors! Last month we welcomed @BigDataBryce to the team who is already making some massive improvements to the codebase.
  3. Due to popular demand we've shifted almost all our focus to a private instance of the tool.

Private Instance

A lot of discussion at the beginning of the year went into prioritizing feature additions. We received quite a bit of feedback around how people we're using the tool, and what is lacking. The request repeated most often was for a private version of the tool; understandable given the sensitive nature of packet-captures.

The first half of the year was spent re-writing the analysis engine, and laying the groundwork for a several major enhancements. Because of this I have little to show from a UI perspective, however the private version will include several UI improvements that were impractical to implement on

File Rendering/PE Details

Current PCAP analysis solutions do a poor job (in my opinion) of representing extracted content. We'd love to change this by providing a simple view of extracted content, where additional post-processing is conducted against executables. 

Below is an example of media extracted, identified, enriched, and rendered in new "files" view. This view will support rendering of all major video types and images, and provide additional metadata around these. 

Analyze 1, 10 or 10,000 PCAPs relies on multiple levels of analysis in order to extract metadata and artifacts from submitted PCAPs. This process is computationally expensive and we we're hesitant to allow this on the public site for this reason. However, we considered bulk analysis a must-have feature for a private instance.

Again, the UI is still very much in the works, but you will be able to upload practically any archive format containing the PCAPs you wish to analyze.

Other Features Additions
  1. Geo View for location based data both inferred and extracted from the capture
  2. Improved Timeline View will include timelines for every protocol, not just sessions
  3. VOIP Extraction & Replay
  4. A/V detection against extracted artifacts
  5. JavaScript de-obfuscation
  6. Integration - search for similar PCAPs across our public datastore.

Public API

Even with the shift in priorities to develop a private instance of this tool we have no intention of abandoning development of the public site. Indeed, our second most requested addition was to expose search capabilities through a public API. Much of the required groundwork has already been laid to facilitate this addition. At this point most of the changes that need to be made to make a public API a reality are infrastructure related. Users can expect a public search API by end of year, early next year.

I am working to share updates more frequently with our users, if you have any feature suggestions or questions please feel free to comment below!

Thursday, January 11, 2018

New Years Resolutions - 2018 Roadmap is almost one year old, and has come a long way since the initial release of the tool. The theme for 2017 was around usability and functionality. We wanted to make the process of analyzing packet captures easy, and provide a simple platform for sharing these captures with others. With the release of PacketTotal 2.0 these objectives have mostly been completed!

Going into 2018 we are shifting our focus towards making the tool more community oriented and more accessible to developers.

The main focuses this year:
  • Improve intelligence aspects of the tool. This means new analyzers and intelligence sources as well as continued improvements to the search algorithm.
  • Build out a community. Much of the analysis found within packet-captures could be further enriched with community knowledge. You will continue to be able to submit packet captures as normal through the web-site, but will also have the option to login and be granted additional access to the tool (Details forthcoming).
  • Add features for researchers and developers. A public API has been in the works for some time, and will be released this year. Going a step further we plan to release a private virtual appliance that researchers can setup locally in their own lab environments. An SDK is also being developed and will ship with the virtual appliance.

Thank you to all those who have had a chance to use the tool or suggested additional features!

Sunday, December 3, 2017

Introducing Trending Pcaps!

Connecting users to the packet-captures most relevant to them has always been a major goal of this project. We have observed that often the most interesting packet-captures on the site are those which are repeatedly visited by our users.

December kicks off with a brand new view to make finding the packet-captures of most interest to our community even easier - Trending Pcaps. This view displays a list of packet-captures sorted by most viewed, and allows you to see which submissions are the most popular within daily, weekly, monthly, and yearly timeframes.

With the recent 2.0 release, the overhead of adding features like this has been greatly reduced, and we have lots of improvements in the pipeline.

Tuesday, November 14, 2017

Introducing PacketTotal 2.0

Introducing PacketTotal 2.0

PacketTotal 2.0 is here, and it is our biggest update to the site to date.

With this release comes a complete redesign of the user interface, and the introduction of several new features:
  • Console view now includes a Similar Packet Captures view, allowing you to essentially "search-by-pcap."
  • A completely redesigned graphs view now includes several new chart types including a time-based graphs.
  • A much more intuitive search interface allows you to much more easily locate relevant packet-captures.

Our vision is to make PacketTotal the go-to resource for analyzing, downloading, and sharing packet captures. Browse a random packet capture now, or upload your own!

Thursday, September 21, 2017

Coming Soon - PacketTotal 2.0

Normally I share these updates at the beginning of the month, but September has proven to be the busiest month since launch. Back in August the processing node engine saw a major re-write, resulting in a more modular programming interface,  allowing for new analysis engines to be added with substantially less overhead. This month has been about applying this modular paradigm to the web application itself, both to the backend and web-interface.

The changes go well beyond simple code-restructuring and engine optimizations. September has been very focused on re-thinking the UI and making it significantly more intuitive to use.

Upload and search will be accessible from the home page.

The updates to the UI extends to every aspect of the new site. Both the analysis and analytics section can be prone to bugs and slow render times during times of high-load. One of the major goals with the new interface has therefore been around improving stability and decreasing load time, especially with legacy browsers.

In a previous update I played with the idea of a static version of the site. I've since abandoned this concept as it seemed rather redundant, and instead simply changed the way the analysis console is rendered. These pages will now be generated almost completely server-side, and allow linking down to the log level, rather than just to a PCAP.

Another major component of the analysis console that is getting an update is CrossSearch. CrossSearch allows users to find similar PCAPs by using indicators in the currently open log to locate similar PCAPs. With the update, CrossSearch will be removed in favor of a Similar Packet Captures tab. Rather than only using the current log to locate similar PCAPs the new view will use all fields within the PCAP to seed the search, dramatically increasing the accuracy of the algorithm.

Similar Packet Captures: Uses all fields within the current packet capture to locate PCAPs with common attributes.
As you can imagine, this view is incredibly powerful, and effectively allows the user to "search by PCAP." In the context of malicious packet-captures the Similar Packet Captures view is also useful for intuiting which indicators would be most useful for building a signature.

Another major component of the site that is getting a face-lift is the analytics section. Like the analysis console, you will be able to link directly down to the log level within the analytics view. In addition to being able to toggle the chart which best represents your data, every log will contain a Transactions Over Time view. Clicking on any point of this graph will show transactions which occurred during that timeframe.

These updates make up about half the changes planned for the release of PacketTotal 2.0  I will be making a second post early next month to cover the updates to the new search builder and the search UI, followed later that month by the release of PacketTotal 2.0!