Processing Node 2.0 & Intel Analyzer

Back in early February I began working towards consolidating PacketTotal's three major components into the same codebase. The eventual goal being a turnkey virtual appliance that security researchers can install locally on their own network for quick PCAP analysis. Previously, the processing nodes, elastic-cluster, and front-end components could not be installed on the same host.  This was mostly because of the way multithreading was implemented in version 1.x processing nodes.

For those unfamiliar with the PacketTotal backend, processing nodes are responsible for receiving and replaying packet-captures through Bro and Suricata, parsing the logs, and delivering the results to the elastic-backend, via the Elastic document API. Besides solving issues with multithreading, version 2.0 introduces a much more modular programming interface, which allows new analyzers to be added quickly and with significantly less code. Expect more analysis engines this year! Version 2.0 also introduces the concept of "analysis stages" to track which engine is currently analyzing your PCAP.

New analysis status page fully implements analysis stages.

The first of these new analysis engines to be introduced to the processing nodes is the "Intel Analyzer." It uses high fidelity indicators found by Suricata and attempts to link them to relevant external content, such as blog posts or write-ups, using that extracted indicator. For example if your packet-capture contains an IP address that is known to be malicious, you may find additional information about that IP in the "Intel Community" tab within the analysis console.

August will be primarily focused on improving the front-end and merging the overlapping storage APIs into one codebase. Fixing search is also high on the list as it is still too fickle in my opinion.

More updates soon to come, in the mean time give the new engine a try!