MITRE ATT&CK matrix has become the standard for classifying potential behaviour of opponents. Its popularity has been growing in the last two years. The advantages of this framework for vendors are clear: they can now map, adapt and improve their detection capabilities. The benefits for other cybersecurity actors are clear as well, such as Red Teams, who can now better emulate the activity of opponents. Or for cybersecurity managers, who can better and more systematically assess their defences and identify gaps. ATT&CK is a large database used to identify common tactics, techniques, and procedures (TTPs) used by advanced persistent threats against computer platforms.
MITRE now incorporates significant changes to its ATT&CK Matrix. On March 31st, as an evolution of the model and part of its roadmap for this year, MITRE released the beta version of the new matrix. New techniques, some discontinued, changes in names and descriptions, and perhaps the newest change: the introduction of sub-techniques, thus adding one more level to the structure of the matrix. In this way, now we have tactics, techniques, sub-techniques and procedures. In total more than 340 techniques and sub-techniques mapped into the 12 tactics that make up the columns of the matrix. A titanic effort by MITRE, that with this update makes available the most complete and systematic catalogue of cyberattacker’s behaviour to date.
Using MITRE ATT&CK to Evaluate Security Products
Despite the great progress brought by this initiative, there is some confusion regarding the interpretation of the application of the model, especially when evaluating security solutions.
“ATT&CK assessments should define the type of coverage to be measured, according to the objectives of each assessment. More ‘horizontal’ coverage does not mean better protection.”
Firstly, and following the publication of the results of the first EDR-type product evaluation carried out by MITRE in 2018 (called Round 1), some participants – not MITRE -, tried to equate the degree of detection coverage of the techniques with a higher protection quality or effectiveness. MITRE itself points out that ATT&CK documents the known behaviour of opponents and does not attempt to be a checklist.
Not all these behaviours can or should be used as the basis for sending alerts or data to an analyst. However, it is tempting for a vendor to display graphs representing a greater degree of coverage of the matrix techniques, regardless of the tactic they belong to. Therefore, the concept of coverage can be misleading. The techniques (and now the sub-techniques as well) can be implemented in many changing ways (procedures) by the attackers, and it is very difficult to know all of them a priori. The result can be different if one or another procedure is evaluated for the same technique and product.
In a study published by OPTIV in 2019, we could verify the different behaviour of several security products, testing several procedures for the same technique vs. testing just one (reference). Therefore, more horizontal coverage does not necessarily mean “better” product.
In fact, not all tactics may have the same weight or importance for an organisation evaluating a given product. For example, if the actual protection capability (including prevention) is being assessed, then special attention should be paid to deeper, more procedural coverage for “left-sided” tactics in the matrix. By doing so, attackers’ attempts to gain initial access to target systems, malicious code execution, or attempts to gain persistence on those systems will be blocked.
These considerations must be established according to the objectives of each organisation when considering the use of ATT&CK. The use cases may be different, and therefore the relative weight and type of coverage for tactics and techniques may be different as well. This is not covered by the framework itself. To complete this point, we should also remember that some evaluations require products to have prevention and blocking functionalities disabled, as occurred in the Round 1 carried out by MITRE, previously mentioned. Otherwise, it would not have been possible to evaluate the detection capabilities of the products.
Model Evolution
Currently, we are waiting for MITRE to publish the results of a second product evaluation (called Round 2). The number of techniques and the complexity inherent in the testing of security products means that this work requires increasingly more effort. At the same time, other security solution testing companies are also working to adapt their testing and mapping to the MITRE model while emulating real attack scenarios, without the limitations outlined above. Without a doubt, there is still a long way to go in this area.
“The model will evolve to include the perspective of defenders and the systematisation of analyses and forms of prevention, detection and response.
In the future, we also expect greater attention to other interesting projects (from MITRE and others) focused not only on the behaviour of opponents, but also on the cataloguing of defences to detect them systematically, as is the case of LOLBAS (Living Off the Land Binaries and Scripts). A compilation of LOLBAS techniques is available on GitHub. Another example from MITRE itself also focused on the defender’s perspective is the Cyber Analytics Repository (CAR) project: compiling analyses that can be used by defenders to detect attacks, also mapping them with the ATT&CK matrix. Therefore, the model will be extended to include the systematisation of defences, from prevention to detection and response.
