The role of a security practitioner is difficult. From operational workflow changes to accommodating the latest application requirement impacting policies, it’s a relentless wave of actions to ensure that users, environments, and data are protected as effectively as possible. After all, that’s management of the attack surface.

This role becomes even more daunting when selecting a new technology to deploy in your network environment. If every product and technology your organization considered worked equally well, choosing a new technology would be more straightforward. However, some technology decisions are made based on too few data points, too little input, and, worst of all, no definitive proof that this thing you are buying works as promised.

It is essential to leverage third-party testing houses that leverage real-world use case-based tests designed to show the effectiveness of the products under consideration. This objective testing is a vital component of the evaluation process, giving you unbiased data to help you make an informed decision. Would you buy a car without taking it for a test drive? Absolutely not, and fortunately there are resources that can help you avoid making an analogous mistake in selecting effective security technology.

What does Security Efficacy mean?

To truly understand the value of security efficacy, we must start by understanding the vernacular. In the context of security testing, efficacy is the ability for a technology to accurately detect and mitigate threats while simultaneously performing its infrastructure role. That may sound like a lot, but it’s easy to break down:

• Traffic volume: Since a firewall most often sits at the ingress/egress point in a network, it must effectively handle traffic flows like a router sitting at the same place. This is especially important in a data center role.
• Application and network configuration: When setting up an effective testing environment, it should mimic a typical network setup, whether physical or virtual. The network topology must be reasonable and not on the same subnet. This environment should include the ability to test the devices’ ability to identify and handle traffic flows as they enter the network.
• Relevant and effective threats: When testing how the technology performs against application traffic, you can’t just assume a piece of malware is effective and targets applications in use. You need to validate its efficacy. Additionally, the tester must also validate that threats are effective, current, and consistently exploiting targets. You can’t use the same old threats all the time to test the technology. If we did that, everyone would eventually achieve 100% efficacy, and the test wouldn’t represent the malware variant landscape.
• Correct functionality: It is crucial to verify that not only is the technology working and blocking threats from entering the network, but also that it is behaving as expected. Confirm the correct behavior has taken place by checking the log data to ensure everything is performing as it should.

Two established organizations that do this are Mitre and CyberRatings. While Mitre focuses its tests principally on the endpoint, CyberRatings, formerly NSS Labs, specializes in assessing security technology across critical capabilities, including efficacy, management, and costs. CyberRatings recently conducted and released results for their Cloud Network Firewalls test, with many notable network security vendors submitting their virtual firewall solutions to be tested against various stress factors from performance to efficacy. This data was also coupled with a cost model based on the configuration of the AWS instances commonly used by customers. The result: a compelling cost and efficacy model to assist in qualifying the selection of a virtual firewall.

Once the efficacy is known for the tested technology, the next step in the consideration process is suitability. I recommend investigating other factors to ensure that the right decision is made, including:

• API support
• Logging details (inclusive of artifacts or indicators, etc.)
• Management workflow ease of change/add
• Licensing schemas

Want to learn more? Check out this Example report from CyberRatings on their most recent Cloud Network Firewalls test of Juniper’s vSRX virtual firewall.