Do We Need a Pentest?

Having firmly established my reputation as the HexCISO – the security antichrist, who constantly tried to turn the security dial down, in clear violation of all the rules of cyber career and product promotions – as you know, I occasionally like to write these little pieces about down-to-earth topics in security.

Very basic, non-exciting, shockingly practical stuff.

Stuff that everyone who isn’t in security actually does care about.

Like… Do we need a pentest?

One of my colleagues wrote to me with this very same question. It pops up regularly enough that, as I find myself with an unexpected couple of free hours this morning, I figured I’d ask and answer here, with the goal of handing out a fishing pole, rather than a fish.

The context of his question was a bit more involved, though, so let’s start there. What he asked was this: “we just initiated a bug bounty program. Do we need to keep our annual pentest?”

Let’s work through this most mundane of security questions, shall we? Because by working through it, we can learn a lot about developing a rational security strategy amidst the cyber-hubbub (have I managed to invoke a kebab in your mind? Mhm… delicious).

The key word in the question is need. It denotes some sort of requirement to be fulfilled or satisfied.

What sort of requirement would drive a need for pentesting? There are two common ones, actually.

First is quasi-regulatory, usually via externally audited compliance. We can name all sorts of standards here, like SOC2 and PCI, but also things like SEC reporting for public companies. It may make it easy to satisfy these kinds of requirements with an annual pentest; it’s a version of having disk encryption in a cloud storage system, a mostly useless cyber control but one that makes box checking simple, which brings significant value in headache mitigation.

Second is in the realm of commercial/contractual clauses. Since the field of security is an ecosystem of intersecting standards, rules, and best practices, many large enterprise customers have incorporated the requirement in their commercial paper for their (especially cloud) vendors to perform an annual pentest and share the results with them. The cyber value of this methodology is debatable, but it can be clearly dotted-lined to establishing commercial accountability in the world of TPRM (Third Party Risk Management). It’s a different form of box checking but it can help smooth commercial transactions, and that can indeed be valuable.

Of course, the latter is only applicable to b2b organizations who actually have those kinds of customers (a.k.a large enterprise).

That’s it for need, pretty much. It’s worth noting that in the compliance category of need, it is sometimes easy to argue that pentests aren’t required due to, say, the existence of other forms of testing. This will depend on the skills of your auditor as much as the black-letter-iness of the requirement in the standard; to continue our example, in SOC2 you can often state “a bug bounty program is essentially a form of continuous application pentesting and all we do is SaaS, so it’s enough,” but in PCI you might find it harder to convince an inexperienced assessor because PCI is very explicit in this regard.

This may well eliminate the need altogether, especially if you’re not selling to large enterprise.

Are we done?

Not in the least. Let’s get back to basics.

Get ready for the fishing pole.

Because the next question is, do we want a pentest?

This is the true crux of the matter, and it applies equally to the entire cyber realm.

In developing a security strategy, one has to consider testing of one’s control environment. Otherwise, you’re just assuming that things are working fine until, well, they don’t. An annual compliance-driven network pentest (usually the cheapest kind, too) will provide a SaaS vendor with exactly zero value with respect to the actual resilience of their cyber defenses. Zero, because while it may highlight some basic surface weaknesses in the posture, it often instills a false sense of security that is large enough to counteract the benefit, and it all averages out.

But what does testing mean? What should it include, and when? This is where people often stumble, because thinking through your approach to testing can be more demanding than checking the annual pentest box.

It gets even more difficult when considered in the context of the business environment. For example, if the company must conserve cash because it’s running out of money and can’t raise its next round of funding – an all too common fight for survival in the tech world – and it isn’t helping you in getting customers (no need), then by all means, drop the annual pentest and save the cash. As your CISO, I would suggest it to you before we even get through the small talk.

But let’s say you’re in a healthy business, and you have some budget for testing. You even have a bug bounty program, but having considered it, you realize you don’t need a pentest.

Is it still desirable?

For annual network pentests the answer is usually “no”. But that doesn’t mean pentests are useless! For example, how about rotating, smaller tests from different specialty vendors that exercise different portions of your environment, say every six months? So you might stick a network one in there every so often if you actually manage a network (cloud-native companies don’t, so it really is a waste of money and effort), but you may want to check your mobile apps tomorrow, your website in six months, your SaaS app from an insider’s perspective in twelve. In eighteen months, swallow your fear and test your staff’s resilience to social engineering attacks with the explicit goal of compromising access to, say, your cloud financial systems of record or even your company bank account.

The first time you run that last one, by the way, will give you far more value than any other test you ever run, because humans are always the weakest cyber link. But it can cost a lot, and not many firms can be trusted or even know how to do it. We executed one of these for a massive organization once, and even though they asked for it, the CTO’s reaction when we showed them how we were one click away from exercising all their stock options for our personal benefit was not the kindest.

The fear was palpable. But it did lead to improvements, so it was a win.

Note that none of this cares about need. If you actually care about security as opposed to paying lip service to it, then it’s time to separate the streams – in this case, of need and want. And it all has to make sense in the bigger context of your business, its accrued liabilities, and where it is in the business cycle.

So screw the schedule, stop thinking of security as compliance, and definitely stop thinking that your annual pentest is some sort of magical threat hunter. It isn’t, not the way it’s typically performed under the guise of need.

You may well find this form of testing desirable and valuable, though.

As my last note on the soapbox before I leave you, consider applying this methodology to your entire security strategy. As a security leader, everything you do has a hidden cost. It’s the fundamental truth of the Dismal Discipline. Train yourself to be conscious of it in everything you do, every decision you make. Your colleagues, for one, will appreciate it, and that can do wonders for your career advancement.

And to said colleagues: does your CISO exhibit this form of well-considered thinking? Or do they simply wield a hammer, demanding authority and complaining when they don’t get their way?

And there’s your fishing pole.

Adieu.