Sustainable and Scalable Ways to Buy Down Risk

[0:01:27.0] Guy Podjarny: Hello everybody, welcome back to The Secure Developer. Today we have another DevSecCon special edition version of it and we have with us Clint Gibler.

Clint, welcome to the show.

[0:01:36.3] Clint Gibler: Hey there, thank you so much for having me.

[0:01:38.1] Guy Podjarny: Clint, before we dig in to the details, tell us a little bit about yourself, what is it that you do?

[0:01:41.4] Clint Gibler: Sure, I am a technical director and research director at NCC Group. We do security consulting and mostly penetration testing of web apps, network, mobile and so forth but we’ve also been doing more work in security engineering and security automation and DevSecOps and many of our clients are thinking about these sorts of challenges and so we’re learning the best way to help them.

[0:02:01.2] Guy Podjarny: Got it, this is like coming into an existing organization, they might have security practices so they have their security people but you coming along to help them kind of evolve their practice if you will.

[0:02:09.8] Clint Gibler: Yes, perhaps they’re a small team and they’re having to support a much bigger development team, so how do we scale our efforts or we have some security engineers and we want to know what is the best, most effective way to sort of spend our time. Yeah, sometimes we build things as well.

[0:02:24.6] Guy Podjarny: We’re going to get into the panel you moderate in a second there but what’s your current kind of best tip when you engage with the customer? I know every customer is different and all those but what’s a repetitive bit of advice that you see that kind of comes up when you do those engagements recently?

[0:02:38.1] Clint Gibler: Yeah, I think one thing that I see is people have a specific thing that they think they should do such as using a static analysis tools like SAST or dynamic analysis tool like DAST. People get very focused on how do I do X better? When I think of better question is what sort of risks are you trying to mitigate, what problems are you trying to solve and then figure out what is the best approach to do that.

Starting from a – what are the requirements and outcomes that you want and then figure out the best way to get there rather than sort of starting with the solution and then trying to make that better. Because that might not actually be the best way to do it, different vulnerability classes are better at either finding or preventing with different techniques. Yeah, I would say that.

[0:03:18.2] Guy Podjarny: Yeah, I think that’s a good bit and it actually came up in the panel yesterday, right? We talked about what was it, I think Netflix right? Sort of doing a big DAST project but not kind of starting from the end a little bit, you know? Saying, you know, what does success look like in this context though? I guess even the best fall for that trap.

[0:03:34.1] Clint Gibler: Yeah, Asta originally mentioned that at our AppSec Cali panel last year and I found that very interesting. Because yeah, they build and release all these amazing tools and it sort of inspiring or at least comforting that even the best make mistakes like this.

[0:03:48.6] Guy Podjarny: Let’s dig into the panel, you had a very esteemed panel that you moderated yesterday with just a great – I guess forward thinking security leads and generally, we’re going to post this on a YouTube link as part of this show and very much recommend everybody to watch the whole thing.

What would you say, you’re there, you’re on the panel, you ask some questions. Sometimes on panels, you get this like uniform, everybody agrees, you find yourself five echoes, phrasing the same thing at the same time, while other questions kind of unravel more debate, right? Or more, people don’t really have that solved yet. What was your impression in the panel that went yesterday, you know, how – which topics do you think were more uniform and people just need to implement them just being tricky work over there. It’s about execution versus which ones we’re just not quite sure how to go about doing them yet?

[0:04:35.9] Clint Gibler: Yeah, that’s a good question. Before the panel, we all talked and we wanted to make sure that it wasn’t just blindly agreeing on everything because ideally, having some differences shows the nuance between how different organizations, with different cultures, view the same thing. You know, everyone here is smart and has good teams, so you know, where are the cases when smart people disagree and we thought that was very interesting.

Despite our best efforts, there is a lot of agreement about many things so I think some common principles or sort of mindsets that are generally the case are working with developers to enable them, right? You have to build guard rails not be a gatekeepers because fundamentally, development teams are building products that make the business money, so our job as security people is to make them be – moving very quickly and effectively but also safely.

I think historically, there’s definitely been a culture of security saying no but I think Zane said it very well in the panel. When he said previously, where security may say yes once a year. Now you have to say, no once a year. And that is a think I see in a lot of successful organizations where the security team occasionally, you can have one or two things that you sort of hard block and say no on but those really have to be like this is the hill I’m going to die on.

This can’t be done. I would say that, investing in security automations so you know, CICD and DevOps, there are all these tools that are making development very quick and releasing very quick. If you can sort of insert security visibility points or hooks and checks throughout this existing infrastructure, that can be very powerful. Then companies have gotten a lot of value form that.

In terms of some of the differences. I think there are some differing views on how valuable security partnerships are and security champions. I think everyone agrees that it’s valuable and important to have close relationships with the development team, they need to be comfortable coming to you with like, “Hey, I don’t know if this is an issue or not, can you advise me?

But I think the question is like, how much do you prioritize that, right? Let’s say you have an AppSec team of say three to six people which is common or even large in many organizations supporting hundreds of engineers. If you have a finite amount of time and you have say, tens of teams. You can’t easily have significant one on one personal relationships with all of those teams.

For Netflix, for example, as they continue to grow and grow, if I’m recalling correctly, they thought that yes, these personal connections are helpful but it’s – like maybe there’s several key teams that you need to be really close with and you have sort of looser tabs on other things. And you sort of make up this scale with sort of security automation giving you continuous visibility into these state and relative risk of different systems.

Everyone thinks partnering with development teams is important and then crucial but sort of how much you weigh the in person one on one relationships versus say, libraries and automation. I think that was a difference based on –

[0:07:36.4] Guy Podjarny: Yeah, different people have, yeah. I guess you know, it does depend at the end of the day on sort of different cultures on what works in their surroundings and maybe the two are sort of not mutually exclusive, definitely at a certain size of organization, you’re suddenly the size of 10 other companies that might be on the panel and you may need your sort of you know, your champions and your people advocating those tools and automation. That is there or being a knowledge hub.

[0:08:00.5] Clint Gibler: I think culture is a big part of that as well.

[0:08:02.8] Guy Podjarny: Yeah, for sure. I think the emphasis on culture seemed to also resonate a lot. One point that I was a little bit surprised that didn’t come up, maybe didn’t get as much support is I think Julian was referring to sort of celebrating successes, there was no dissent about it, everybody was agreeing about it but I don’t know, should that have come up more in the conversation? Did I just kind of misread it or do you find that’s a topic that people agree on or is it sort of still hit and miss?

[0:08:29.9] Clint Gibler: Yeah, I think celebrating success is very important. Yeah, I think we didn’t talk about it too much but I think there’s pretty strong consensus there, in terms of not only building good relationships and not being sort of the security boogieman but also, you know, going to security teams and bringing a pie or cupcake and saying, “Hey, you know, we remediated all of our critical high, medium and lows, you know, everything, clean bill of health. We hit it with an SLA’s, let’s have a little mini party.”

And sort of hold people up as a success to the rest of the organization and just being very positive about – I know as Segment for example, I have a couple of friends there and they have a nice security team within the rapidly growing startup.

They have sort of an internal security leaderboard in terms of this developer just – you know, fixed a bunch of important security issues or brought to our attention some things we needed and they have actively been celebrating this. It’s had a huge cultural value in terms of just getting everyone passionate and excited about security.

I think we chatted in previous panels a little bit more about celebrating successes and that maybe our subconscious is like we’ve already –

[0:09:32.9] Guy Podjarny: It’s already old, you know?

[0:09:35.1] Clint Gibler: Yeah, I think there’s a lot of room for innovation and how you celebrate and win but yeah, celebrate security wins but yeah, I think that’s very valuable and nice to you.

[0:09:43.4] Guy Podjarny: You know, one of the topics that I kind of loved that question a little bit or on the sort of measuring security and understanding it. Definitely no crisp answers, you know? I think every panel is started by saying it’s hard. How do you see whether it’s the answers or other insights? You know, you’re very much on top of the world. What do you think is the state of the art today around measuring security and how well you’re doing, like a security team measuring, whether doing their job right or not? Were there key insights in the panel that you want to draw on?

[0:10:15.0] Clint Gibler: Yeah, I thought that was an excellent question for the panel and I appreciated that question, so thank you.

[0:10:20.1] Guy Podjarny: I try to make things hard.

[0:10:21.3] Clint Gibler: Yeah, the panel members looked to each other and — yeah. I think this is something we should be thinking more about as the security industry. I think it’s a bit of a failure on our part to be honest. If you think of other industries for example like marketing or sales or things like that, there’s very clear metrics for success, right? If you went to your marketing team and they were like, we spent 10 million dollars. Great, how useful was that in terms of getting us new leads or whatever and they were like –

Trust us, you wouldn’t have a lot of confidence in them but in security world, you gotta just trust us. Things are better now. I think it’s very important to delve more deeply into this. So, I guess one of the things we mentioned, keeping track of just simply like number of vulnerabilities and what severity can be misleading as you roll out additional detection tools or just get better from maybe you previously didn’t tag issues in Jira for example, security issues and then you start doing that and all of a sudden now your dashboard is like, “Oh, look at how many vulnerabilities we have.”

So basically getting better visibility and integrating new tools that are better and may make you look worse if you are only focusing on say one dimensional a number of bugs and how severe are they. But I think that you still want to capture things like that. So we actually had a dinner with most of the panel afterwards and someone, I think Zane, was saying was sort of calling out, you know this is like a bad metric. But then I think [inaudible] and I were arguing like, “Whoa, yes that in itself is not a useful metric however, you need that information in order to say for example we have a lot of [inaudible]. So let us systematically try to eliminate that.

So I think we all agreed that you should capture that information but how much value and importance you place on it and how it’s used sort of there is some nuance there that’s not obvious. I think in AGILE, sort of, the manifesto how do you — time to detection and time to fix I think that is a very important metric. If there is another talk and they’re talking about the OODA Loop, right? So if you can detect to that and attack that’s happening and respond faster than they can successfully exploit it then that significantly reduces the risk.

Zane and his BlackHat talk for me a year or two ago had an excellent story of where they instrumented an alert on JavaScript so that if it fired on their site it would send telemetry back to them and they actually saw that trigger, they fixed the [inaudible] before the before the bug bounty person reported it and —

[0:12:51.1] Guy Podjarny: Yeah a different type of exploit I guess on there.

[0:12:53.0] Clint Gibler: Which I thought was very funny. So yeah I think ultimately come down to risk, some innovation in the space that’s been happening is companies are working on like automatically like programmatically inferring risk of various services, based on various factors you can infer. So for example, is this internet facing? Is I critical to the business, if this is an AWS instance like how many of them are there like this service is represented in one or two live nodes versus thousands?

Is it touching sensitive [inaudible], what classes? So I think there is risk models you can generated programmatically and you also can hook that into historical vulnerability data to say, “Oh this is both critical to the business and tends to have issues.” So maybe we focus more there or maybe there is live issues.

[0:13:37.2] Guy Podjarny: Yeah, it is sort of all of the combination of all the signals that they have available. First of all, I found it very insightful kind of your comments on — again it’s simple but the fact that you are incentivized. If you are incentivized to just keep the number of vulnerabilities that are currently open at the lowest number then you are basically incentivized not to find any more. So it is a little bit of the thing to cope with, you know as you do want to increase your understanding.

But it is also very understood how your KPI’s might look much terrible there. I do find one of the comments that I think Asta made was how it didn’t really have like a sharp answer off the cuff on how they measure security poster but what they do measure well is just the delivery on the projects that they have chosen to do.

So I think maybe that is in mind with what you are suggesting over here, which is okay, like if we are going to reduce — we took on a project, we decided maybe a little bit more loosely that cross inscripting is an important risk to tackle. But then there are concrete measures about how do we did reduce that level. She was talking about delivering on the road map and the likes. So I found that kind of a good model as well as the comment I think from Zane. We kind of keep quoting him over here on just starting from a mindset of saying how will the attackers succeed.

If I were an attacker then how likely is the attacker to succeed in penetrating and just starting from that perspective versus from what can I measure and then try to break it back down into what you can measure.

[0:15:01.3] Clint Gibler: Yeah I think ultimately it is about risk like what are the riskiest, most scary things or organization and then how do we buy down that risk ideally in as sort of sustainable and scalable way as possible.

[0:15:12.4] Guy Podjarny: Yeah so maybe a couple more questions. One is that you were really, really adapted that the panel was mentioning that this person here that talk and this person there that had that talk on relevant topics that came up. You know how do you keep up, how do you advise people stay up to date on these types of interesting evolutions and DevSecOps or otherwise sort of security?

[0:15:31.3] Clint Gibler: Thank you, yeah. So the honest answer is I have spent hundreds of hours watching talks and reading blog posts and just chatting with friends over drinks. Yeah, I think there is too much out there to really be abreast of everything. So I think it is helpful to come up with like a handful of topics that you – so if for example you want to be very familiar with one area just like sort of acknowledging to yourself that you are not going to be an expert in everything and then just really focusing on that.

Like for me for example, I am really interested in automated bug finding and DevSecOps and sort of modern AppSec like practices. So different conferences tend to have more content in certain specific areas. So for me the various OWASP ones like AppSec Cali and EU and Global and then obviously there is like a number of other ones BSidesSF and others who are good as well. I think if you find individual people who give a talk that you think is very good, looking for other work that they have done.

Like actually before this week I watched like five talks from Julian, because I was like, “Oh this guy knows what’s up.” And I would say I think it’s challenging. One challenge, so I have a bit of an academic background and I think that one thing academia does better than industry is being aware of related work. I think there is a lot of redoing work or at least if you were to draw a Venn diagram it is like 80% overlap between a lot of industry talks, which is fine.

I think there is unique value there but I think less time an attention is spent on being aware of what other people are doing. So I am actively trying to do that but again, that takes time and it is hard so.

[0:17:04.8] Guy Podjarny: That is a great practice though if people do it. If they had like a references slide at the end of every talk you know that would be –

[0:17:09.1] Clint Gibler: Yeah and some people do and I try to. I think that is awesome and yeah partly to sort of help people with this challenge. I have started an application security and DevSecOps newsletter. Just super informal, just like, “Hey, here is a couple of talks I watched. I think these are the best in the space and here’s maybe like a couple of bullet points or one or two page summary of it so you can quickly get the main points, see the key slides and figures and ideally it would save some time and help you be better and more efficient.”

[0:17:36.6] Guy Podjarny: Yeah, it sounds super useful. Where can you find that newsletter?

[0:17:39.8] Clint Gibler: Yeah, so it is at programanalysis.is is the name. It is the Iceland registrar. So programanalys.is/newsletter yeah and it’s like once a month or once every other week or so just hey, here is the most useful links I have come across, here is some talks I really like for example like Arkady Tetelman who was at Airbnb, give a really good talk on data driven bug bounty where he showed how Airbnb used that information to basically invest in the right security engineering efforts to sort of really buy down risk is one thing that I wrote about but yeah so —

[0:18:15.7] Guy Podjarny: That’s cool, now I want to watch that talk on there. I love it, you know thanks for writing the newsletters. I feel like a lot like my content conception comes from newsletters you know and just trusting other people’s opinions, to sort of filter the internet to help you find the right one. So you know I very much advise people to go check it out and thank you for spending the time doing that curation, I think is always a good deed for sort of evangelizing knowledge.

So Clint, before I let you go here, we have been going long as I am known to do a little bit, you moderated the panel that’s great to get an opportunity to answer some of the questions yourself as well but if you are on the panel, if you had an insight, an opportunity to share just a thought or choose a question, what is an insight or a perspective that you would have liked to share?

[0:18:58.1] Clint Gibler: Yeah that is a good question. So one thing that you have seen at NCC Group or working with clients, a couple of times clients have said, “Hey we are trying to use SAST or we’re trying to use DAST help us make it better.” And I think that there’s definitely room for improving and tuning all of these tools in terms of reducing false positives, increasing signal and things like that but I think that it is useful to think about just to take a step back from the very tactical view.

And think you know, what are we trying to do here, what risks are we trying to either buy down or mitigate or reduce or ideally even remove? Because when you think I’ve like “Okay, how do I make this static analysis tool better?” Generally you can but that might not be the best technique for finding a specific vulnerability class, for example. So think companies that are very in the weeds thinking like, “Okay, how do I use this tool better?”

I would encourage them to at least spend some time stepping back and thinking what are all the different issues that our company is facing, you know if you were to say graph — here is all the vulnerabilities we have over the past one to two years. Group them into vulnerability class you know like XXS, SSRF, CSRF, things like that and then saying, “Okay what are we seeing, what is the relative impact and then what is the best approach to solve that issue?”

And I think the primary tools in your tool belt are probably building a secure wrapper library, you know light weight static analysis or lightweight dynamic analysis and different vulnerability classes are better or worse suited to based on your tech stack as well as this specific vulnerability. So yeah, I would say just take a step back like rather than saying how do we make this tool or this process better, think is that even the right thing we should be doing?

Because an amazing world class hammer is probably not the best tool to put in a screw, right? So that would be advice.

[0:20:47.2] Guy Podjarny: That is a really, really good advice you know I guess we get an opportunity over here to share it hopefully you get more stages. Clint this has been a pleasure, thanks for coming on the show.

[0:20:55.0] Clint Gibler: Yeah, thank you so much for having me.

[0:20:56.4] Guy Podjarny: And thanks everybody for tuning in and I hope you join us for the next one.

[END OF INTERVIEW]