[0:01:22.6] Guy Podjarny: Welcome back everybody, thanks for joining us again to The Secure Developer. Today we have another DevSecCon, special edition version of The Secure Developer and we have with us Doug DePerry from Datadog, thanks for being on the show, Doug.
[0:01:34.7] Douglas DePerry: Absolutely, happy to be here.
[0:01:35.8] Guy Podjarny: Before we dig in, tell us a little bit about what you do, you know –
[0:01:39.5] Douglas DePerry: Yeah, sure. I work in security at Datadog, I’ve worn many hats in my three years there. I was originally hired to be the director of product security but you know, for the first year or so that I was there, I was doing mostly infrastructure security stuff which I kind of learned on the fly.
And then, you know, after that time, I was concentrating on product security and building out that team and building out that practice, we tried to focus on a few different areas within product security, we had a team of developers, you know, working to create applications and services and secure libraries and things like that.
We had a team of breakers who were doing code reviews, penta, it’s almost like internal consultants and then one of the other things that I was focusing on towards over the last six to 12 months was customer safety and that’s almost like an incident response throughout detection team but at the application layer.
[0:02:28.3] Guy Podjarny: All of those fall under the product security mantle on it or is there things you did beyond –
[0:02:32.7] Douglas DePerry: Well they did, recently we’ve gone through some organizational changes as we have you know, with security as the company’s continued to grow. Datadog has been, we’ll get into too many buzz words but you know, hyper growth and as the population of the engineering team has exploded and so as we do that, we’ve consolidated some of our things and so I focused lately on defensive efforts, right?
What we did was we took the product security team and the infrastructure security team which were doing very similar things that I just, you know, different ends of the stack and combined them into a security engineering team and so we now have several teams of developers focusing on platforms and services that’s consumed by the security team as well as you know, the rest of the engineering org.
We have several teams of breakers now that are focused on infrastructure, scanning and penetration testing as well as code reviews and design reviews and things like that. So my piece of this new pie now is active defense team which is you know, concentrating on threat detection, alerting, things of that nature as well as customer safety which is you know, mainly we’re focused on preventing account take overs and helping to provide secure defaults for customers within the product.
[0:03:38.6] Guy Podjarny: Yeah, the risk of going down a little bit of a rat hole, just like product security and infrastructure security. I guess there are now security engineering but is an infrastructure like in a SaaS company like Datadog product, what was the logic behind separating or merging them?
[0:03:52.1] Douglas DePerry: Yeah, sure. I think it was just you know, if you look at any number of companies with large security teams, especially modern software and technology companies, there’s not necessarily any one good way, best way to do it. I think just that was what the CISO had decided when he did it, you know, he’s like hey, I know this guy Doug, he does application security, been doing it for a while, It’s on him, coming and be director of product security and like Noah is, he’s really good infrastructure security, going to have him build up that team and I think it was just kind of the model we chose.
[0:04:21.6] Guy Podjarny: Playing to people’s strengths versus kind of some logical separation from –
[0:04:24.9] Douglas DePerry: Yeah, exactly. There wasn’t –there wasn’t, you know, it’s not like there was a ton of overlap, even though we were each kind of separating our teams into these specific things of building and breaking and you know, threat detection, that sort of thing. Trying different way to –
[0:04:41.5] Guy Podjarny: I guess iteration is the key.
[0:04:42.1] Douglas DePerry: Yeah, I’ll let you know in a year or two.
[0:04:43.3] Guy Podjarny: Indeed. How it all worked out. You’re a part of a great panel yesterday, you know, there were a lot of great questions and conversations in the panel which you know, I guess I’ll just encourage people to go watch the YouTube video to sort of see the whole things. But digging into a few of the topics. You know, one of the key themes that did repeatedly come up when in this panel of like modern security organizations, was around kind of developer buy-in, right? Around sort of engaging with developers.
Can you tell us a little bit, how do you see and how do you maybe enact engagements with the engineering teams from the security team or the security engineering team to help build secure software?
[0:05:15.8] Douglas DePerry: Yeah, sure. We’ve tried a number of different approaches, you know, over the years, you know, like I said, I was saying before, you know. I was a big – I’ve known Zane Lackey for a long time and I was a big fan of a lot of the work that he did and spoke about from his time at Etsy and so I do believe that it’s a lot of like hearts and minds sort of approach where you know, the dev teams got top understand what your motives are and because traditionally, security was seen as a blocker, right?
Sitting there on the bridge with the staff, you shall not pass and as we try to move towards a faster and better way of developing in software, you know, that just doesn’t work, you know? When you’re pushing production dozens of times a day, there’s no way you’re going to review all of those things.
Even if you try, two people would rat around you, you know? Culture was very important part of Datadog, you know, in the engineering culture and how we do things and how we treat each other and work with each other and so we started with just getting to know teams, getting to know their stacks, proving that we knew what we were talking about.
We embeDouglas DePerryed –
[0:06:12.5] Guy Podjarny: Just kind of earning the respect of the engineers?
[0:06:14.9] Douglas DePerry: Exactly. We worked with embeDouglas DePerrying security engineers with development teams and you know, it was a little bit of here, let me help you do that securely and a little bit of just like raising awareness, right? I think raising awareness is a big thing because no one wants to write insecure code and no one wants the company to get hacked but you know, they just don’t know and/or it’s not like necessarily their initial focus.
Their initial focus is to ship product, focus is to ship product features, right? Just having someone sit with the team and just having them walk by the security guy every morning as they go to their desk, get your ability up.
[0:06:49.4] Guy Podjarny: Security person. They’re like physically collocated just sort of sitting by within the dev team and then is it like – the security engineers would actually take on a task on the sprint towards – how did that work?
[0:07:00.2] Douglas DePerry: Sometimes you know, depending on the team, depending on the general pieces of the application that they were – that the dev team happened to be working on at the time. Some of it was there, let me help fix some of these little hanging fruit or these security bugs, you know? I can fix this, I can fix that and help kind of teach and learn along the way as well and just kind of help to push the mindset that securities everyone’s responsibility.
We’ve also had really good luck with doing that on a more temporary basis. Two, three weeks, a month or two or a quarter, embed someone with a certain team because you know, resources are finite, right? You can’t – I can’t embed an engineer with every single dev team but we had a lot of good come out of that as well and just like information sharing from both sides, you know? We can learn about how that particular team works better, we can provide them some value by fixing some bugs or just having someone that’s sitting there that can just bounce questions and bounce ideas off of.
Whether they’re building something new or just maintaining existing code.
[0:07:56.4] Guy Podjarny: No, fascinating, it all comes down to people and kind of yeah, their hearts and minds kind of in. Do you cement that practice today like do you – you talk about a bunch of these things in like past tense, is this still the current methodologies, is there anything more different, sort of you know, like evolution of that dev con too?
[0:08:12.3] Douglas DePerry: I mean, for the most part, we still have engineers that are embeDouglas DePerryed with certain teams. We haven’t done the temporary, you know, we just call it TDY, temporary duty where you know, they go with it, individual teams for a short period of time.
[0:08:23.3] Guy Podjarny: Per your shift, yeah?
[0:08:24.3] Douglas DePerry: Yeah. Honestly, we just kind of got too busy, it was a good thing in a way, right? All this raising awareness to trying to understand what’s happening in the organization and just – because again, I’m talking about starting from scratch here essentially. Having you know, knowing that – trying to put forth the idea that security team is here, we’re here to help you, we’re here to make your job, your life better, easier, hopefully faster, you know? Was kind of the original goal there.
Once the awareness started to grow, we started getting a lot more inbound requests for code reviews, design reviews, architectural review, that sort of thing. Yet, we were hiring fast but the engineering team was hiring even faster and they were even larger than us to begin with and so it just – you know, we weren’t drowning but we were very busy, we didn’t really have a spare resource to be honest to go kind of – do such thing with the team for a while.
[0:09:12.4] Guy Podjarny: I think though, I love this is a kind of a model or sort of something – a bit of advice because it actually talks about the situation than most organizations are in which is they don’t have it, you know? Set up already and I guess what you’re describing is a culture shift so once the culture has been established, then maybe you don’t need kind of those artificial kind of mechanisms anymore. You can sample them and kind of cement it.
But mostly it’s around that inbound. Sounds awesome you know? It sounds like the right model. This was also – you know, I would like, kind of in line with a lot of the topics in the panel. Another kind of big theme I guess related because as we talk about developer adoption was tooling, what’s your approach to tools, technology tools, security tool that you sort of pull in? How do you judge them and then what’s important about embeDouglas DePerrying them about being successful with them?
[0:09:55.0] Douglas DePerry: Yeah, we try to approach the build versus buy argument as unbiased way as possible you know and it’s hard because you never really know sometimes how deep that rabbit hole goes until you’re halfway down and so what starts out as, “Ah yeah, you know you I am going to take a couple of weeks to throw this thing together” turns into months and then you don’t want to stop.
[0:10:15.8] Guy Podjarny: It never as easy as you think, yeah.
[0:10:16.8] Douglas DePerry: Yeah, it turns out writing software is difficult.
[0:10:18.6] Guy Podjarny: Yeah really.
[0:10:19.4] Douglas DePerry: But no, I did really embrace the fact the security needs to write more code. We need to automate more that is how you are going to get this for multiplier effect and be able to keep up with the speed at which code is being deployed and so it depends what it is, right? We looked to putting a lot of effort into writing tools that would interact with CI, right? So you know we use GitHub we are looking to push a lot of our tooling into running per commit basis or what have you.
And so just because we didn’t want to always be responsible for every single alert either, right? Security is everyone’s responsibility, we want to provide you with in the developer, we want to provide the development teams with actionable messages, tasks based on the code that they wrote like in context, right? If you wait 24 hours after a certain commit has been pushed then you don’t like – Developers move on they are doing something else now.
They don’t necessarily remember, things are moving very, very quickly. So if you can do that as quickly as possible, get a very tight feedback loop and then like this, we are not trying to dump everything on an individual developer but if it is a dependency, vulnerability, you know the answer is just update the dependency. If you can’t for some reason reach out to us, here is how you contact us and we will be happy to help you and then the hope is that only happens you know once in a blue moon because you know for the most part our construction is just going to update that.
[0:11:46.7] Guy Podjarny: Yeah just update and go on yeah and then so you do those and it’s all about the developer self-service or self-sufficiency the right time for these types of messages. What are common missteps that use the security tools and what’s a pattern of a security assist like, “Oh not again” you know you have another tool that does X and that’s just a show stopper for us.
[0:12:09.1] Douglas DePerry: I don’t know that we’ve run into too much of that to be honest. You know static analysis is the easy dart board here but you know at the time that we are looking into a static analysis solution it was – a lot of those products are kind of centered into a more legacy development model and listen there is some value in that. Don’t get wrong like there are certainly a large amount of compliance frameworks that require X amount of skins and it’s whether or not that service is good or not is beyond the scope of this conversation but –
[0:12:42.9] Guy Podjarny: You still need to be compliant.
[0:12:44.4] Douglas DePerry: Yeah, exactly and so that is why we had to get that thing like look, we paid for this thing. Let’s see if we can work it into the workflow that I was just describing and so that’s what we did and so we wrote a tool. We unimaginatively named the MiDouglas DePerryleware and it was essentially like a framework that we could create aDouglas DePerryitional plugins for one. So we had our static analysis plugin and then we had our dependency vulnerability plugin.
And the idea is that we can kind of grow this plugins, we have the framework that can run in the handles, the comments on the PRs and what point they should be running depending on how long that plugin meet, aesthetic analysis plugin could take some time to roll even if it is just a small snippet of code. So that was our approach and then there was two folks on my team that actually spoke about their work at last Con and Deep Sec in 2018 and she tried to share that knowledge ago.
[0:13:32.1] Guy Podjarny: Yeah I like how that takes advantage of the different makeup of the team itself as well right? Because if it is people that can write code then they can take the solutions and adapt them to the way that you are working versus being a bit more captive to instead of having the tools be perfectly matched to your surrounding. So you know each one of these is like in our honest conversation doing something right but just teasing out.
And I do think that people should watch the full panel, not all of this was discussed over there. Maybe one last bit that did get raised there was one on measuring. I am measuring success on it. So you know talk about tighter feedback loops but even not just the tighter but the bigger feedback loop, how do you see that? You know you talked a little bit about the ability to forecast to something maybe like a focus area for the industry for yourself but how do you know if you are doing your job well? How do you measure yourself, your team?
[0:14:18.5] Douglas DePerry: Yeah every day that goes by where there is no report on incident, I find them now.
[0:14:22.4] Guy Podjarny: Well it is a measure, hopefully it is not the only one who knows that.
[0:14:25.9] Douglas DePerry: But I mean it is difficult and I think the fact that and I believe Zane said this on the panel towards the end there, you know the fact that you know that question is asked and everyone is like, “Hmm yeah, good question, you know tough problem” you know that’s not good for the industry. I think the closest thing that I have seen as far as trying to do better at that is what Ryan McIan has done with forecasting. So he had published a ton of blog post about it.
We have spoken with him at one point maybe a couple of years back when we were starting to – when he was really pushing this stuff and it was a little bit new – I mean not new for him but new to kind of the industry and so I really like that in theory and for whatever reason we got busy. We moved on, we prioritized, yeah exactly things happened but that is something that I really like to focus on coming up because you know front models you know like, “Well what I want to know is how likely is something to happen if possible” right?
So I think threat models give you a better idea of your tech surface and what you can do to prevent bad things from happening but you know the likelihood it is very important because your resources are limited. Resource are finite. We all know you need to focus on the thing that will get you first but how do you really decide that and I think the more that you can because it is difficult to measure certain things right now it just makes that very difficult or makes that error prone and so I think forecasting if done correctly can really help with that.
[0:15:54.6] Guy Podjarny: Yep, no I think fundamentally as imperfect as it is just the muscle that we have to build.
[0:16:00.0] Douglas DePerry: Yeah exactly you just have to try –
[0:16:00.8] Guy Podjarny: You are forecasting one way or another you’re forecasting. You are deciding what to do next quarter. So you are guessing at something better and you might as well make a science out of it and then decide to make it more.
[0:16:09.4] Douglas DePerry: Exactly and you can come up with a priority list every day but it is going to change by 11 AM and then it is going to change again by 3 PM and then it is going to change four times overnight and it is difficult to keep up and so I think it is just like the world we live in, right? It’s just technology is moving very quickly, security is moving very quickly and so you know, if you can isolate certain things to focus on I think it will provide you with a much better bang for your buck.
[0:16:34.8] Guy Podjarny: Cool and so before I let you go here, we are going long already maybe one question I didn’t get to ask on the panel, which is once you have been through this journey, what’s the one key evolutionary key learning that you can point to and I think that surprised you that you didn’t expect and you think is actually significant in how you operate today?
[0:16:55.3] Douglas DePerry: You know the first thing that comes to mind to be honest is the people aspect of security. You know like I knew that – well obviously I think you are working with people, you know? And people are a factor in this but once or twice in this position I did get all really did surprise me how much more negotiation and compromise and communication, how important those things really were and again it feels trite to say it but that was really something that really –
[0:17:21.4] Guy Podjarny: What you’re feeling viscerally.
[0:17:22.5] Douglas DePerry: Yeah exactly that really surprised me and again for the most part our engineering at that extremely helped me. It was open and honest, everyone is working hard and it’s great but they have their areas of focus and so we have our areas of focus and communications. Sometimes people just want to know what’s going on, right? And so I think I definitely learned a lot in that aspect in that side of things. The technical problems themselves aren’t really that difficult.
But it culture, it’s business processes, it customer feedback there is all of these other variables at play here that have nothing to do with –
[0:17:58.4] Guy Podjarny: Yeah with the tech with the underlying tech or risk. That’s great insight. Doug, thanks for coming on the show.
[0:18:04.9] Douglas DePerry: Absolutely it was great.
[0:18:05.9] Guy Podjarny: And thanks everybody for tuning in and I hope you join us for the next one.
[END OF INTERVIEW]
