[00:01:29] Simon Maple: Hello, everyone, and welcome to the final episode of The Secure Developer in 2020. Welcome back to regular listeners and thanks for tuning in. Today, of course, I’m not Guy Podjarny. Of course I’m not! My name is Simon Maple. I’m going to be your host for a very special episode this time around, where we take a look back at 2020. This time, me being the host, joining me as our amazing guest is the man himself, Guy Podjarny. Guy, how are you?
[00:01:56] Guy Podjarny: I’m doing well. Thanks. I’m getting used to the change of roles here.
[00:01:59] Simon Maple: I know. Did you like the intro? Did I get that right?
[00:02:03] Guy Podjarny: Spot on, as always.
[00:02:05] Simon Maple: Excellent, excellent. So, Guy, why don’t you tell us a little bit about yourself because you don’t normally introduce yourself too much in the podcast?
[00:02:12] Guy Podjarny: Yeah. I think, I wonder how many listeners actually know this. I am the Founder and President here at Snyk, and I’ll spare you the whole history element of it. But I started this podcast about four years ago, and the rest is history.
[00:02:27] Simon Maple: Awesome. Well, this is the second time we’ve done this together, in fact, where we’ve taken a look back at the previous year. Let me give you some stats. It’s always interesting to hear the stats at the end of the year, and this year we actually really did ramp up the number of episodes and the number of people that we had on the podcast. So, in 2020 overall, which was a little bit of a rollercoaster ride for many reasons, of course, as we all know, but this year we did 38 episodes, which is almost double, in fact, our previous year in 2019. It’s much, much more work to actually find and get all the speakers as well to do all the recordings, so amazing number of episodes. In fact, 42 which is double the number of guests that we’ve had on the podcast as well, so really, really great to see the podcast growing so fast.
Let’s jump straight back into looking back at 2020. In terms of some of the recurring messages that you heard about and you spoke with many of your 42 guests around, what were some of the themes and recurring discussions that you had across those 38 episodes?
[00:03:33] Guy Podjarny: Well, there were many, many, many of those. It’s still kind of a little mind-blowing that we’ve done 38 episodes. So, there were a lot of recurring themes. The one that really jumps to mind at the top is security champions. That went from maybe a casual mention in the 2019 episodes but maybe started building momentum to being very, very kind of common mention in the 2020 episodes. I think it was about eight different guests described their security champions program, and we also did one of our sort of experimental kind of mix and match episode, meshing them together.
I think it’s been very clear that security champions is a good program, a good means that people appreciate, and how to scale security and really kind of help amplify the team, and the practices are forming. You see some recurrence. You see some methodologies that get repeated from guest to guest, but it’s still quite a flex. Security ChampionS is definitely, definitely a big deal, and I liked some creativity over there. Jeff at Medallia talked about how they give Stanford credentials to the security champions that go through the training of the program. At Twilio, I think Yash was talking about how they cycle through the security champions, so more people get that type of role. They do also have a progression path.
There was also a lot of commentary about the separation between security communities that just allow developers that care about security to congregate and share practices from actual security champions program, where they actually are expected to spend a certain percentage of their time on security and have some formal interactions. Security champions is sort of at the top.
[00:05:12] Simon Maple: This is always very interesting. I’m hearing more and more about how our customers at Snyk, as well as people in general, are using more and more security champions in their development organizations. Is there any parallels that can be drawn against other different testing environments? Do you also see [inaudible 00:05:28] champions or performance champions or any other champions in other disciplines that you see existing in the development organizations?
[00:05:37] Guy Podjarny: I think the term ‘champions’ has somehow – It might be security-specific. I haven’t heard it in other contexts, but I think what you do hear about is you hear it about guilds or other terms like that. You talk about a front-end guild or the likes. In fact, I believe Mike [Shema] at Square talked about how it’s called a security guild for them. So, you do have this notion of communities of practice or of sets of people that have a certain competency, and they share it. I think what’s a little bit different in security is that security augments that with an expert body that sits off to the side in the actual dedicated security organization, and you don’t see quite as much of that going on. But it’s all shaping up, so maybe some of this will expand to other fields and definitely security has more to learn.
I think if security champions kind of at the top, another key theme that I’ve heard was ‘Celebrating Success’. This one almost had a wave. I used to hear a lot more about it at the beginning in the early episodes. Then somehow in 2019, it came up. There were definite mentions of celebrating success, but maybe it wasn’t quite as strong a theme. It came up in kind of full force in 2020. To an extent, it came in the context of COVID as well when people are talking about how do they keep people encouraged. There’s a lot of favorites in this theme. I think, I really liked Mike Hanley from Duo. He talked about them actually building like a little neighborhood-friendly bot there that if someone was first to patch or first to do something that was security conscious, it would tell them good, and they had all sorts of other kind of recognition elements.
I had an episode that I was really happy about with Kyle Randall from Optimizely who was was the very first guest on the podcast. Then he came on four years later to talk about it, and he talked about how those security hero t-shirts that they gave to encourage developers back then were a hit but, over the years, has also kind of systemized in scales and realized that, sometimes, celebrating success is as simple as recognition in some public forums.
Probably the most sort of structured and large-scale variant of this sort of recognition and celebrating success came from Roland Cloutier who was the Chief Security Officer at ADP. He since moved on to be the CISO of TikTok. On one hand, they have these CISO awards that anybody can give anybody these types of thank you awards, which are great, but also introduce all sorts of kind of gamification and encouragements that are, again, on that recognition. So those are three. It actually came up a lot more than those three, but I think celebrating success and recognizing developers that sort of do well about security has really kind of come up as a strong theme as well.
[00:08:07] Simon Maple: That’s a really superb thing as well. I think the number of pressures on a developer for the amount of time they spend on different things and very heavily with developers particularly when back in the day when I was actually writing code. Can you believe that, Guy? Back in those days –
[00:08:20] Guy Podjarny: The dangerous, dangerous days.
[00:08:21] Simon Maple: It was. It was very dangerous for all. But, yeah, the pressures on a developer to actually just deliver on time irrespective of quality half the time, right? It’s extremely important to get those features out. So, to recognize things which aren’t necessarily directly focused on the functional requirements almost of a specific feature is extremely, extremely important. I think it’s excellent that that’s –
[00:08:42] Guy Podjarny: Yeah, absolutely. It can even be perceived the other way around if you don’t recognize that you’re almost penalized for spending time on security versus on the things that you all recognize. So, you really have to balance it out if you are to get your developer team to engage and actually play ball as part of building security. Those are probably the two primaries on the sort of the dev side. I think there have been a couple of interesting ones on maybe more the sort of the ops and security side, the other piece, so definitely a lot more conversations about cloud native security this year.
We have quite a few mentions about DevOps as a model for security, which I love. I’ve always kind of loved DevOps analogies. I think security is kind of walking in the footsteps of DevOps in many, many different ways here, trying to embed security into dev. Gene Kim told a whole bunch of stories about different companies that are kind of walking through it and embracing it. I loved his mention of flow from DevOps and applying it to security, talking about allowing developers to stay in the flow and how that helps organizations embrace DevOps and do that successfully. If you think about that and all the operable things, can you make them a natural part of the developer flow? Then how can you apply that to security? That was a good DevOps analogy.
Security chaos engineering. I love security chaos engineering. It just sort of feels like such a promising methodology, chaos engineering as a whole. We had both Kelly Shortridge from Capsule8 and Aaron Rinehart who actually kind of built chaos engineering solutions. Both of them talked and gave some great tips about how do you embrace these attacks. So, you can actually test if your system is resilient to attacks, whether ops or security. That’s why the term attacks is actually used for chaos engineering, even regardless of security, so using that. That was great. We also had kind of a little bit more on the sort of the softer side.
I guess that Gene Kim is also on a softer side or a process side with 10x, where really 10x is this sort of modern banking surrounding, and Neil Drennan there is the CTO. Really basically embedded security, the only methodology in which security is tracked from a Dev Team perspective is in the same agile meetings, same place that they measure all these other aspects of their quality. A lot of that DevOps practices and how do they embrace them and how do you kind of have security come along.
I guess next to that, on the cloud native front, there was maybe some specific focus on cloud security, and I think that was probably one of the messier topics this year in terms of people not really having an answer to it. If last year it was containers, this year I think it’s more cloud security where there was definitely a lot more mention of cloud security. How do you kind of rein in the sprawling assets of your sort of cloud setups? But also, I think as companies embrace infrastructure as a code more and developers engage more with it, it really becomes hard to define or necessary as well to define the developer’s role in this cloud security setup.
We’ve seen some examples of this. So we had Erkang Zheng who’s from LifeOmic, and they built this tool called JupiterOne which is in the space. They’ve since spun that out, so that’s actually its own company right now, and they talked about how the CMDB, the config management database, the sort of this repository of assets that you have, how is that working in the cloud. Surrounding developers need to engage it. We’ve seen Spotify open source backstage, which is a system that also tries to sort of deal with that type of management of assets.
We had Teri Radichel talk about understanding that cloud security starts with cloud architecture, and that in turn is in the hands of the developers. This whole cloud native theme, whether as a helper with DevOps methodologies applied to security, but also kind of the messiness a little bit, especially when it comes to cloud security and infrastructure’s code was definitely a topic this year.
[00:12:15] Simon Maple: I don’t want to almost preempt some of the future questions. But in terms of the cloud security obviously, you mentioned this is something which is a little bit messy this year. Obviously, cloud’s been around for so long now. You’re heavily talking about the infrastructure as code part of that, right? Is this something that you see being much more solidified going forward in 2021?
[00:12:32] Guy Podjarny: I think so. The bigger picture is really I mentioned it occasionally on the episodes, but the world of cloud has kind of messed up the lines between IT and development. A lot of responsibilities that used to be in the realm of some central IT organizations have now become capabilities that are provided as a service, whether it’s part of your infrastructure as a service or SaaS capabilities. Those are now application services. They’re controlled. They’re configured. They’re run by developers. But it’s a new realm of responsibility for developers. It’s not something that they’re familiar with and it’s also new to the people on wherever it was that those assets were managed before.
I think, last year we talked a lot about sort of VMs and containers and how containers are much more an evolution of the app than they are necessarily an evolution of the VM. But they feel like an evolution of VM. It feels like IT used to manage the operating system and patching it and all of that, and they still do for VMs. In containers, it’s most natural to kind of just move on and think the same way and just apply your VM security practices to your containers. It takes a moment to realize that it doesn’t work that way, that containers are defined in a Dockerfile. It sits in a repo, and they’re patched by running a build, and that you have to sort of embrace more of a developer approach to it. I think, this year I haven’t really heard any disagreement on that front, so I think that’s accepted.
Cloud security and infrastructure as code, I think it’s not just the infrastructure as code piece. It’s the fact that developers need to have a role here, right? When you start embracing Terraform or Kubernetes, configurations or helm charts, what you’re doing is you are moving decisions from about cloud assets, about knowing what’s there, into the realm of development. But it doesn’t fully, fully move the development. It’s just partially moved to development, so the industry kind of needs to work through it.
Yeah, I think there’s going to be a lot of those conversations in 2021. I don’t know that it’s a one-year resolution. It might also be the theme in 2022 and it has at least two pieces to it, which is the sort of the ‘Infrastructure as Code’ piece and how does that relate to manual changes. I think that’s hopefully the key theme in 2021. How do we secure these decisions as part of infrastructure as code? Then it also has this whole CMDB of who’s managing the assets, what’s where? I think that’s a slightly more nascent stages type problem, so maybe that’s the 2022 piece.
[00:14:56] Simon Maple: We always talk about how the development teams change and how the development role has to adapt to accept all these different things, security included. But what about the security team? Have you seen as much change in the way the security team works over the last year?
[00:14:56] Guy Podjarny: Yeah, absolutely. There’s been a ton of conversations about those this year, and I’ve seen it through Snyk customers, through sort of reading in the industry, and definitely through the guests on the podcasts. Part of it is what I was alluding to now, which is when something transitions, both parties needs to be there. When new responsibility transitions to dev, it doesn’t fully transition, so security needs to be there. I think what’s been acknowledged this year is that there was a certain critical mass of movement and that security teams don’t need to just tactically deal with container security or cloud security or whatever at dev, but they actually need to restructure.
Again, this is something that there were mentions of it in 2019. Justin Somaini from Box, he’s now at Unity. He discussed this probably most succinctly in 2019. But in 2020, it came up a lot. We had Adrian Ludwig from Atlassian talked about their product security team. In another conversation we had at Nikon, he actually mentioned how they have a product security team that includes both the cloud pieces of the dev team, as well as the application security, anything around product security. But when they advertise for roles, they advertise for application security teams because otherwise they get product managers applying for the roles, which goes to show that they think that this is the right terminology and various others.
I think Yash at Twilio also. They call it product security. Others embrace that term, but the industry isn’t used to it. But that’s sort of one aspect of it which was the scope of how do the teams structure. Application security is historically about vulnerabilities in your custom code, but they’re also the team that is aligned to development. How does that work? How do they interact and who’s responsible for container security? Who’s responsible for this cloud security? I think that’s a transition, but I’m seeing more and more of these product security teams, which I like, which takes a different lens on it. It’s not a tech separation. It’s the statement of who’s securing the application and who is securing the infrastructure? I think that’s a healthier split. I think Brendan in Toast was stuck to the application security side, but he I think defined – their definitions I think are pretty good if you want to check them out.
That’s the org structure. Then the other theme that really kind of arrived is about skills and re-skilling. My favorite episode around re-skilling was Roland Cloutier who I mentioned before from ex-ADP. In a variety of conversations including that conversation I mentioned from Justin Somaini, we talked about sort of turnover. A lot of people talk about how they hire into their teams today people with coding skills and with more of a developer empathy. A lot of them. Sacha Faust from Lyft and Amazon, a lot of mentions about how you need some amount of empathy to developers and some amount of coding skills to deal with this new surrounding where you need to be building tools to help secure software and you need to be working through developers. You’re not doing the work yourself.
But then what I loved about Roland’s episode is he talked about re-skilling, so not just about who you hire, but they have a whole program about re-skilling the team. What I liked about it is the forward-looking aspect of it. So, if you have a large security team as they did in ADP which I think was in the thousands, it’s a very large team, then you have to think ahead and say, “Well, if I don’t want to just turn over this team. Many of them are amazing kind of talented individuals,” you want to start re-skilling them now in batches as you transform your enterprise. Even if only a portion of your tech is maybe now in cloud and maybe in this new surrounding, still you need to start thinking ahead, putting the train, and putting the education alongside these new hirings. I really, really liked that setup.
All of those are kind of this realization that DevSecOps is about more than just the tech. It’s about the actual org structure and the changes. I guess on a positive note, if you think about what this has done to ops, it’s been all goodness. You basically had sysadmins turn into SREs, and their salaries doubled, and they’re definitely very impactful to the organization. Their impact scaled up. So, something along those lines is hopefully happening to security.
[00:19:04] Simon Maple: Yeah. Well, sounds like some really excellent changes coming in, which is great to see, and I think a lot of what you mentioned here are, of course, planned changes that we want to make. Now, of course, let’s address the elephant in the room as it were and talk about what’s happened that wasn’t planned over this year.
When we did the 2019 summary, we were, of course, in the London office and we did this podcast across the desk. We are right now over Zoom, and COVID has had an impact on the way – we’ve all had to make changes in the way we work. With DevSecOps and security in general being so reliant on the relationships across teams, between people, individuals, how has the COVID impact and the changes that’s made into our workplace? How’s that affected the way we do work together and integrate as one team?
[00:19:50] Guy Podjarny: Yeah. COVID was super interesting from a security lens perspective, beyond the shock that everybody went through including security people, especially organizations that weren’t as used to working remotely. For us, I think it was a bit easier, just being kind of more remote-friendly from the outset, but it was a bit of a shock. It was also interesting – I don’t know how many listeners know this, but there’s typically a lag of a bunch of weeks between the time we record an episode and the time it airs. Sometimes, it’s shorter but then it was also interesting just to even see the delta between kind of statements and learnings as a world community really, not just the security community that might have happened in just a few weeks between a recording and publishing.
It was definitely an interesting experience, just from a podcast production perspective, even before we started anything else. It came up a lot about challenges. I think the first and most obvious challenge that happened in security was the getting used to remote work. You see as an industry, it’s a little bit less about developers but it has become the top priority for people and that sort of securing surroundings. Doing things like zero trust networks really came up or Cloud Access Security Brokers kind of, CASBs, all of those topics as organizations just adapted to everybody being remote. Even if you supported remote employees before, suddenly you need to put more emphasis on sort of security responsibility or security impact of how they do their work and if their home computer is compromised, etc.
That was one topic that I think it was just a shift of priorities. I feel most organizations perceived it as a step forward. These are things that they wanted to do anyway. Everybody wanted this sort of zero trust surrounding or wanted more self-sufficient services. All the conversations I had about it were, yes, we have to shuffle and we had to drop a few things off the priority list to get this done, but it is a good piece of work, even if we eventually kind of get back to working in the office. That was one.
The second thing that talked about COVID or that COVID kind of touched on is the notion of relationships, and there was a lot of commentary about trust, and you could see this. I don’t want to sort of start naming names, but different guests might have come from a place in which they already had established trust between the development team and the security teams, and they were basically reaping the benefits of it. They weren’t worried that developers were circumventing things or not thinking about security. They weren’t reliant maybe so much on hallway chatter and being able to look over someone’s shoulder. So, I think it just demonstrated and highlighted how important that trust is that you’re not a dictator, that you’re not a controller as a security team, but rather you are a partner, right, and now you’re here to help the development team be secure versus to force anybody to be secure.
A variety of other investments like automation and visibility, all of those DevOps-ish type investments basically paid off in the surroundings. So the end result was those who already had it now kind of valued it and reaped the benefits, and I think were able to move faster. Those who didn’t try to now accelerate building it out, but trust is not something you can just easily accelerate. The automation thing you can, so there’s more emphasis there. We also saw that in the sort of the Snyk business impact as people really invested in automation and dev-friendly tools.
Then maybe just like a little bit of a rosy, sort of a positive note. For a bunch of people, COVID presented an opportunity to structure some relationships in a positive way that they didn’t have before. The two that I liked best was if I keep coming back to Adrian. He just gave a lot of great nuggets. He set up office hours at Atlassian for people to talk to which is much easier. It’s just a lower cost now that everybody’s remote. There’s no overhead to meeting individuals. He set those up, so people come in with specific questions. Ryan Ware at Intel talked about how he set up these 15 minutes stand-ups or sort of quick syncs with people. Or, again, previously he relied on maybe a little bit more sort of random conversations that might have occurred. So people did use this as an opportunity to just create a more frequent conversation with different people between security and dev, which I thought was a very positive thing.
[00:23:53] Simon Maple: Really interesting. I think COVID, while it does bring many, many challenges that we do all have to kind of get around, not just from a security space, of course, I really do find that what you mentioned there about the split between people who have adopted the more modern practices reaping those rewards and those who haven’t trying to accelerate that, I find that very interesting in terms of how you can create those relationships, build that trust when you are all remote. That must be so, so hard to do without being in the office, without trying to create those new processes, create those new bonds together. That must be extremely challenging.
Let’s move on because I know we’ve done almost 25 minutes on the first question. Let’s get to the second question and maybe we’ll have time for a third batch.
[00:24:31] Guy Podjarny: There was a lot of meat to it though –
[00:24:33] Simon Maple: There was, absolutely. No. Really, really, really great summary. Now, we’ve had a huge number of guests on, and you’re name-dropping a fair bit which is getting a little bit embarrassing. It’s amazing, right? You’re name-dropping some great people, and it’s just testament to the number of amazing people that we’ve had on. We have great feedback from a lot of our listeners about what they learned from some of the top people, what works and what doesn’t work in their environment.
What about you, Guy? What have you learned from the guests and what experiences have you gleaned from their experiences?
[00:25:01] Guy Podjarny: Again, kind of the short answer is like a ton. I feel this is what I love about the podcast, like this best job. I get to basically ask smart people with great experiences, questions that are of interest to me, and I learn from their answers. So I learn a lot. I tried to think about kind of condensing it to a few sort of specific practices that really I think shaped some of my thinking after I’ve heard them. One that comes to mind is Shannon Lietz’s conversation about secure abilities. So, Shannon Lietz runs the Red Team at Intuit, and she’s a longtime DevSecOps advocate and she talks about how you can use a four-nines measure to measure security. That’s a really interesting approach. She talked about that and about the importance of measuring exploitability and kind of weighing exploitability in issues that her team might find proved to be exploitable. How do they feed them into the system?
What I liked about it is there was a whole kind of challenge around measuring security through the episodes. But what I liked about this approach of the four-nines is that it can be rolled up really well because you have this sort of four-nines measure. Now, you can have a weighted average across projects, across risks which I really, really liked. That one is definitely how I’m thinking. In fact, Patrick Debois and myself at Snyk here, we’re sort of thinking about structuring something and building there, so stay tuned for that. That one I loved. It definitely sort of changed my view on metrics.
[00:26:27] Simon Maple: On that, who do you feel that will really help the most, because I know developers are obviously trying to spend as much time as they can building and testing and things like that. Will this kind of thing help them to almost state the importance of spending that time on security or is it more for maybe the security teams to be able to show the value that they’re injecting into the project?
[00:26:47] Guy Podjarny: Well, a good metric should help all of the above, and the idea is to really measure what you intuitively value, so being able to actually measure a combination of your risk and the way that you’re handling that risk. But I do think that the four-nines measure has an extra added value that it has some intrinsic affinity or familiarity because of uptime. I think most developers, most ops people are familiar with 99.95% up or things like that, so I think it has an added advantage for dev and DevOps to appreciate that metric. But fundamentally, it has to be broadly applicable. Otherwise, it’s not a good metric. It needs to be measured by everybody. It needs to be valued by everybody. That’s for measuring.
One tip that I got on the episode from him himself but also in reference was, again, from Adrian Ludwig. He talked about the whole journey that mobile security underwent to rethink security and how cloud security represents that opportunity for us to sort of rethink application security. I loved that thesis. Mobile security has indeed successfully rethought operating system security. It is dramatically more secure than the desktop operating systems that we have because it rethought some of the fundamentals. Taking that lens to cloud security is something I find very inspiring, thinking about how, if we got it right, we can really properly encapsulate every microservice. We can automate more things. If we took the right approach to cloud native appsec, we don’t need to just match or keep up with security. We can leapfrog. We can really do something that is spectacular.
He referenced Google’s BeyondProd which is a model that is interesting that tries to define this. I think even more than that, I was just inspired by the vision, so I really like that. I really like this notion of don’t just think about how do you catch up to the cloud. Think about how can cloud help you rethink security and not be stuck with yesterday’s problem. I liked that. I think I already mentioned security champions, but one of the models I liked also was on the other side, which is this notion of AppSec partnering. This came up I think even last year from my envision with Sara Dunnack there who was talking about how they’re modeled in this partner program. The penny that dropped to me in these conversations is the duality of security champions and AppSec partners.
On one side, you have people on the AppSec side partnering with a sequence of engineering teams or sort of application teams. On one hand, it’s a good relationship because those AppSec people, they have a relationship with the leads in those application teams, and so they can collaborate better. But also, on the security team, that AppSec person knows more about the applications than maybe the rest of their team about these specific applications, and then security champions is reversed. Those security champions, again, act as a bit of a liaison to the security team. But also, amongst their application teams, they know more about security. So I find that model really, really nice. That wasn’t really one takeaway. It was more of an aggregation of these different models.
Then I guess the last one I’ll give is like a little bit more tactical but very concrete which is Tad Whitaker at CircleCI. He had a very positive spin on compliance. We keep talking about compliance as this negative. It’s good to be compliant. But from a security perspective, oftentimes it’s perceived as security theater or something that you do because you have to. Tad came with a very refreshing approach early in the year, our conversation I think was maybe even pre-COVID, and talked about how they used FedRAMP. He gave specific concrete tips about how to achieve FedRAMP compliance but also had a very positive approach to how FedRAMP is a great way to raise awareness to security and how everybody can rally around a business enabler or a business opportunity to be FedRAMP-compliant and therefore be able to sell into sort of the federal government. That’s really great.
In fact, Thomas [Owen] working here at Snyk who’s our head of security, he actually came with a sort of the same approach a little bit later in the year when we had that conversation. So I really liked that. I think I got a little bit absorbed in that critical view maybe of compliance. I think that shake up was a positive. So, there’s probably dozens more but those are the ones that come to mind.
[00:30:49] Simon Maple: It’s interesting that last one that you mentioned. It’s great to hear that, even here at Snyk, we’re taking these and we’re actually implementing them. Not just concepts but some of the more concrete tips that people mention and actually implement them in our org as well.
While we have guests that have a lot of great oversight across their organizations in the security space and beyond, but when we think about what people can actually take away, like that last example you mentioned on compliance, what people can actually take away and implement themselves, what other concrete tips or methodologies would you say you really found useful for people to take away and do themselves at their own orgs?
[00:31:26] Guy Podjarny: Probably every episode has a bit of a nice nugget there, but one that comes to mind is Nitzan from Spotify who came, and she’s new to security. She came into security from Agile and from testing. She started her journey by doing this survey of what is the state of application security or product security at Spotify? Then the trick I loved is that she then used the response to the survey as a means to identify her best allies. So, the people that were most excited to give her input, to share their views, who were sort of keen to make sure that their views were heard, naturally, those are the people that cared the most about security within the development team, and so the learning process, the survey was actually a great tee up for identifying her champions, her leads as she said about growing that. I love that trick. I think it works everywhere. In fact, you brought it into security from past experience, so that’s a great one to use. It’s so simple and yet so few companies do it.
I love this notion of 10x and how Neil and team there put vulnerability burn down charts as part of their daily scrum meetings. Not a lot of teams in general show burn down charts of things in daily scrums. But still, whatever recurring metrics that you’re showing, showing security there is just a natural tenant. I think it’s a really easy step. It’s nothing. It’s added chart. Don’t put all your security metrics there. Put one, but it just raises security into the daily conversations, and security visibility is also important. I found that to be a great, great trick. So, that’s another one I’ll share.
I think more maybe more on the enterprise side, Wendy Ng from Experian, she had a nice approach. She was rolling out, I think that journey is a challenging journey to embed DevSecOps in Experian, but she took this approach where she identified the teams that were doing best. Instead of saying, “This is the way you should do it,” she identified the teams that did DevOps best and DevSecOps best. Then she interviewed them and basically introduced those practices to other teams that were not doing as well. So, instead of her coming and saying this is the way it should be, she identified the ones that are doing it best. In a large enough org, there will definitely be a wide variety and approach that. I like that because it was proven to work. These are methodologies that are proving to work inside your organization and scale that.
Larry Maccherone from Comcast, who had this incredible podcast about data, he had a bit of a similar kind of gradual approach. He did a bit more centrally identify many, I forget, like several dozens of metrics of practices that should be rolled out as part of the DevSecOps rollout. But he didn’t go off and say, “You need to do all 40 of them.” But rather he said, “Okay, here is,” I think it was like 10 of them? “Let’s get those done.” Then, when one of them was achieved, the average threshold, everybody was achieving them. Now, you say, “Well, now everybody else is doing this correctly, so there’s no real reason for you to say it cannot be achieved, this level of success on this practice.” Now, that’s kind of set up. Let’s ensure that that’s a requirement from ready and add another one that we’re leveling up. Pick another one of those dozens to build up, so it’s good.
It’s outside of the podcast but I had someone phrase it correctly which is, “Security needs to be ahead of where the rest of the org is. The bar should be ahead of where you are, so it pulls you up, but not so far ahead that it’s out of reach, right? That people are not inspired by it. So I like that.
Yeah, I think those are the concrete ones. I already mentioned security chaos engineering which I love. So, if you’re using chaos engineering, adding security chaos engineering attacks into it is a great tip.
[00:34:57] Simon Maple: Excellent. Let’s move on to a question, which we also asked last year, which is around what security and development teams are doing particularly well today and where we feel they need to improve?
[00:35:09] Guy Podjarny: It’s always tough because, to an extent, we get slightly more forward-thinking security leaders on the podcast. There’s a bit of a selection bias of people coming on the podcast that they’re on this journey to embed the development. But still, I feel the whole narrative of security as an enabler move away from the kind of controlling, dictatorial model and think about security as service providers and enablers and empowering entities succeeding through developers. That came out far stronger this year. Guests volunteered it. They suggested that this is how they worked. Also, many of them didn’t necessarily think about that being unusual. People kind of mentioned this as like, obviously this is how we think, so there’s definitely a momentum about it.
I also had some interesting conversations about this from large organizations that are more decentralized. Ian at Cimpress was a good example that comes to mind, where they’re literally a service provider to a whole bunch of different business units who are quite autonomous, and so really kind of took that approach to the next level, which I thought was really nice.
[00:36:14] Simon Maple: Do you feel like that is largely because these are just the fruits of hard work that’s occurred over the last year or two? Or do you feel like there’s a greater shift in the way we’re working which is causing this almost change in our thinking of the way things work?
[00:36:26] Guy Podjarny: Yeah. I think it’s a combination of a need and the art of the possible. I think on one hand you’re seeing a need. You see that DevOps is happening. People are running faster and faster. That you’re unable to control or dictate to them, and so you sort of see that methodology and maybe even experience that the pieces where you are, even if not intended, acting more as an enabler, they’re picking up more support but feel like the need, the fact that the previous approach, so the carrot and stick here. The stick is happening where people are circumventing security teams not embracing this approach, or those security teams are just getting burned out because they just can’t keep up. They can’t live it up.
On the flip side, I think what you’re saying is you’re seeing a lot more examples and demonstrations of things that are working. I’d like to think the podcast is one of those means, and DevSecCon as a community sharing and demonstrating where some security leaders who have found techniques that work share that with the rest of the world. So, you see the alternative. It’s working there, and I feel like there’s an increasing number of capable people, security sort of staff, who would basically not come work for a team that works in a different way. So, I think you also see a little bit of the talent push, and there have been several mentions of people who have left previous jobs because it wasn’t that surrounding. That was just a bit of a doomed to fail if you don’t embrace this approach, so a bit of a combination.
I think still on the working-well side there’s been a lot of emphasis on communication, and it really came along throughout. I think that was actually even the primary theme of Douglas DePerry who was at Datadog. I think it was even his primary emphasis on communication thesis of if you communicate well good things will come from working between dev and security. I think that also if you communicate better, development tells you, “This is what we need. We need you to not tell us. We need you to help us do these things ourselves.” That’s the patting-ourselves-on-the-back piece. Oftentimes, we sort of think about what’s not working well.
We’re going to talk a little bit about cloud security. I don’t want to sort of belabor that point too much but I think cloud security right now is a mess, and it needs work. That’s probably a piece that definitely is tough. I think it’s a little bit tough because there is a little bit of a job security element to it when you talk about cloud security containers were a new thing. They came onto the scene. They weren’t threatening anybody’s job. I’ve been doing this for 10 years now, somebody needs to do it. When you think about cloud security, if you think about like asset management and IT security and who’s managing the networks, there’s actually a bigger shift or a bigger risk for people thinking they need to re-skill, and they might not be able to. So, I think there’s a bit more clinging to your current position.
But also, frankly, containers are smaller than cloud security is just a bigger topic. So I think that one’s a mess and, as I said, I don’t think it’s going to be a one-year thing. Then the other one, which I think I actually mentioned in 2019 as well, is measuring security. It’s amazing how nobody has cracked that nut. We talked about it a ton. We talked about data. We had never much around – You have to if you haven’t listened to that episode. You really have to go back. Probably it’s the best example I’ve seen on measuring security and how it’s rolled out in the organization. It has some interesting insights about how practices might have surprised you or, for instance, on how, if you invest in finding vulnerabilities, but you don’t improve or you don’t invest in fixing vulnerabilities, your security posture is measured by a variety of risk indicators. For him, it’s actually worse. You actually get worse. Even though you know about more, you just have this sort of false sense that you’ve invested already and you don’t fix it versus if you invested. If you invest in find and fix then, you’re actually doing better. There’s a whole world of those insights.
That was probably the closest, but it’s still miles away. We had ecosystem data session. We had Alyssa Miller and Gareth Rushgrove from Snyk, and we had Alanna Brown from Puppet. We talked about ecosystem data. So, I think we’ve gotten a little bit better at kind of measuring ourselves as an ecosystem. But measuring one’s security like an organization security status and practices, I think that is still very, very messy. As I start to sense patterns from the different episodes, I find that eventually people boil down to one of three primary measures, which are either one or more. They can use one or all of those three. One is they might be measuring their risk level. How many vulnerabilities do I have? Is the simplest example of it.
The second thing is that they are measuring security control’s efficacy. How often was something blocked, right? Or how often was an issue discovered? So, that’s more about sort of the effectiveness of the control itself and that’s more judge your vendors or kind of measure whether you’re using the right tools or not, if you should swap.
Then the third, which I think is actually the most popular, is that people are not measuring so much security as adoption, which is they’re saying, “I’ve logically decided which security controls I need to have where, and really what I’m measuring is the rollout. I’m measuring how often is the security control in place.”
I think that’s probably the best I’ve got at the moment in terms of recurring themes around measuring security. But what’s very clear is that we don’t yet know how to measure it and kind of the old adage of if you can’t measure it, you can’t optimize it comes to mind. So I think that’s going to be holding us back. I know I personally expect to spend some time on it this coming year and I think it’s a topic that the industry continues to think about and to talk about.
[00:41:55] Simon Maple: Yeah, absolutely. And very complex. Just from the sheer number of things that you could measure, as well as what you can reliably take as a result from those missions.
[00:42:03] Guy Podjarny: Absolutely, and it’s hard. Measuring security is invisible. Risk is this murky thing, and so there’s a reason it’s hard. It’s not that people aren’t trying or aren’t smart. It’s just a tough problem.
[00:42:14] Simon Maple: Let’s look forward now and go into what we feel is going to be happening in 2021. In terms of some of the biggest security risks and challenges that you feel that our future guests are going to face in their organizations, what do you feel are going to be – I can almost even foresee some of the answers here from what you’ve already said. How do you feel guests going into 2021 are going to be most concerned with going into their organizations?
[00:42:39] Guy Podjarny: Yeah, and I think I’ve been sort of alluding to a bunch of those because it’s not like in 2020 you’re done, and then in 2021 you’re doing something else. So, from a practical perspective, from a specific technologies, I think the one that’s going to be at the top is this whole notion of cloud security. DevOps will accelerate and cloud would be used more. It would go from 10 to 20 to 50% of your stack of your applications. It’s going to be an immediate burning problem. Also, as infrastructure, as code gets adopted, I think that’s going to be a theme. So I would expect, first of all, cloud security as a whole will come up, but specifically infrastructure as code. As applications use more cloud, then they would increasingly find that they find a problem when it is deployed and they want to move it earlier. Lo and behold, that earlier is in the Terraform script that sits in their repository.
Today, when you talk to organizations, which I do a ton of, I talk to hundreds of organizations every year, most of them don’t have a good, structured way to identify security mistakes as part of their infrastructure as code scripts and processes early on. So, I think that’s going to change in this year, and we’re seeing a lot of industry movement there. I think that’s probably at the top. The second one I guess is measuring security. I don’t think that 2021 is especially here for this except for volume. It’s just like there’s going to be more activity, and so the impact of measuring security is going to grow. We talked about that. Then I think the last one that comes to mind is really the security team split that I mentioned before. It’s not just an evolution. It’s really important. People need to have the right affinity, the skill sets change. I do think 2021 is going to be a pivotal year in terms of defining those elements. Those are I think the three top ones.
[00:44:25] Simon Maple: So, to get some good guests on in 2021 then, Guy, to talk about this, who do we need to recruit to talk about these topics? Who would be your ideal dinner party group of guests to do those talks?
[00:44:36] Guy Podjarny: Yeah. Well, I guess it ends up touching on similar. So definitely anybody who sort of feels like they have some good insights on measuring security, I would love to talk to them in general and to bring them on the podcast to share those. I think it’s an area where nobody has a solution, but we need to piecemeal it together. We need to have everybody share this trick they found and that trick they’ve found, and I’m happy to be a platform to help get all those pieces and then eventually sort of create the whole picture. So, definitely would love to get insights about that and I guess also the security team structure. I already asked probably every guest that starts, how do they structure their security organization? But if you’ve evolved your security organization, I would love to talk to people about it.
Notably in larger organizations, I think it’s a little bit. I’m saying this with all due respect as someone growing a company here at Snyk as well in hyper-growth, but it’s a little bit easier when you are forming. You’re a cloud native organization and you’re forming your organization this way. It’s much harder to restructure your team when you already have hundreds of people in that organization, and they have different skills, so we’d love to hear those types of stories.
The second topic is indeed on the cloud native side. I think what we haven’t discussed as much is the runtime side of things. Alongside the changes of moving the infrastructure definitions into dev, another thing that is happening is that the knock and sock are coming together closely. You see security offerings in the APM space, like Datadog and Elastic, kind of becoming more sort of seem like functionality. That’s cross pollination, the incident response of security versus the incident response of an outage, how and if should those be related. I think those are really interesting topics. I’d love to talk to a few more guests about those types of changes.
I guess the last thing I would say is a bit more about community than it is about a topic, which is I’d love to have a more diverse guest list. We feel, running this podcast and looking for experienced guests to bring on, really places you in front of the diversity problem we have in the industry, and you just sort of see how many sharp organizations. You look for female or other diverse leaders of those security organizations, and there just aren’t enough of them. So, I think a part of it is you can’t become what you can’t see type notion. A part of it is historical baggage, right? Even if we’re better as an industry in hiring juniors that are more evenly split, people that have experiences, there are a little bit of yesterday’s mistakes that lead us here.
But still, I’m quite committed to making that investment. So, if you know of bright women who have a security perspective, whether they work on security. Not just women, sort of diverse guests to have as a whole who can share kind of a good lens on security, whether they come from the ops side or the dev side or the security side, but on these topics, I would absolutely love to see that. Or, by the way, talk about diversity initiatives. We had Tad Whitaker of CircleCI mentioned before. He also created and runs Shecurity, which is a great org initiative to actually help women get into security. We’ve talked about this with Vandana Verma. I talked about it with Alyssa [Miller]. We talked about their initiatives, but I would love to raise those up. I think we need to talk about it. We need to find opportunities to both give air time to the problem and, once again, just like the measuring and just like all those others, inch our way towards a solution, find the right practices that make it work, and go from there.
[00:47:57] Simon Maple: Really, really important problem, so really valuable to bring up and talk about. That’s great to hear. Guy, over the last year as well, you always will very often talk to people and ask people what piece of advice they’d give to the listeners, what their pet peeve would be. So let’s throw that question at you. What would your one piece of advice or pet peeve be across the whole of 2020 that’s really still great to see?
[00:48:22] Guy Podjarny: I had a hard time reducing things to like three or four bullets and getting it down to one, but I guess it is kind of my own doing here.
[00:48:28] Simon Maple: How about this? How about you name 10, and I’ll cut you off after the first one?
[00:48:33] Guy Podjarny: I don’t think I can say listen to the podcast, although that’s a good – If you’re still here, you probably are. I would say don’t boil the ocean. I encounter a lot of angst from people saying, “I would love to be there. But in my org, it’s really hard.” Similarly, I encounter like a lot of paralysis by analysis. No, we’re trying to design this perfect new reality into which we will move security and how we practice or – I feel that’s just not the way DevOps and Agile works, and you can be stuck in that world until you basically have a really big fire and you have to rebuild the house.
My advice would be pick something and get started. Whether it’s cultural, about hiring a new team, whether it’s a piece of technology automation that you want to put in, whether you want to introduce some automated fixes to sort of a specific security threat or put a measure in your stand-up meetings, just pick something. You don’t have to have a massive, complete, holistic program. Once you hit some critical mass, so you can go off and you can scale it. You can do it in parallel, but don’t boil the ocean.
Pick something and get started, moving security down the right journey, whether it’s a whole transformation or on specific technology space like this sort of cloud native AppSec space.
[00:49:46] Simon Maple: Wonderful. Guy, that’s the last question we have. Big, big thank you to all your hard work and all the guests and topics that you’ve brought onto the podcast and discussed over the last year. I hope you enjoyed it.
[00:49:57] Guy Podjarny: Yeah. Well, I have a whole blast doing it. So for me, it’s all upside.
[00:50:01] Simon Maple: Wonderful. Thank you very much. That, of course, is a wrap now for 2020. So we really do hope you enjoyed the show. Thank you very much to all for listening, following along, and also sharing amongst your colleagues and friends. This year, as everyone has, we have adapted with COVID, or to COVID should I say, and one of the things that we’ve done is combined our community group. So we had the MyDevSecOps virtual group, as well as the DevSecCon physical conferences, and we’ve combined both of them together to make just one DevSecCon brand, and that brings The Secure Developer into the DevSecCon family. As part of DevSecCon, we will continue in future to do physical in-person events, as well as many of the virtual meetup and programs that we do at DevSecCon website.
In 2020, we’ll be bringing you a whole load more content as well on this podcast. As well, of course, as many, many more different types of programs on the DevSecCon community platform. Of course, please do let us know if you have any feedback or comments, or if there’s anything else you want to see or hear about at email@example.com. With that –
[00:51:09] Guy Podjarny: Even in 2021. It’s like – I think you said 2020 or –
[00:51:11] Simon Maple: Even in 2021. I did say 2020.
[00:51:12] Guy Podjarny: Even in 2021, we’ll do the –
[00:51:15] Simon Maple: Yeah, still living in the past. Yeah, absolutely 2020. Thank you very much, Guy. Anything you want to leave us with?
[00:51:20] Guy Podjarny: No. Just thanks, everybody, for listening in, and really you make this happen. Be vocal. Share your views, what you like, what you dislike, and we’ll adapt accordingly. Thank you very much and have a Happy New Year.
[00:51:32] Simon Maple: See you all in 2021.
[END OF INTERVIEW]
[00:51:37] ANNOUNCER: Thanks for listening to The Secure Developer. That’s all we have time for today. To find additional episodes and full transcriptions, visit thesecuredeveloper.com. If you’d like to be a guest on the show or get involved in the community, find us on Twitter at @devseccon. Don’t forget to leave us a review on iTunes if you enjoyed today’s episode.
Bye for now.