Five Ideals for Better DevOps and Security

[0:01:15.9] Guy Podjarny: Hello everybody, thanks for tuning back in to The Secure Developer. Today, we have a great luminary in the world of DevOps and an early security advocate and leader, Gene Kim. Gene, welcome to the show. It’s great to have you here.

[0:01:28.4] Gene Kim: Great to be here and congratulations on the exciting start to 2020, Guy.

[0:01:32.3] Guy Podjarny: Thank you. We could cover – it was really tough to sort of pick the topics and talk to you because you know you have such a rich history in both sort of the security side and the DevOps side. Instead of me kind of doing it justice, can I ask you to just you know, you kind of went into security and then out of security – tell us a little bit, you know, your journey but not really, I guess you’re sort of still there. What was your sort of path into security with Tripwire and then you know, past that until today?

[0:02:00.2] Gene Kim: For sure. I’ll first give you my official introduction and then I’ll weave in kind of my strange journey in and out of security as you said. Been sitting high performing technology organizations since 1999 and that’s the journey that started back when I was a CTO and technical founder of a company called Tripwire which is in the information security space. I was there for 13 years but the journey of study of high performers continues to this day and so the research really focus on these amazing organizations that were simultaneously the best in development.

They were the best in operations and also the best in security. Kind of not unsurprisingly, that led to stumbling into the DevOps movement which has been one of the most exciting, rewarding journeys in my entire career. That led to writing The Phoenix Project in 2013, The DevOps Handbook in 2016 and most recently, The Unicorn Project.

How does security fit in? I got my first job as a system administrator when I was in high school. I was writing automated QA tests for the file system for Sun OS and also doing system administration and it was during that time when the Internet Morris Worm hit and so I couldn’t help but notice that there was one person at the university, generating all these amazing papers on what were the methods of exploits, how could better prevent the technique and correct for it. That was Dr. Eugene Spafford.

That’s why I went to Purdue. That’s why I wrote the original version of Tripwire which went on to become the most widely used creative tool for Unix at that time. By that time, 1997 came around, we decided to commercialize it through the company. It’s kind of – the funny thing was that it was my observation that these high performers, they weren’t high performers because of great security organizations, they were high performers because of great operations organizations.

I gravitated towards ops because it was my observation that that’s where the saves were made. We saved the customer from terrible developers who blew everything up in production because they didn’t test enough. It was ops who protected our data despite ineffective security people and so that’s kind of how I think that led to me staying security from a different angle than the most security profession and now, as someone who self-identifies not as an ops person anymore but as a developer. I have a whole different sensibilities around security. I want someone just to do it for me.

[0:04:03.4] Guy Podjarny: Well, you’re basically – you represent every piece of the DevSecOps kind of trio, you know? The personal and visceral type of way. I think – I love that and I think you know, sort of probably the ops perspective and maybe even the start of that and you can probably even almost claim Tripwire was like an ops – I guess, info sec product but probably had more appeal to sort of ops people than many other security offerings out there. It’s not that much of a surprise that kind of that world has led you to sort of you know, digging into the merger of those spaces.

[0:04:36.8] Gene Kim: Interesting. In fact, to resonate with you, right? It was like, it was our observation and we actually validated this through benchmarking and very much like we did in the state of DevOps report was that, the organizations that had the best security outcomes were the ones that had the best change control strategies and configuration control strategies.

It’s about making sure that what was running in production matched no good build, that changes were visible and you know, we’re authorized. The thing about authorization probably is not true now, nearly 15 years later but it was really this kind of aha moment that security had more to do with not the vulnerabilities [inaudible 0:05:12] practice, it was the practice of internal security really had to do with things outside of security which I think was a very powerful epiphany for me.

[0:05:17.7] Guy Podjarny: Yeah, for sure. Let’s maybe sort of dig in into kind of that knowledge that’s stuck in your mind there. Let’s start that journey with The Unicorn Project. You just wrote that book and it touts, not to do too much spoilers over here, right? But it revolves around these five ideals. Can you give us like a quick overview of what the book is and then we’ll kind of start with DevOps and then switch to security?

[0:05:41.4] Gene Kim: Unicorn Project is based on Parts Unlimited and the same timeline as The Phoenix Project. Instead of being told from the ops leadership perspective, it’s really told through the eyes of a senior developer and I think, what made me gravitate towards that subject is really, that its all these problems that still remain.

There’s all the absence of awareness of all these things that are required to enable developers to be productive. This is also other problem of like data. Just like the DevOps movement, rightly pointed out how difficult it was to get code to where it need to go which is in production so that customers are getting value. Same thing exist with data where it’s often locked in systems of records and you know, data warehouses where it takes weeks, months or quarters to get to where it needs to go which is in the hands of developers to use in daily work.

I think the other thing that I wanted to really explore was this very often is a very – a lot of resistance to support new ways of working, right? To have people in functional silos who want to do things away that they’ve done this for last 20 years and information security is certainly one of them but it’s not just information securities, it’s QA, it’s development, project management. All the things we need to change the way we work and so the vehicle to tell that story was the five ideals. The five ideals are really meant to articulate kind of, the things that I just have a lot of passion and I think are very important to me.

The first ideal is locality and simplicity. That’s really meaning to what degree can developers do what they need to do in one place versus scattered or splattered everywhere. The second is, what I think the outcomes are, focused flow and joy. When developers are able to do what they need to do without having dependencies on scores of other people. You know, they can actually focus on what they need to do and we get a state of flow, meaning, that state we feel when we lose track of time or maybe even sense of self, right? Because we love the work so much.

That requires a third ideal, improvement of daily work. Improvement of daily work even over daily work itself and technical debt. There’s no way to eradicate that without that ideal. That requires the fourth ideal, psychological safety, you know, to what extent can we talk about problems without fear of being blamed or ridiculed or being diminished. And then, the fifth ideal is customer focus, what extent can we really focus on the needs of the customers for what customers depend on, right? The core competencies of our organization that create lasting durable business advantage versus context which is everything else, which might be mission critical but it’s not what our customers value. Those are meant to really embody those concepts.

[0:08:06.3] Guy Podjarny: Got it. Powerful ideals and you know, []inaudible 0:08:09] can write a book around those. If we dig a little bit into those, maybe like an interesting – these are all told from a developer’s lens, right? You’re sort of thinking about those organizational ideals but their former developer lens, right? Maybe, let’s kind of dig in to each one of them a little bit and talk about how can they manifest in the context of security, right?

Because the book is mostly around the developer experience and the DevOps surrounding but as we said, you know, the whole thing is intertwined. Let’s kind of go through them one by one instead of the – I’ll play kind of the annoying devil’s advocate a little bit as we explore.

[0:08:42.2] Gene Kim: Absolutely. Yeah, the first locally and simplicities, I love – the best metric around this, I would call the lunch factor, right? If I need to get something done. How many people do I need to take out to lunch, right? In the ideal, it’s like the Amazon ideal of the two pizza a team. It means, every team can independently develop, test, secure and deploy value to our customers without having to deal with 20, 30 different teams, right? In a complex deployment, you probably have to take the entire building out to lunch, right? That means, it’s slower, the outcomes are worse and so forth.

In the ideal, right? Everything is not tightly coupled together. To deliver a feature or implement a security control, ideally, we just need to make a change in one place, one file, one module, one name space, one application, one container, whatever. Not ideal, you have to change everything. I think in security, is equally, if we sort of invert the point of view.

In security, we want the same thing, right? We want a central point of control and I think this is why things like Kubernetes side cars or aspect oriented programming is so important because it means in Kubernetes, right. If you change the side car, every component or every container depends upon that side car can also benefit from it. Can change it in the side car and then every proxy, every database connection and so forth can benefit from that. I think it’s great for development and it’s great for security too.

[0:10:04.1] Guy Podjarny: The principle, there’s almost the security, there’s development and then there’s the security the developer might do. I like that, sort of the side car notion or maybe other sort of – I guess framework, sort of you know, platform, maybe it’s like React’s resolution and sort of cross set scripting, right?

[0:10:22.9] Gene Kim: Great.

[0:10:23.2] Guy Podjarny: Crack at that as well. You have some security elements sort of built into the platform or the security could inject into the platform for it.

[0:10:32.4] Gene Kim: Nice.

[0:10:34.0] Guy Podjarny: Would you say, when you talk about locality though, when you talk about something that applies only to this application, does that imply that – would you say developers should basically think about each one of these systems as local to the other systems as well? You know, that other two pizza team that sits off to the side?

Would the principle here be to also kind of disregard them or sort of think of them as just like the adversary and kind of protect your perimeter of not actually go through [inaudible 0:11:04] but you know, just in terms of like your role in, put some parameters, treat them as untrusted?

[0:11:09.0] Gene Kim: I think that might be a part of it. I think maybe an extreme example is what happened when in Amazon in their Obidos system in 2005 where basically, everything over the years, everything became complected together where in order to make any change at display logic or business logic or anything, you had to touch that one system, right?

All the database procedures and so I think you could call that – yeah, exactly right? That happens to all of us and so the countermeasure for them was that famous Jeff Bezos memo that said, the only way the teams should interact is through hard API’s, right? That forced the rigid separation, right? Not so that they aren’t trusted, it meant that every – re-enabled teams to be able to work independently of each other and so I think, a side effect when you have that kind of bounded context or domain driven design is suddenly teams that we’re once intertwined and completed together can finally, are liberated from each other and so they can actually work independent.

The result at Amazon is amazing, right? They went from no one being able to make changes to doing – the numbers. I think it was 11,000 per day in 2011, 136,000 times per day by 2017. It’s a great proxy to what extent are developers able to do what they need to get done.

[0:12:21.9] Guy Podjarny: Is it fair to say that from a security perspective, for you to be able to provide or sort of allow that autonomy. Then to an extent, each of these teams also needs those API’s to be security boundaries like they also need those API’s to say, well, I want trust that other team has validated inputs and things like that before they got into my back end micro service because I guess, you’re also doing end to end teams but still, you have to think about those API’s as a secure interface.

[0:12:50.5] Gene Kim: For sure. I think that’s something even more profound that happens which is that all of these teams are working independently and the ideal, right? They’re using common platforms as a security teams build for things like authentication, brake limiting and so forth because ideally, we don’t want each team to create themselves.

I think one of the best examples of Microsoft, it did internal benchmark many years ago counting how many XML parts in libraries they have and the answer was like, 300. Everyone had to build their own because the cost of taking a dependency was so high, their formulation was we should have one and we should use whatever Bing uses because that’s likely the most correct, the most performant and so yeah, that is indeed the case with their one engineering system which is they have one XML library that is used across every Microsoft product.

I think we can have those security concerns, or expertise, whether security, performance, QA, embedded in platforms that can be universally used then everyone benefits from that expertise.

[0:13:49.0] Guy Podjarny: Yeah, for sure. I guess that’s kind of where simplicity kicks in. Locality, we could kind of come out of this and say locality is locality from the other teams and you need to just as you’re sort of driving API interfaces, those API’s should be trusted and then simplicity is really where you need whether be it side car or input sanitization library or whatever it is, you need your security team and your platform to kid of help build components in and you’re not local from the platform.

But that’s okay, you’re building on the platform and the platform should make it simple.

[0:14:19.4] Gene Kim: Another model side effect, right? If you find an issue in the input sanitization library, you can fix in one place and then in quick – ideally, very quickly, that can then be inherited by every other organization, every other team in the organization. That is definitely satisfied that principle locality.

[0:14:36.2] Guy Podjarny: Yeah, excellent, yeah. They’ll get there very fast if you do 136,000 deploys a day. That’s an especially sort of past. Okay cool. We solved 20% of the world’s problems over here, let’s go to the next 20% so the second ideal, focus, flow and joy. How would that manifest for security?

[0:14:55.0] Gene Kim: Here’s kind of the funny security journey for me. It’s my belief that for developers, right? We want their best energy spent on solving the business problem. You know, what that means is that we don’t want them having to worry about things outside the application, right? In terms of, “I’m going to worry about input sanitization or authentication, authorization, secrets management.”

In fact, what I found now, self-identifying as a developer is that there are all these things that I used to love doing, especially as an ops person that I just despise doing now. I hate anything dealing with the download configuration files. I hate connecting anything to anything else because always takes me a week. Secrets management, I always get wrong. I’m the jackass who always put secrets accidentally into the repo, and I don’t know how to take them out. I’m the person who you don’t want doing anything involving Kubernetes, I loath. These are all important things. Dependency management, right?

It’s like, updating dependencies, very important, so all of these things are important. It’s just things that I don’t want to do anymore. I don’t say it to diminish these things. I think it just shows what the economic cost is to take a developer out of that state of flow, you know, where they’re doing their best work and then having them deal with something outside the application so it really in my mind, it just shows how important platforms are, where you could take the expertise of you know, people in infrastructure, operations, QA and security and use that to support developers who don’t need to worry about it.

By the way, when I use the word flow, I’m really citing the work of Dr. Mihaly Csikszentmihalyi. He gave the best TED talk ever about flow, the state of optimal experience and he describes flow as I mentioned, you know, that sense when you’re working on something, you lose track of time, even sense of self. That transcendental experience we have, you know, when we just are doing the work we love and you don’t have to worry about googling Java time formats, right?

[0:16:44.2] Guy Podjarny: Yeah, absolutely, and you find yourself coding at three AM because you just didn’t notice that it’s three AM.

[0:16:50.6] Gene Kim: Exactly,

[0:16:54.4] Guy Podjarny: You’re just in that sense of creation. I think that’s an excellent place and absolutely a place where sort of security sometimes kind of interrupt and you feel like you know, you’re being poked incorrectly. Okay, I love that and I think that principle works I guess for any aspects of work. Maybe – I have an example, tell me if it sort of, you know, works well. Like a mistake I often see which is to – in the context of dependencies, oftentimes, people would break the build, whenever a vulnerability is discovered and then one, I guess, breaking of the flow is, you run a build and since the last time you ran the build, the new vulnerability has been disclosed and it effects one of those libraries.

You’re in the flow and you’re’ running this build. You haven’t touched this dependencies, you don’t even know what that is and you break the build. That might be like one small example of it but it’s about like, consider the sort of the developer flow and what are they doing and when should you interrupt them? Like this is a critical, “pull the cord, stop everything, I don’t care that I’m interrupting your flow, you have to fix this right now.” Or is this something you can report in a more elegant way, less interruptive way, a sync way.

[0:17:54.4] Gene Kim: That’s such a great question and I almost want to answer it two ways. One of them is through the third ideal of improvement of daily work. Here is the basis, I have a couple of experiences that might resonate with people. I mean I do all my programming in Clojure which is a functional programming language that runs in the JVM and even there, there is a lot of updates and components but that is nothing compared to the NPM ecosystem where it is just orders of magnitude worth.

I personally had an experience where you don’t build your stack – in this case it is like React Native, you don’t build it for 60 days and 60 days later, it won’t even compile anymore because of God knows what and so that is the level of how dynamic the supply chain is just breathtaking and so I got a chance to work on the software supply chain, I say the software’s supply chain with that [inaudible 0:18:45] where we looked at the main central ecosystem and one of the most astounding findings was around –

So the psychographics of developers and we asked one question like to what do we consider updating dependencies and then patching vulnerable components painful and three clusters emerge, high pain, low pain and medium pain, right? And if you look at the low pain people, one of the startling differences is they are 10 times more likely to update dependencies as a part of daily work. I meant that if you – whether it’s daily or weekly or some definition of regularly, it meant that when very important vulnerabilities were disclosed, and you have to change them right away, the ones who are experiencing the least amount of pain were the ones that had some practice in doing it.

And more importantly, they had some rigour and discipline about doing that not just when emergencies happened and it does suggest that it was Jeremy Long, he is the author of OWASP Dependency Check Project, it really validates something that he’s talked about, which is really the best way to stay current on patching is just to update dependencies all the time and that was certainly validated in this research.

[0:19:47.2] Guy Podjarny: Yeah that is excellent and I think first of all very elegant segue into that third ideal but also like I think absolutely like basically you want to invest in being able, in being ready to fix in the context of dependencies and then a lot of that is just to be on top of things and I like saying that if updating is a risk and not updating is a risk, you can just pick which one and the vulnerability gets disclosed it just changes the risk equation. It just basically says what is it that that balance sways because not updating is now a bigger risk that it was months ago.

[0:20:21.3] Gene Kim: It’s nuts. In fact like another stark difference between the high performance and low performance where the ones with the lowest amount of pain were 11 times more likely to have some strategy at either being at most current or current minus one and it says there is – they have some sort of strategy that says, “You know here is where we are going to be at. We’re not going to be a bleeding edge, we are going to be a bleeding edge minus one or bleeding edge plus two weeks,” or something that gets us out of this horrible problem where we are so far behind we don’t even know how to get to the latest version.

[0:20:52.2] Guy Podjarny: Yeah that journey is like filled with peril on its own right from there.

[0:20:55.1] Gene Kim: Whether there is a few presentation that I don’t know, the GitHub folks made about their journey from updating from Rails three to Rails four and you know, it took them I think years just in time for Rails five to come out and makes you wonder, there is something really, really wrong when just staying current is this much work and yet that is the reality.

[0:21:13.4] Guy Podjarny: Yeah it is well and maybe back to your roots a little bit, in the server world, we’ve actually built a little bit of a norm that you just constantly update. You know, you constantly take this feat of updates often times without even knowing what you have just updated ready to take the next thing and not doing that is the anomaly, are the ones that aren’t typical. Well, in the application dependency world it is the opposite, you have to opt in so.

[0:21:36.4] Gene Kim: Yeah in fact maybe over drinks sometime. It is and I think it is based on this very crazy fact of life that despite semantic versioning it is acceptable that you have breaking changes in your API which in a more just world – wouldn’t life be simpler if we could actually trust that our components won’t break us when we update them.

[0:21:56.7] Guy Podjarny: Yeah soon to make those. Well interesting kind of equation there as well around the face of innovation. So I think we have covered three. I guess we got to throw in that sort of improvement of daily work in this last one. The next one is really interesting, psychological safety. That one is finicky for security. How do you think how would you apply psychological safety to security?

[0:22:16.8] Gene Kim: Oh gosh that is such an interesting observation, yes. So psychological safety was also covered in the State of Dev Ops report and in the Accelerate book that I co-authored with Dr. Nicole Forsgren and Jez Humble and you know the way that we talked about it was through the western organizational topology model. So the low – bad culture is, information is hidden, messengers of bad news are shot, we discourage novelty.

Whereas good culture is measured by what they call generative, we seek information, we train messengers to tell bad news, we encouraging bridging between teams and novelty is encouraged and The Unicorn Project did force me to go back into the literature and look at the famous project by Google, Project Aristotle, Project Oxygen where they found in their multiyear quest to understand what makes great teams great, the top on the list was psychological safety.

To what extent do people on the team feel safe to say what they really think without fear or ridicule, castigation, blame and so forth and year over year that is the most decisive factor in terms of correlating with great teams. It wasn’t the manager, it wasn’t roles and responsibilities. So that was really neat to look through the literature through that lens and one of my friends, John Smart, he was formerly the head of better ways at working at Barkley’s that eventually expand across all of the major divisions of operations there.

He is now a partner at Deloitte. He gave a talk at DevOps Enterprise just a couple of months ago about the need for psychological safety and security too just really painting how treacherous and dangerous it is to report bad news and when you are in a regulated industry and when you can actually disrupt business operations by raising an issue that is – he spoke very bluntly about it is that that is not a psychologically safe environment right?

That you are discouraged to tell bad news so I think there is this incredible symmetry about how important it is for development as well as oddly enough, I think one of the most psychological unsafe places to be is in security where I think many, many high profile security breaches where there was the largest loss of PII on the planet where I think pretty obvious that someone was sending out some bad news and they were just not capable of communicating that to the right person with adequate fidelity for the right things to happen.

[0:24:24.7] Guy Podjarny: Yeah, it is amazingly both one of the areas that it’s hardest to maintain psychological safety and yet the most important because it’s precisely the place in which you want to, you want people who see something to say something or you want them to be informed. You want to be informed but it is quite tough.

[0:24:42.6] Gene Kim: Oh by the way a friend of mine John Willis he told me about this amazing story at Fannie Mae where they found an Apache stress vulnerability and they actually took one of their most major business applications offline over the weekend because it was the opposite. They actually said, “Our mission is to protect the American mortgage holders so it is better to take the application down than to potentially jeopardize them by leaving the application up.” And I think that paints a shining opposite example of what could be.

[0:25:07.9] Guy Podjarny: Yeah absolutely that is a great example like clearly it is not awesome to take a system down but it is very consumer and sort of customer minded course of action for it. Do you think when it comes to security mistakes, when you think about the security incidents that happen due to – inevitably there is almost always someone made a mistake, which we are all human, are there any specific best practices you’ve seen around trying to assess security incidents or even proper size breaches, you know, are there organizations or examples that do this well for DevSecOps, for application security and the people involved?

[0:25:48.1] Gene Kim: For me I wouldn’t say that this is practical yet but I think this is the true north that we should be aspiring to. So in the Fannie Mae example they said, “We must take this application down because of harm to consumers and how it jeopardizes the real mission of the organization is so jeopardized by this. So it is better to take the application down than leave it up running in an insecure state,” and then really the question then becomes how do we engineer the environment, the systems, so that next time around, we can actually have more assurance that we can more quickly seamlessly and painlessly upgrade and patch just like we should be able to do an OS hot fix as you said, silently, invisibly, painlessly, constantly.

And so that’s a heavy lift and I think ultimately that that is where we need to go and so there’s more on the engineering side but I think that is really the most – that is the problem that really attracts me.

By the way, the other thing I’ll – it just came to mind when we talk about psychological safety is there is so many examples of where information was hidden in an organization that turned out to have horrific outcomes whether it was Boeing 737 Max or the diesel engines emissions fraud at Volkswagen, I mean those are all similar examples where despite all the norms being set by leadership or maybe because of the engineering norms, the norms set by leadership the information was hidden.

That did a grave disservice to the organization, consumers, society, etcetera so I think in the next decade or two it will be more obvious on how do we sort of encourage the right things that happen all the time.

[0:27:19.7] Guy Podjarny: Yeah absolutely and I agree with both the importance and the complexity or the difficulty of doing it. So which gets us to the fifth ideal, which I think we started touching on; the customer focus. So how do you see customer focus coming in, in security, coming into play in security and maybe even specifically for DevDec if you will for security over the digital world inside the business?

[0:27:43.8] Gene Kim: Yeah and this one I have the least numbers of answers on but I know it is important. So for me to paint why this was such an aha moment, I got to spend the day with the CEO of Compuware kind of the famous mainframe vendor with my friend, Dr. Mik Kersten the CEO of Tasktop. We went there because we learned so much from him over the previous two years. So we are walking to the campus I am looking at the agenda and the first thing on the agenda is a data center tour.

Now I turned to Mik and I actually apologized to him like, “I don’t know why we are getting a data center tour, I am not sure what we are going to learn by looking at their halon extinguishers,” and yet it was such an aha moment because you walk into the data center it was essentially empty because their two Z mainframes that were running and then the rest, it was empty data center space. You look on the floor and there is all these green outlines like in a murder scene where the racks used to be with these tombstones that said what business process used to run there and how much money did I save by getting rid of it?

So there was things like you know the ERP systems, there was the email systems, there was the treasury functions desktop backup and so essentially the formulation was cores are core competencies that customers pay us for and everything else is context. So things like the payroll systems their customers don’t care and so they essentially were able to reallocate $8 million of costs from back office to RND, which customers do care about and so for me that was just an incredible aha moment of how context can starve core.

And so to be able to reallocate $8 million from context to core was an aha moment and so I think it just shows one of the aha moments that came after that was how important the systems that support dev productivity are because you know the more we can make our developers productive so that they can focus on what they need to do without having to worrying about these non-functional requirements like security the better.

And so it means that it a lot of systems might go away but it does elevate the importance of the systems that developers use in their daily work.

[0:29:41.3] Guy Podjarny: Yeah, I like that and I guess if I can rephrase it a little bit you might say that security teams themselves need to remember that the goal of the organization is not to be secure. The goal of the organization is to provide value to customers and being secure is one of those means of doing so because you are not successful at providing value to customers if you’re putting their data at risk but fundamentally just being there stopping and stifling innovation, saying no to everything not letting anything through helps you be secure all the way to bankruptcy, right? That helps you fail like that.

[0:30:14.1] Gene Kim: And Alex Stamos who is the CTO at Facebook and before that Yahoo, I mean he had this marvelous post about what the modern security organization looked like and a lot of it was around the same principles like best way to eliminate PII risk is to get rid of it, right? Move it to another vendor where it is their core competency and he paints this vision of reducing the vulnerability footprint and service area by two orders of magnitude. So I think that is very much in line with customer focus.

[0:30:41.4] Guy Podjarny: Exactly yeah and I guess one other observation I would also make is that with some previous guest that were here one of the most notable was Jeff at Slack who talked about four of those components that you keep trying to highlight or describe them in terms of customer success and Slack did some great complaints about it actually like Andy Ellis said at Akamai, Akamai being secure is almost one of the key selling points and so in those cases if you are going to do it flaunt it a little bit right? So if you notice and then point out and describe it as a customer value and being able to just represent that to the world.

[0:31:11.9] Gene Kim: And what a great place to be because that is now funded out of the RND efforts. It is being funded by the same place that generate features which is the ideal place to be and you know if you have to secure these back end systems that no one cares about that is part of the cost center I mean yeah it is difficult maybe impossible to get funding for that.

[0:31:30.4] Guy Podjarny: So you got a whole world of other things that I want to talk about. We got to squeeze two more in maybe because I think we are already going a little bit longer than intended. So one is that you mentioned in passing is this state of software supply chain report that you have done. I know you have dug into four hypothesis there, can you give us the lightning summaries, people that can go off and read the report but the lightning summary of what were the four, what was this report, what are the four key hypothesis and what happened?

[0:31:57.8] Gene Kim: Perfect. Yeah, so there are four hypotheses that were set out to confirm or disprove. Finding number one was that project release frequency would have better outcomes and that turned out to be absolutely true. They were I think four times as likely to be used among those popular projects, they have more developers assigned to them, right? So that turned out to be true. The second hypotheses were projects and components that update dependencies most frequently also have the best patching behaviour and so that also turned out to be true.

The third hypothesis was we thought that the components with the least number of dependencies would be the most secure and that kind of make sense, right? It is because if you have fewer dependencies it is easier to update, etcetera, not true. Many found that it was actually the opposite. The projects with the most noted dependencies had the best patching behavior. They had more developers, they were more popular, very interesting. They update dependencies 50% faster.

The last finding was actually the most troubling so we thought that the popularity of the project was measured by GitHub stars or forks or even central downloads will be the most secure and that turned out to have no correlation on anything. The time to update, time to patch, release frequency etcetera, which is I say troubling because often I am adding dependencies to my project and the only thing I look at is the number of stars and forks, which turns out to have no predictive, no correlation with anything you care about. So this industry definitely does need better ways to discern which components should you take or not.

[0:33:24.7] Guy Podjarny: Yeah absolutely this is fascinating and I think the more light we shine on the complexity here the better informed we are. I guess that is the definite principle as well, which is you can’t optimize which you can’t measure, right? And we have to really dig into those to sort of understanding all the problems that plague us and how is it that we can approve to get better as an industry and as a community.

[0:33:47.0] Gene Kim: Absolutely.

[0:33:47.7] Guy Podjarny: One last bit, I ask every guest this. So you are not going to get away without it as well, which is you have a lot of insights and you talk to ops teams and security teams, pick your audience if you will but if you had one bit of advice or maybe it is a pet peeve that you want to make sure people stop doing or a bit of advice for a team looking to level up their security through, what would that bit of advice be?

[0:34:13.4] Gene Kim: Yeah, I will give one high level one that’s almost applied to but then one very concrete one and I think the whole notion of shifting left, baking security into every stage your daily work, I mean that is one of the most decisive findings in the State of DevOps report but then to do something very technical is something that I am actually not very good at and I am always looking for better ways is that advice from Jeremy Long is that the best way to stay secure in terms of updating is just to update dependencies all the time.

And it is not something that I am terribly good at but I know it is important and then it is certainly born out by the research the notion that we have to make updating independencies something that we do in general and that will pay off in spades when there is a vulnerability. My favourite example that Jeremy Long told me about was the prime faces vulnerability where the fix actually was shipped years ago but when the vulnerability was discovered that is when it started getting exploited.

So the people who stayed up to date never had the vulnerability. It was actually the publishing, the CVE that led to the mass exploitation of it. It was so wonderful to validate that wonderful intuition in the research. So something that we need automation to do and I think it is something that if developers aren’t doing it in the better universe security teams will be helping create better tools to make that possible so developers don’t need to worry about it.

[0:35:27.6] Guy Podjarny: Perfect, very concrete and I couldn’t agree more. So Gene, thanks a lot for having you on this, for coming here to join us on the show. A pleasure as expected.

[0:35:36.2] Gene Kim: It’s great to be here, wonderful. I’m very much looking forward for our next hang out.

[0:35:39.9] Guy Podjarny: Indeed and thanks everybody for tuning in and I hope you join us for the next one.

[END OF INTERVIEW]