Celebrating Success

[0:00:51.0] Guy Podjarny: Hello everyone and thanks for tuning back into The Secure Developer. Today, we’re going to have another of our mix episodes, these are episodes where we look to a sequence of different conversations that I’ve had the pleasure of having over the years and pick a theme or topic or a practice and just sort of get multiple perspectives on that topic to help share them as one package.

Today, we’ll focus on ‘Celebrating Success’. It’s been pretty well accepted by now I think that we’re all humans and that if you want to incentivize developers to do well in terms of security, then really, you should recognize them, not just when they are doing something wrong and missed it which is the default mode in security of being (inaudible) but rather, when they are doing something right, you want to celebrate that success and recognize it.

[0:01:43.0] Guy Podjarny: To start us off here, we’re actually going to go all the way to the beginning of this podcast where Kyle Randolph – who at the time was at Optimizely – talked about how they celebrate success and really have been doing it quite well for or so years ago.

[0:01:49.0] Guy Podjarny: A lot of the conversations we had there was about celebrating successes which is always tricky in security because not getting hacked is not sufficient because hopefully, that’s the normal course of operation, otherwise you’ve got other problems.

Did you consider that, are there ways in which you somehow sort of celebrate either an investment or a success in security?

[0:02:19.2] Kyle Randolph: Yeah, the company has different ways to recognize success, and we kind of tap all those for security and make up some of our own as well. Company-wide, there’s a way for one employee to thank another employee for something, and that’s very public and recognized each week at the company all-hands meeting.

Using those things to thank people who have even just fixed one security bug, anything with security, or they had a good security question, to give them thanks there. Stepping that up a bit more, in quarterly retrospectives: what went well, what didn’t go well.

Security guy, you can always just say, “Well, security wasn’t as good as it could’ve been.” But more pragmatically, talking about who did do awesome security, like, “This team is great! They fixed all the security bugs they said they would.” Or, “Everything they made this quarter they had a design review before they built it,” and so we knew security was baked in at that very early phase.

Going further than that, giving out T-shirts. We actually give out T-shirts that say, “Security Hero” on them. This is more exclusive, so it makes people want to step it up and really go above and beyond to make a security contribution. Maybe you eliminated cross-site scripting, you built it into the framework your team uses. Then that would warrant a Security Hero T-shirt. And then you get recognized in front of the whole company.

[0:03:33.0] Guy Podjarny: Nice, that one you need a slightly higher bar because if you start giving them out too much, then you actually lose the prestige of having that shirt.

[0:03:41.1] Kyle Randolph: Yeah, so many different tiers of reward there, in recognition.

[0:03:44.6] Guy Podjarny: I guess are there are specific examples, you gave a little bit with the cross-site scripting framework or with sort of the team that is fixing this, but do you have a specific story that you feel kind of manifests, really demonstrates how they succeeded? You know, converts or the like?

[0:04:01.4] Kyle Randolph: Yeah, cross-site scripting and cross-site request forgery. In both of those cases, we had an engineer who may have had a bug or two assigned, and they’re like, “I’ve seen this before. I don’t want to see this again.” And so they just went out or their way to really say, “Here’s how we’re going to fix this comprehensively.”

In one case, this one team with cross-site scripting, they didn’t even talk to me. Within their own team, a guy made a presentation like, “Here’s why cross-site scripting’s bad. Here’s how we’re going to make sure our team never has it any more.” And then they’re like, “By the way, we did this, we banned cross-site scripting.”

That was awesome, and I don’t even have to get involved. They just took it upon themselves to do it. That’s even better than roping the security guy in to tell you what to do. But to decide on your own to do it?

[0:04:45.7] Guy Podjarny: That’s great. That’s definitely reaping the fruits of your labor, right? When you sort of planted those seeds and they grow inside. This is all process and people and how do you celebrate them.

[0:04:56.6] Guy Podjarny: Next, we’re going to bump forward a bunch of episodes but still go a fair ways back to episode number 23 where Zach Powers who is the Chief Security Officer at One Medical talks about their practices at One Medical. They do a lot to really recognize developers when they do well on security and they invest in it a fair bit and one of my favorite phrases from this conversation was the idea of hoodie driven security. Let’s have a listen of what that means.

[0:05:25.7] Guy Podjarny: What do you do about incentives? It’s one of the challenges that oftentimes comes up is developers are not in the daily use, so they’re not incentivized. They’re there to build new functionality, and if they don’t deliver a feature, somebody comes knocking. But if they built a security flaw that gets discovered a few weeks later, maybe it’s the security team that gets thrown under the bus. Hopefully, nobody gets thrown under the bus, and it’s all positive, but how do you incentivize or encourage the dev team to indeed embrace this ownership amongst all the many others they have?

[0:05:57.6] Zach Powers: It’s a good question. At most companies, to be honest, there is no positive incentive other than the finger-pointing. At Salesforce we definitely tried a range of positive incentives, and I’ve carried that on to One Medical. Part of it is simply high-fiving somebody for doing the right thing. Part of it might be, everybody loves swag. If you want an awesome hoodie, security teams know all about awesome hoodies. We’ve done things for individuals who continually do good security practices, make great decisions, have them do a rotation or work on what recourse coalitions.

Have them work on a special project to step out of their day to day routine. Most engineers love doing that because they don’t like looking at the same section of code all day. In a coalition, we get a cross-functional group together and say, “We’ve got a really hard problem to tackle, and we want you to help us tackle this problem.” So, giving new opportunities is a good way to do that. We’ve done things as silly as teach lockpicking classes. Things like that. Just finding something fun and memorable to positively recognize in a public fashion that this engineer is rocking it with security, and here’s why and give examples.

But then giving them something fun and meaningful in return. It does go a long way. The security team at One Medical often invites software engineers to happy hours. Where we’re not just having a drink, we grab a whiteboard, and we discuss things, and when it is talked about or experienced in a more positive manner, I do believe it goes a long way. People will sometimes say it’s a security champions program, and some of those do work for sure, but I would say this is more just publicly and positively recognizing when people do good security behaviors.

A little bit of swag goes a long way. Some really nice socks, a coffee mug, things like that go a long way. But I don’t see it happen at many companies, to be honest. If you slow your work down to produce better code, at many companies, you’re penalized for that. That’s definitely not the case here. You need some executive alignment to be this positive about it. At One Medical, at companies like Salesforce, I could name a bunch of them here in the Bay Area, we have a common philosophy that it is better to produce quality code than to have to go back and have to fix it later on. Because it usually takes longer, it usually involves some angry customers. It’s way more thoughtful to do it upfront.

[0:08:36.6] Guy Podjarny: I love pretty much everything about that model. You gave a whole bunch of examples, and none of them included bonuses or financial motivations, because I don’t think that’s really what sets the– You’ve got to have these hoodie driven security incentives or swag.

[0:08:51.4] Zach Powers: It goes way better. At other companies, we’ve tried this in experiments, and the cash bonuses don’t really work that well.

[0:08:58.8] Guy Podjarny: They create almost a cognitive dissonance, where people think that they’re doing it just for the cash. If you’re giving them something fun, clearly they’re not doing it for that, but they’re still enjoying it, and it still has the positive association that comes with it.

[0:09:11.7] Zach Powers: ZP: It’s key though, and change it up, so you don’t always give the same hoodie or the same sticker or what not, the same T-shirt. Change it up. Because if people expect that, “If I do this I’m going to get this thing,” it cheapens the experience. There’s somewhat of an unexpected surprise. They don’t know when they’re going to be rewarded, but they realize that there’s a culture of recognition.

When the software engineering team at One Medical gets together, every couple weeks everybody gets together for an all hands. We will sit down with the security team and call out and publicly thank people for very specific actions. They’re not asking us to do that, but it definitely goes a long way, and it promotes a cultural momentum that these are good things to do and that it is OK to take the time to produce better quality code. I’m an evangelist about that.

I think empowering software engineers and letting them make decisions, but also recognizing them for their good decisions and good work produces way better security than not.

[0:10:12.2] Guy Podjarny: Yeah. Fully agreed. I love that. I also feel like the teams that have the best handle on this indeed do this. I’ve had the PagerDuty security team come on the show, and they were talking about Indeed, awards that they give out, and they’re not monetary they’re just recognition. Sometimes, I forget who mentioned this, but somebody talked about giving explicit security training elements to it like send someone to a certified hacker CEH type of course, so that they can have something to add to their resume in terms of formal, “You’ve invested in it. We can develop those skills.”

Because at the end of the day that helps your career as well in the long run, but fundamentally it’s all around getting that positive sentiment around it. The world of security uses the term “Shame” a lot and uses the term “Pride” very little, and we need more of that pride in it. We talked a lot about the software engineering background within the security team. You have the engineering team, and you train them up, you give them these positive recognition and hoodies to drive the right behavior.

[0:11:09.5] Guy Podjarny: Next up, we’re really going to have almost a statement from Siren Hofvander who at that time was at Cybercom, just talking a little bit about how can you celebrate something that didn’t happen.

[0:11:22.9] Guy Podjarny: I think I understand and relate to the notion of celebrating the work of developers for the good security they do. Let’s hone in on that and then later come back out and talk about the tips and tricks on how do you prioritize. Give us some examples of how have you celebrated success. You highlighted the session handling, or the likes. Give us a few other examples of this positive mindset.

[0:11:45.5] Siren Hofvander: Well, we have “Cake for no reason” day. We’ve had “Cake for all the things that didn’t happen,” because the system didn’t fall over so let’s celebrate that.

[0:11:54.3] Guy Podjarny: Okay, so we got three great perspectives but there is two great ones still to come. Next, we’ll have Mike Hanley, who came from Duo into Cisco and running their security practices internally and they’ve automated and again kind of built into their systems some nice recognition systems, let’s have a listen.

[0:12:17.6] Guy Podjarny: So when you work with these development teams or you work with other people in the organization, how do you keep incentivizing them or how do you keep them engaged in this desire to be a part of your security team and feel good about it?

[0:12:30.1] Mike Hanley: The best way to do this and the cheapest way to do it is really to focus on reliably identifying, surfacing, and celebrating good security behaviors and good security hygiene. We have a friendly neighborhood security bot that reaches out to people on chat and says everything from, “Thanks for updating your phone to the latest release. You are one of the first few people to do that. You are doing your part to keep us secure.”

To providing in some cases top bonuses to people who raise their hand and say you know, “Hey, there’s a problem over here and I want to be in front of it from a security standpoint.” So, it’s much cheaper in the long run for you to focus on identifying and reinforcing what those good behaviors are and making it very visible what the desired behaviors are compared to going around and chasing people with a stick. Nobody wants to be chased with that, but they would much rather hear that they did a great job and have examples of what the organization expects and desires from them behavior-wise.

[0:13:23.1] Guy Podjarny: That’s awesome. Do you think actually is that bot, you know, are these things you’ve written about? Or sort of you have published anywhere? It feels like everybody should be using the —

[0:13:29.8] Mike Hanley: Yeah, yeah. In fact, I should probably write a blog post on this. But I’d say, if you ever see a Duo employee’s laptop in an airport or at a conference, one thing you’ll note on there is, you know, people love stickers. So even little things like the first folks to report a phishing email that come in to the security team will sometimes give them a special kind of edition sticker that says they were the first to report a phishing or attack campaign on the organization.

Again, just to recognize the importance of them executing on their day-to-day security responsibilities and helping them realize that those things are not trivial, and in fact, we love hearing from people when they have questions and comments. It shows that the extended sensor system is working well and people feel empowered to execute on their security responsibilities and it’s their contribution that matters.

[0:14:13.0] Guy Podjarny: And last but very much not the least, we’re going to go to episode 33, where I had two great guest from Segment, Leif and Eric, talking about their security recognition and in general that episode talks about developer training and a lot about engagement with developers but this is just the snippet to talk about how they recognize success and celebrate it.

[0:14:37.4] Guy Podjarny: So on that note, we talked about education and we talked about engagement with the team, when you talk about this notion of positive – I know you’ve done some things to celebrate successes, can you give us a couple of examples of when somebody does a good thing around security? Do they have some stickers? I remember some mention of a crown.

[0:14:56.2] Eric Ellett: Yeah, the stickers come out as part of our training. When you complete the training we have this hacker man sticker, that is this online meme that we use quite a bit in the training itself so that people can show “I did the training.” Another thing that we’ve started, and I’m presenting at OWASP next month at Uber, which is a leaderboard. It’s effectively this gamification platform that we built that celebrates those small wins that people have.

When people come to you and say, “I think I noticed this issue or I noticed maybe some PII in this log.” How do we recognize those small wins? This leaderboard is basically this UI. I really got enticed by Halo 2 and the notion of matchmaking back when I was in high school or middle school, and how people can be ranked. Basically what happens is when you do these small things we’ll recognize you and you gain experience points.

Everyone starts off at level 1 and you’ll get, depending on the type of thing that they’ve done, you’ll get 15 experience points or 25 experience points. When you get 100, you go to level two. It posts all the great things that people do every Friday in our security Slack channel so that, not just the people that were part of that interaction like the security team and the developer, but even the VP or the CTO or people that are higher up can say “This individual has done all of these great things this past week or this past month,” you’ll see that recognition happening in the security channel overall.

[0:16:30.6] Guy Podjarny: That’s awesome. What types of actions do they get points for?

[0:16:33.3] Eric Ellett: So we have a vulnerability management program here, like most people, and we rate our vulnerabilities. If you find a P1, we’ll give you 100 points. If you go out and find these things it’s 100, because P1 is the worst type of vulnerability. If you fix it, we give you 50. Because fixing could have been because we assigned you a vulnerability to fix. But people that are out there proactively finding these things, we give you 100 and that’s an automatic level up or they just –

We have a catch-all like going above and beyond for security. If they ask someone to badge in, this isn’t even just for engineers, our salespeople are on this board. That’s typically because they’ve asked someone to batch in and that was maybe trying to tailgate. Another thing that we’ve done with this is also open it up to other people, so it’s not just security giving these points.

We’re not always around to watch people tailgate, so we’ve had other people that are not security engineers or on the security team submit these points through the Slack command that we have and so we’re just really trying to build a culture of people recognizing each other for doing awesome security things.

[0:17:35.3] Guy Podjarny: That’s awesome. I still think stickers are good as well, even if they’re just from the training. I think you also showcase that it’s important. But I very much loved the leaderboard and those results.

[END OF INTERVIEW]

[0:17:47.6] Guy Podjarny: That’s it. I hope you’ve enjoyed this conversation about celebrating success and feel like celebrating some success on your own. Once again, if you enjoyed this format of episodes and you find them useful, please let us know at thesecuredeveloper@snyk.io and if you don’t like it or if you want to see a different format for it, we’d love to hear from you as well. Thanks and I hope you join us for the next one.

[END]