Security Champions, Assemble!

Guy Podjarny: Hello. Thanks for tuning back in to The Secure Developer. Today, we’re going to try a different format of an episode. Over the years, I’ve had the pleasure of having various guests that have tried, sometimes succeeded, sometimes failed, and definitely always learned to embed security with developers. They all had maybe that similar mission, but they have different learnings along the way and they tried different approaches.

Oftentimes, different guests might mention the same practice or similar practices, but again, they have a different perspective on it. And they might have different experiences to share. What we’re going to try and do here is combine different perspectives on the same practice into a packaged episode, so that you can learn about that practice and get the different flavors of it, to see what works best for you.

For this first format of this episode, we’re going to focus on security champions. We’re going to hear four different perspectives on this practice. Security champions, at a high-level, deals with tapping specific individuals in the development team to act as security champions. Typically, those people get trained maybe a little bit more on security. They might be more interested in security and they, on one hand, act as security center points within their development teams. On the other hand, help communicate and connect to the security teams themselves.

We’re going to hear perspectives on security champions that range from being in small companies and also in different levels of maturity.

[EPISODE]

Guy Podjarny: First, we’re going to go back to episode 59, when I spoke to Steve White of Pivotal, that was subsequently acquired by VMware. He shares his experiences about security champions not just from his time as a chief security officer, but also working with other chief security officers that are Pivotal or VMware customers. Let’s hear Steve’s perspective.

Steve White: There’s a lot of different ways to get after that. I’m a big fan. I’ve seen a lot of success in what I have historically called the security champions program. The security champions tend to be — it’s like a way you help the development teams get invested and take ownership of security for their code, the stuff that they’re creating.

It’s difficult to try to train everyone all the time and get them really enthusiastic about security things. I think that’s not effective for most organizations; to try to get an entire development team jazzed about security. It’s like, pick same ideas, like I have on the security side as on the developer side, pick a person, one person who’s part of the team, who has some enthusiasm for security. Who has some understanding or some background and why it’s important and invest in them. Give them some additional training, delegate to them some responsibility, that perhaps the security team might have held within their arms previously.

Find a security champion and say, “Hey, we’re going to invest some training and we’re going to invest in some responsibility, that now because you’re on the team, we’re going to take off the reins some. We’re going to take off some controls, loosen the guardrails, and give them more flexibility within this operating framework, because you now have a security voice embedded within the team.”

That person, because they are a day-to-day functioning member of the team, can find incremental ways to help the team make small changes, or do small things a little better. I’m a big fan a kind of that natural growth of security awareness inside of a team.

The second part of that is, frankly, it’s to move security tooling and security validation earlier in their process. We all like to talk about this ‘shift left’ thing with security. Although if you ever look at the DevOps lifecycle, it’s a continuous loop. So how you shift left in the loop is beyond me. But nonetheless, we still use that terminology. For me, it’s simply, “Where do I provide security feedback to developers in a more timely fashion and in a way that’s consistent with the way they work?”

That’s always been one of the challenges is, in security, we’ll run our SaaS and our DaaS tools somewhere down in the pipeline. And all we do is send them a report of a thousand probabilities in their code, and we’re done. That’s just not how you build those bridges. It’s not how you build that awareness. Finding ways to give those developers actionable real-time feedback as close to the time they write the code as possible. Then making sure that when you’re providing security-related findings or feedback or what have you, that it really actually truly is actionable. It’s not fanciful. It’s not, “Well. We have this tendency in security to take the high ground.” It’s like, “All the vulnerabilities have to be gone. There can’t be a single line of misplaced code.” That’s not really where we need to be.

One other advice I’ve heard from others who I respected in this space is, “Pick a particular security problem you’re going to focus on, say, for a month and have all the development teams focus on that one class of vulnerability” — like SQL injection. We like to pick on that, because that’s a pretty bad one. All the teams focus on SQL injection this month, and so, we’re going to turn the knob up. We’re going to turn the noise level up on SQL injection this month, and we’re going to do everything we can. Then next month, we’ll look at something else. Engage the teams in that conversation about what it should be, how they should receive the feedback, what’s best for them. If you actually engage the developers in that conversation, I think you will ultimately get better results. Those are some of the keys, yeah.

Guy: Yeah. All great advice. No, I very much resonate with the whole shift left and say, “It’s not shift left, it’s top to bottom.” Go to some of the — from central governance to central controls, instead of bottom-up and then part of teams. This whole question about the practicality of the security champions program, I mean, lots of good things to say about it. One of the pushbacks is that organizations don’t always acknowledge that this developer, the person in the development team who’s now been added some form of authority, might have a different job. I mean, how do you recommend, or how do you see that working best for organizations that do that security champions? In terms of the role description for the security champion? Is it a percentage less work that they do on the product side? I mean, how does it relate to their day job?

Steve: Well, that’s a really interesting question, and I think is also, in some ways, a cultural question, right? If you are trying to measure output of individual developers down to the level where if they’ve spent two hours on security champion work, it would show up in some developer productivity metric — I think you need to question the cultural approach to that. Because frankly, output of development organizations should be focused minimally on the team output, right? There is very little external measure I would argue that is effective at measuring explicit individual developer productivity, and especially in organizations where you’re pairing, because now it’s like, “Well, am I measuring the pair?”

Frankly, I would first say if you’re trying to measure individual developer productivity at that level, I think you need to ask some tough questions about — is that really effective and is that really the culture you want to drive? If you take that up a level and you’re measuring productivity and impact, it’s really more about impact and feature flow and team metrics, right? Are they getting impactful things to the customers? Do the customers love what they deliver? Do they deliver frequently? Those kinds of metrics are what’s important. I would argue that you’re not going to see a big change in that by having one person spend some time on security champion types of duties.

The other thing is that once trained — so there is a training period. Then during that training period, there may be some reduction in flow, but the responsibilities of the security champion are really just to speak up during planning or speak up during design sessions to ask the questions of the team. “Did you think about this,” or “Did we include this?” Or, “Is this something that really should go through a deeper security review?” It doesn’t take a lot of work. It’s not a big investment of time. It’s like a focus, versus amount of effort kind of thing, so it shouldn’t take a lot of time.

Guy: Next, we’re going to go a little bit further back in time to episode 42, where I’ve had the pleasure of having Kate (Whalen) from The Guardian on the show. Kate came into security within The Guardian from the engineering group into security. She basically saw the security champions program, both from the dev side and from the security side, and actually found herself organizing and driving it a little bit.

So The Guardian’s security champions and general security practices have evolved a fair bit since this episode, but I still feel this is a very valuable perspective.

Kate Whalen: Before I even joined The Guardian, we always had a system of “Security Champions” or “Security Agents.” We switch between the two names. Trying to get developers who are interested in security, or just want to learn more about it to come to semi-regular meetings — where we might try and learn something together. Or run through one of the small training games that various websites have, or maybe use it as a bit of a knowledge sharing opportunity. That’s been really good as a way of ensuring somewhere that people can ask for help, or talk about things that might happen to them.

It’s also a really good forum to discuss security incidents, or potential security incidents. If someone’s noticed something strange going on, then they might mention it at that type of meeting. And then it drives a bit of discussion, and then you might have someone ask a question about a particular vulnerability and then someone else explains it. That’s a really good way of doing communication and engagement.

Guy: How does it logistically operate, this group? You mentioned sharing and meeting, how often does this group meet? What are the rough ratios of security agents to developers?

Kate: I’m normally the one that organizes it. If I’m being organized, it’s once a month. It might just be like myself from the InfoSec team there, and then hopefully developers from our different development teams. A mix of people from different seniority levels and different projects, so hopefully covering most of the areas. Which is good because then if we have an announcement to make, or a bit of guidance or advice to give out, then you can encourage them to share on with their teams.

On top of that, you might also want to do additional sessions. I’m running a Halloween session, so I can share all of the scary stories that have happened over the last year.

Guy: That’s a great one. Do you also use — you talked a lot about these people coming along and sharing within the group, problems, and sharing experiences. Do they also serve as good advocates, or as extensions of you, inside the different teams? Do you have any learnings about what did and didn’t work in trying to achieve that?

Kate: Yeah. I think something that works quite well is having smaller focus sessions, so not necessarily having too many people along, because if you’ve got lots of people in the room, it tends to make some people less likely to speak up, or more worried to admit that they don’t know something, or to ask questions.

Particularly with new developers, I like to try and have them for a one on one or maybe two on one intro session to explain how we approach security at The Guardian. And the fact that there is a shared responsibility and that they should never feel bad about asking questions about it, even if it’s just saying “Is this quite right?”

Then it’s a good time to also talk to people about how to look after passwords and how to set a strong password, and why multi-factor authentication is amazing, because with our developers, I imagine like most places, our developers have got quite privileged access. So they can run pretty much anything on their laptops and they can access an awful lot of systems. You really, really, really want to make sure that they understand how not to get hacked.

Guy: For the next one, we’re going to go even further back in time to episode 24, where we spoke to Omer Levi Hevroni, which at the time was a maven — which is a security champion in their local program at Soluto, which is a child company of Asurion.

We actually have the pleasure of having Omer now work on application security at Snyk and is doing a great job. Let’s hear his perspectives on being a security champion at Asurion back then.

Omer Levi Hevroni: This is what I started with. I started at Soluto as a developer. I think it was two years ago. Yeah, it was more than two years ago. Asurion started an initiative called — it comes from assurance security. They understood that they have more and more developers all around the world and they can no longer maintain security from sitting in Nashville, or in San Mateo, or wherever they’re sitting, influencing the entire R&D all over the globe.

They started a program called ‘Security Maintenance’, which is basically what other companies called their security champions. It’s basically the same thing. Which is, instead of going and hiring external security experts in each R&D and started to embed them into the R&D, they choose developers from all around the globe, give them security education.

For example, I did a [inaudible 00:14:46] pen testing course. It was one week of a course and then a test. We did this course and we started to work together as mavens, we are now a group. We have a Slack channel, we are talking a lot. This is how I get into all this security AppSec world. Currently, here at Tel Aviv we have two events and we started to add more, because we understand we need more. This is how it all started.

We are now interacting. There is someone in the assurance security team that is responsible on the mavens, so we are communicating a lot with him for guidance and help and to learn more of what we want. Here at Tel Aviv, all other people have a lot of freedom to add things and try to think about things that can improve the security here.

Guy: Yeah, do the mavens, typically like yourself and the other maven, the names catchy, just roll with it —

Omer: You know that the source of the word maven is English actually.

Guy: Oh, I did not. No.

Omer: Yeah, it comes from the word ‘and within’, which is Hebrew, no. It’s really funny.

Guy: No, I didn’t know that, I guess, because it’s history. Yet another reason to have two mavens in there, in the Tel Aviv team. Do the mavens typically have a part of their day job, like is your day job security? Or is your day job a bunch of other things and security and you’re just passionate and therefore, find yourself doing security? I guess, how does that similar or different to other mavens across Asurion?

Omer: That’s a good question. One of the main challenges of being a maven and I’ve seen that with the entire mavens is that you need to be a maven beside your regular day job. I very fast understood that this is what I want to do full-time. I think not a lot of mavens who are doing it full-time. I did it full-time for one and a half years. Then I joined the DevOps team, so now I do a part of a regular DevOps task, like building Kubernetes clusters or stuff like that and security task can start, try to combine them, but it’s definitely very, very hard to find the right spot between doing your regular job and doing the maven stuff, because it’s not easy. You have tasks that you need to complete and you need to somehow think about security things. This is a challenge of being a maven. I had lucky to not have it.

Guy: Yeah. I guess, with the versatility also of the program, where it’s relevant then people would have that as a full-time job versus not. Very cool. I think the maven program. I want to spend a couple more minutes on that one still, just because it sounds like it’s working well and I think it’s a program that many people try to do. Can you tell a little bit about how, like, is knowledge shared within maven? You mentioned the Slack channel internally. What other good tooling, or methodologies work to share information between mavens?

Omer: We have a few virtual meetings, one or two one in a year. We also try to meet in a big conference. For example, in two months, some of us are going to be at the AppSec California. I’m going to give a talk there. We will be there. We will talk together and hang out. Last year, we did an internal con for all the mavens, like AppSec California and we might do it again this year.

Or for example, do it at DevCon Blackhat USA. This is an example to where we can meet, share ideas and learn from what others are doing. Sharing is hard, but we try to do it as much as we can. And different mavens have different challenges. This is also interesting to see what they face, what we are facing. There is someone from assurance security team who is responsible for all the mavens. He also helped us to say, okay, they face the issue you are facing. Maybe if you talk with them. We do the coordination.

Guy: Yeah. Yeah, it makes sense. Basically, like a component of it is just natural social interactions. You’re on the Slack meeting. You meet up. You naturally go to some of the same events. Then you meet up and share knowledge there and some of it is a little bit more coordinated by the person in the central security team, whose job it is to make this program successful.

Omer: Part of his job. Yeah.

Guy: Or part of his job.

Omer: Yeah. My job is responsible for application security, so it makes sense that mavens is part of his job.

Guy: It’s one of his primary tools, I guess, or primary means of succeeding that.

Guy: Last and very much not least, we’re going to talk to Yash from Twilio. Yash was on episode 66, a bit more recent. They run a very sophisticated, in my opinion, or at least very thought through security champions program, with great incentives and recognition for the champions in that program. Let’s hear Yash describe the program.

Yash Kosaraju: Product security specifically works very closely with engineering. We look at it as a partnership of sorts. We really don’t want security to be a roadblock, or a hammer where we say, “Hey, do this.” It’s more of a partnership where we work with engineering and come to collaborative decisions saying, “This is the right way forward for the company.”

Security champions program is something that helps us do that on a large scale. The idea of the program is a engineer on every engineering team is their dedicated security champion. Then there’s one security engineer who’s their dedicated security partner. This security partner and champion meet on a regular basis and then talk through what the engineering team is doing, what the security team is doing, what reviews they need. What help security can provide, do things securely and make sure that they’re building robust and secure solutions.

What helps in this format is all the reviews are scheduled on a timely basis. Otherwise, the other side of it is engineers build the whole product. Once they’re ready to deploy, somebody says, “Hey, let’s do a security review on their blog,” for you don’t know how many days, or weeks it can be based on the project size. With the champions program, these reviews happen in tandem as the engineering team is building. And we can give real-time feedback and it’s easy for them to make changes. There’s very less friction and we don’t affect their timelines. At the same time, we help them do things the right way.

Guy: Yeah, it’s secure. I’m curious a little bit about some ratios here. You mentioned that the security champion is aligned to someone in the product security team. What’s the rough ratio you aim for in terms of product security? How many security champions does every product security person work with?

Yash: That’s something that I’ve been debating with myself for a long time. What’s the appropriate ratio of security engineers to engineers within the company? I don’t have a good answer. What we’re trying to do is instead of talking about the numbers, talk about capabilities within the product security team and also in terms of champions, look at how big or significant the product, or product group that these insurance teams belong to. And then divide it based on that. It would be an engineer who has five products under them, or products under them based on the size of the products.

Guy: Yeah, that makes a lot of sense. It’s still the mission orientation that a product security person is still more mindful and aware of the substance and the surrounding context, when a request comes in, but indeed some products are big, some are small, some require — some are more hairy from a security perspective than others.

The security champions within the teams, is this an official role? Do their managers expect them to spend some 20% of their time on security? Or is it more just general interest? I mean, how do they manage that extra responsibility?

Yash: It is an official role. The percent of time that they spend on security is team specific. We don’t mandate a percentage of time that the champions spend doing security stuff. What we expect them to do is meet with their security partners regularly, talk about their team security posture, and work with their partner to improve their product security posture over time.

Guy: Yeah, that’s awesome. Are the security champions themselves a community within Twilio? Do they also engage with one another, or is there specific education for them, or is it more about the one-to-one relationships?

Yash: It’s both. We do have one-to-one relations. This year, we wanted to expand on that security champions as a community, do happy hours, and a lot of other social events, but COVID-19 happened. Everyone’s stuck at home, so that isn’t happening.

The other thing you mentioned was training. Within the champions program, we have a subprogram called the ‘Advanced Champions Program’, where we have red, blue and purple categories, which have challenges, requirements to do X, Y and Z things, which are aimed at offensive for the red, defensive for blue and purple, we modeled it as a cloud security subspace within the program.

As they do these, we give them more perks and responsibilities that otherwise they wouldn’t have had. The ideology behind it is you take these trainings, you do these challenges, you level up your security knowledge, and then we as a security team will trust you to make these decisions — which now you have enough knowledge to make the right decision for.

Guy: Yeah, that sounds awesome. That’s a great thing to do. I must imagine that these things are acknowledged as a skill set and a learning by different management teams might help them as they move up the ranks, or as they might want to move into security, in some cases.

Yash: Yup. That is true. This ties back into how much time a manager expects their champion to spend doing security work. Once they start their journey on these different tracks, they generally put it into their sprints and work on it as a story ticket, like something as a general work item, rather than on the side. Most of the managers are happy that people on their team are interested in this and spending time learning security.

Guy Podjarny: That’s it for the security champions program. I hope you enjoyed this format and would love to hear from you about what we should be doing differently about it. If you can email us at thesecuredeveloper@snyk.io.