Leveling Up Security in Big Organizations

[00:01:29] Guy Podjarny: Hello, everyone. Welcome back to The Secure Developer. Thanks for tuning in. Today, we have with us someone who has kind of this journey from ops to security and now tackles a pretty sort of sizable amount of security as Chief Security Officer at Medallia. We have Geoff Kershner. Thanks for coming onto the show, Geoff.

[00:01:45] Geoff Kershner: Thanks for having me. I’m looking forward to it.

[00:01:48] Guy Podjarny: Geoff, before we begin, tell us a little bit about yourself, like what is it that you do and what was the journey that you took into security and into specifically your current role?

[00:01:57] Geoff Kershner: Yeah. Thanks for asking. I’ve been in high technology for about 25 years now. I came to the Bay Area in California in 1995 and started working at Sun Microsystems and went to eBay and PayPal and then a startup and then a consulting firm and now landed at Medallia as the Chief Security Officer. I’ve had various roles. I started new technology operations and gravitated in and out of security throughout my career. It’s always been a fascinating area for me.

Actually, in my last consulting company prior to joining Medallia, we focused on high-growth and high-scale companies and helped companies that were dealing with massive problems with just growing its scale, and security always came up as a topic too. I was the most experienced in our firm around security, so we decided to start up a security practice there. I got engaged with Medallia. I was part of that and I’ve been really excited to help them build a strong security program there.

[00:02:55] Guy Podjarny: You got lured into the dark side. I don’t know which one of them is the dark side. You got lured into actually operating and running that from the consulting side. Has that been very different? I mean, how’s the shift from consulting to running the team?

[00:03:10] Geoff Kershner: Well, the shift from – I mean, obviously you have a bit more ownership when you’re actually running the team and you have known the day-to-day responsibilities of it. It wasn’t a hardship because everybody in our consulting firm was executives or leading large teams and stuff before in an operational capacity, so we weren’t born consultants. But the biggest thing that I like about Medallia when I came there is they had a security mindset from the beginning. It’s not like we were introducing a new topic there.

So that is something that I was actually looking forward to in a company. Somebody that took it seriously, understands how important it is to our customers that the privacy and security is an element that you have to be an element of how you run the business.

[00:03:52] Guy Podjarny: It sounds like you came into lead a team but the mindset was there.

[00:03:58] Geoff Kershner: That’s right.

[00:03:57] Guy Podjarny: Let’s maybe veer into the organization. So what does it mean? What does your organization look like, the chief security officer organization? Maybe like what was the journey? You’ve been there for about three or four years. How did that change as you grew?

[00:04:13] Geoff Kershner: When I joined the company, the security team had started. I think it was probably two or three years prior to that where they really had it. They had security people in the company but they didn’t really have a team until about two years prior to me joining, and it was really kind of just getting on its feet and developing. The way that I structured it is really it’s a pretty standard way to do it. So I have a risk and compliance department that’s responsible for all our compliance programs. We have security operations in-house, so a 24×7 security shop, and we have a large team that we call product or platform security that is responsible for all the aspects of SDLC, doing our penetration testing to [inaudible 00:04:52] many tools that are integrated into our platform.

My kind of philosophy on building the team around security is when you have a siloed security team, it doesn’t work. When people look at you as an organization that makes all the rules and comes and sort of acts like a policing force or something like that, it has to be something that pervades the entire organization. But we also have the security engineering team that is responsible for building the security features that our customers use, so it teaches around encryption, SSO, all that kind of stuff they build and support. They actually report – They are peers of mine that report in kind of dotted line into me and report into our CTO, but I think that’s great that they have responsibility for acting like engineers.

To take it a step further, we also have what’s called a security champions program. We put that in place a couple years ago. We have folks in every single engineering organization, sometimes more than one person if it’s a larger team. But these are people that we went through a program. We actually get them a Stanford credential around security. So it’s pretty enticing to them because security is interesting to a lot of people. We don’t force anybody to do it. We get volunteers and they like it. They like to have that on their resume. It’s a pretty powerful thing, but this gives us an opportunity to not only have to kind of go to people that are thinking about security but kind of train the trainer approach, and that’s how we really think about scaling the organization.

If there’s a vulnerability in a certain part of the platform, we know we have somebody that we can go to. But also, it really helps them think about security from the very beginning when they’re building their products.

[00:06:34] Guy Podjarny: I love the champions program and the specific credential that maybe digging into that a little bit. This is your setup. There’s like one or more champions per team. What does a team mean in this context or what would you say is roughly security champions?

[00:06:48] Geoff Kershner: A development team. Sorry to interrupt you. Like a development. Our engineering and development teams are aligned around our products. We have dozens of products on our platform and we have engineering teams that are aligned towards those engineering and product teams. So if there is a team around reporting, a team around APIs, a team around mobile team, a team around all those different things, each one of those will have a security champion or a champion who can work with and develop through this program.

[00:07:16] Guy Podjarny: Is it safe to say it’s roughly like a 1:6, 1:10 type ratio over the champions to it or even more?

[00:07:23] Geoff Kershner: Yes, depending on the team size. A team of 10 might have 1, a team of 20 might have 2, a team – That kind of thing. The other thing I like about it too is I don’t think the right thing for any organization to do is to build a security empire either. I’m not about empire building. I’m about having a lien and very effective and efficient team, the best talent that I can find on my team that obviously works on projects and works on those various things, but I want a team that is kind of train the trainer and having that approach of everybody thinking about security, not just my team.

[00:08:01] Guy Podjarny: For those people that are security champions, is it acknowledged beyond the credential? Is that an acknowledged part of their job today? Do they have another responsibility, some expectation that some percentage of their time would be spent on supporting the other developers in security work? How does it work on the CTO side?

[00:08:19] Geoff Kershner: It does. The expectation is that you’ll have a certain percentage of time. Now, it varies by team. Typically, it’s going to be anywhere from 10 to 20 on the low end to up to 30 to 40. In a lot of cases, some of the teams actually have a backlog or security work in their queue, so it can be more than that. But, yeah, the expectation is a percentage of time through that program.

Now, we typically do it for six months or a year and kind of refresh it. Sometimes, people volunteer and they want to do it again, which is great. Sometimes, people are just like, “Okay. Now that I got that kind of training, I’ll move on, and somebody else will take the turn.” So it kind of varies team by team.

[00:08:57] Guy Podjarny: I think that’s actually like an excellent idea. In part is because it keeps them fresh and these people always know more. But also, while you might have one champion on this 20-person team, over time there might actually be 3 people out of those 10 that have been qualified and have kind of walked a mile in the security shoes and will have sort of this subject matter knowledge but also the empathy for the security needs.

[00:09:23] Geoff Kershner: Yup. My philosophy too is like security is going to fail at a company if it feels forced. You really have to have people understand and appreciate and own, and that’s why we do a lot of top-down work around security. There has to be. I mean, you have to deliver results and manage risk and all that. But at the same time, if security is going to be successful at a company, I strongly believe that you have to get people to believe it and adopt it and make it have the mindset where people are really thinking about it and want to do it.

We’re fortunate in time that, especially in engineering team, cybersecurity is a very, very hot item, so it definitely doesn’t hurt to have some skill set on your resume [inaudible 00:10:05]. It makes you very marketable. Not that we want our people to be leaving and going somewhere else, but it’s a very high in demand skill set.

[00:10:14] Guy Podjarny: It’s true value because it’s valuable for you in the company, and maybe unfortunately it’s valuable when someone tries to hire them out of your company. But in the meantime, it’s [inaudible 00:10:24] for them to sign up. I love that security champions program. Thinking about that lien team that you run where you hire great people, what is it that you look for in a hire within your team?

[00:10:37] Geoff Kershner: We want people that have strong security background and have great technical kind of acumen too or compliance kind of acumen and business acumen. I think having like – For the more technical roles like our security operations team and our product security team. I forgot to mention. We also do our own internal red teaming. We do our own kind of ethical hacking and stuff like that, so that’s one of the functions that we do. But for those teams, I think it’s important for them to have kind of a broad range and skill set. Not necessarily expertise in one area or another but having folks that can really understand the technology stacks that we’re working on and figure out how tools and systems and stuff can integrate with those.

I think a lot of the roles on the risk and compliance side is very customer-facing, so we do a lot of work at producing collateral around security. We talk to customers all the time. We have customers in the Fortune 50 in various different verticals from banking to telcos, to government, to healthcare, and stuff like that, so very highly regulated and security-conscious customers, and my team talks to them often, and we have to convince them that our security is great. For those folks, it’s very important that they have some business acumen and they not only understand technology and compliance and security but have the acumen to be able to talk about it to customers and explain our features and the value that we can add as a security team.

[00:12:11] Guy Podjarny: Got it. I mean, this makes a lot of sense and I think those are the engineers that help secure your own platform, and therefore they communicate with customers to help build the trust that their data and trusting you is legitimate that they are in safe hands. Separate from the engineering team that I think you mentioned reports to the CTO that builds specific security capabilities and crypto capability or the other, although I imagine that that team reporting to you also deeply understands what has been built, so they can communicate to that.

[00:12:43] Geoff Kershner: Sure. We’re connected at the hip with those teams but just the fact that they are more engaged in the engineering process and the build and all that. It makes some kind of move at a different pace and have priorities, but they’re very close aligned with my team.

[00:13:00] Guy Podjarny: A different structure.

[00:13:01] Geoff Kershner: I do have a lot of influence over the product. We’re in that for our security, as you can imagine.

[00:13:06] Guy Podjarny: How important is it to you to have a hire into your team know how to code, actually make script code? How significant [inaudible 00:13:15]?

[00:13:16] Geoff Kershner: It is important for a couple reasons. So a lot of the tools and services that we implement, we’re also responsible for doing the deployment of those, deployment and support. On the other hand too, I’ll take our product security team as an example. They’re the ones that work on our SDLC and manage our vulnerability programs. For them, we do our own testing. We also hire a third parties do a lot of our testing. It’s very important for them to understand the code and understand the technology behind it because we’re the ones that will triage and look at vulnerabilities. If we’re going to hand something off and find a vulnerability, whether it’s through our own testing or through a third party and hand it off to an engineering team, we want to understand the context and offer solutions to them. Hey, you should do it this way or this way. Here’s a recommendation for you.

If you don’t have that kind of skill set on your security team, you’re going to lose some trust with engineering teams and IT teams and everybody. You have to have that technical acumen to be able to work with those teams and work together with them on solutions.

[00:14:19] Guy Podjarny: I guess maybe drilling into kind of that product security team work and shifting a bit into the engineering sides. You mentioned the security champions program. That’s clearly one path of working with engineering. I imagine it’s primarily the product security team. How do they work with the engineering or, for instance, run vulnerabilities around tooling and maybe even specifically around ownership? Who owns what aspect of the work there?

[00:14:43 Geoff Kershner: Yeah, that’s a great question. When we think of ownership, I’ll kind of start there and unpack it the other way. When we think of ownership, we want system owners and people that are building the products, just like people building, supporting the server stacks and networks and all that to own the security like the same way that they own availability, the same way they own quality, the same way that they own the features. When you get that kind of ownership, if there’s a hack or an intrusion on your product or something that you built, I think it only makes sense that you’d feel some ownership and responsibility that you could [inaudible 00:15:20] something better.

With that regard, that’s kind of the mindset that we want to think. Having said that, the way that our team interacts with them is very much from the very beginning. We follow an agile process. We go through planning cycles and all that. From the very, very beginning, our team gets involved with designs. We’re dong designs, doing threat analysis on designs. When products are being developed, there’s a whole process where the teams can create triggers and stuff that automatically sends a signal to engage our security team in reviewing and helping them at the kind of review stage of before the product gets really deep in the development.

Then at the other various stages too, we work with the engineering teams to help integrate tools that will help them automate the process for doing things and finding vulnerabilities early on in the process and giving them the tools that allows them to automate them so that we don’t have to get up or we can find vulnerabilities early. They can do it as they’re building their products. So the ownership is really nice. We own the tools. We own the design. They own the products. Where the two meet are through our SDLC process.

[00:16:28] Guy Podjarny: Interesting. I think there’s a lot to unravel around that design aspect of it. You mentioned there’s a bunch of keywords or sort of signals that trigger engaging the security team. Is that an automated process? Is it literally some key words in an issue or is it more guidelines?

[00:16:45] Geoff Kershner: It’s semi-automated, so there’s product standards that we create. Through those product standards, it requires engineers and developers and product folks to fill out information about what they’re building and based on how they answer the thing. Is this a new service? Is this going to be a new type of data we’re interesting? Is it how we managed something like that? That will trigger or this is going to require some security review. There’s some kind of like self-answer Q&A, and it’s not perfect but it’s pretty good. It’s pretty good at catching most things, and that triggers an engagement from our team. That is a bit of a manual step, but that’s how it’s done today.

[00:17:23] Guy Podjarny: But if I echo this back, this tooling, this semi-automated process helps the development team know when to engage you and when they don’t, let’s assume, engage you. Therefore, you reduce the number of times they do need to engage with a smaller amount which is manageable or allows you to scale, and that is important to bring in that security expertise.

[00:17:46] Geoff Kershner: Yeah, that’s right. It’s a very collaborative process too because, like I said, we had this product security team that works on this too but we also have the security engineering team that builds a lot of future stuff. They’ll often also get engaged in helping some of the – If there’s a big thing that’s like data science and stuff where you’re doing that kind of work around there, that may require, “Okay, we really need to look at how we’re designing the security around that.” So it’s a very collaborative effort.

The thing that I really like about how — I wouldn’t say I’m going to take all the credit for this, because we have a lot of wonderful people on the team. But when I got there, it was very much of an us versus them kind of organization but it’s really flipped. People come to us for help which is great. People say, “Hey, we’re building this thing. Hey, we’re thinking about this new product. Hey, I don’t know if this seems like it might have some security issues.” It’s great that people are reaching out to us versus, “Oh, my God. Here comes the security team.”

[00:18:44] Guy Podjarny: Yeah, that’s great. I mean, I think that ability to collaborate in having the mutual love or mutual appreciation is really core for it. How is your product team structured towards the team? When you say, “They reach out to us,” is there like a partner, a designated product security person that is mapped to a bunch of development teams? Is it more of a pool?

[00:19:08 Geoff Kershner: Not really. Yes, we’re growing in scale and we do have folks that have more expertise in one area versus the other. We have tons of different products on our service. We have webinar sets. We have surveys. We have video. We have voice. There’s – Medallia, the context basically is a company that does customer experience and creates – It basically pulls – They use the word signals again, that signals from surveys to when you go in a website and you’re asked to answer some questions about this or that. That’s often us but there’s video, there’s voices, all these new facets that help our customers engage with their customers, and that’s what we’re trying to help. We’re trying to help pulling and give them rich information about those sources.

So there’s lots of different technologies. We’ve actually acquired quite a few companies in the last two years, so they come with their technology stacks and stuff like that, so it’s kind of impossible for one person to understand everything. We are getting a little bit more specialized in our services. But in general, like I said, it’s a relatively small team. There’s a way that they can reach out to the group. Then we meet regularly and triage and assign and say, “This one’s yours. This one’s yours,” and try to keep people around the areas where they’re most comfortable and most knowledgeable.

[00:20:23] Guy Podjarny: Got it. I think with the security champions program, you also have people that are right there to engage, so they might have their local kind of depth in the application, even if their security depth is not as deep but it’s substantial. A bit of a topic of choice for me recently is really the whole world of cloud security. How do you think of product security versus cloud security? You think about cloud platforms. You think about containers. Are those under the purview of the same teams, at separate team? How do you see that delineation?

[00:20:55] Geoff Kershner: [inaudible 00:20:55]. We’ve tried to [inaudible 00:20:58] for our organization. But cloud, it’s very much that. We run our services in both our own collocated data centers and in some public cloud for certain aspects. Most of it is in our own collocated but we’re a software as a service. So, yes, we have to think of the cloud and think about containers and all that. So all that kind of security blends together with our teams. We have to think about how we’re securing containers, how we’re securing deployment.

We have the typical development engineering organization that I was talking about. It builds all the features and functions, but we also have a cloud organization that manages our data centers, SRE type functions, so the DevOps, folks that are doing all the deployments and all that kind of stuff. We work very closely and hand-in-hand with those guys too. They’re also big partners with us. Just like the development teams, we work with those.

The security champions program doesn’t always necessarily fit in. But if they’re very security-conscious, they also come to us so often or we go to them. We partner with them. We meet with them every week and the same thing. When there’s tools and stuff like that that we need, we work with them to select them, to build them, to deploy them, all that stuff. It’s a partnership.

[00:22:08] Guy Podjarny: In this context, the we is the same product security team. Within the security organization, it’s the product security team that is the destination or is there another team within the security team?

[00:22:18] Geoff Kershner: It’s actually both. Depending on what it is, it’s a combination of the product security team and the security operations team. Our security operations team does far more than just managing alerts and all that. They’re actually building and supporting a lot of our tools, so they respond, and so like the scanners and stuff that we use. The selection of those, the working with the cloud teams on deployment of those, all of those are between – It’s more of the infrastructure components generally fall more in the security operation side.

[00:22:49] Guy Podjarny: Got it. My favorite piece of technology that’s in the twilight zone is containers, so who would you say has the responsibility right now to tackle container security, so security images?

[00:23:02] Geoff Kershner: That’s – It’s a combination of my team, the security operations team and working with the cloud organization, the SREs and the site reliability engineering team and the DevOps. Between us, we’re the ones that are responsible for the majority of those.

[00:23:18] Guy Podjarny: Got it. You’ve mentioned several times owning the technologies or sort of giving the tools to help these other parts of the organization build securely. Maybe let’s dig into that a little bit. What do you look for in a security tool and in a security technology? How do you approach an evaluation of a security tool?

[00:23:39] Geoff Kershner: That’s a great question. That’s another thing that I had to make sure that we’ve developed since we’ve been here. So when we’re doing an evaluation of a tool or platform or a service, we have a pretty thorough process that we go through. First of all, my approach to that is we don’t necessarily want go and find the latest, greatest, nastiest thing that isn’t really well-tested and really good. We don’t want to be the first users of most tools. It’s very interesting and it’s very easy for engineering and the technical folks to gravitate towards wanting to go to the big shiny object and stuff like that. But first of all, we want to make sure it’s kind of tried and tested, and there’s always a budget element to it too, so it doesn’t necessarily wanting the most expensive thing.

But I think the more important things that we look for is how it’s going to fit into our technology and our stack and our ability to support it. You may have two tools that have the same kind of features and capabilities. But if you don’t have the skill set to manage it and work with it, then there’s no point of having it. You got to go hire people to support it. So we look for something that will work across our technology stuff. Like I said, it’s getting a little bit more complex as we’ve acquired some companies. We’re expanding globally. There’s lots of introduction to new technology.

So it’s important to a degree possible that a tool has to be able to support all the different technologies and has to be able to operate in our data centers and in the public cloud if it needs to, and so you’d have to be able to build and deploy it in all those locations. I think the integration like into our SIM and all that is important. How are we going to be able to run a tool, collect data, and make sure we can manage it effectively, all these different elements.

We have a process and a criteria that we go through to say how do we down-select to a few, how do we run a proof of concept to the site. The other important thing, like I said before, is if there is an element, whether it’s an SDLC tool, something that we’re going to integrate as part of the build and development process versus something that’s going to run in our data centers like a scanner, a container, a security, something like that, then we have to get the teams that are responsible for those involved as well, and that’s why it’s great, for example, to have the security champions program too.

First of all, I can like, “Okay, we’re going to do something they do that does static or dynamic analysis. But to get a few people from the engineering team and make that the security champion project, we work on it together and figure out how we can get them engaged. That’s also – Like I said, not to give you your own little pitch for your product, but our developers loved it and that’s important to us. Our developers love [inaudible 00:26:13] and found it very easy to use and all that, so that’s great. Let’s do something that they’re happy to work with as well.

[00:26:20] Guy Podjarny: That’s awesome. First of all, always appreciate kind of a happy customer too. But also [inaudible 00:26:27].

[00:26:27] Geoff Kershner: I just gave you a little plug.

[00:26:28] Guy Podjarny: The approach of ensuring that sort of the relevant stakeholder is indeed involved, and I especially like I think embedding the security champions or once again raising the security champions program and how it plays a role here and being the obvious entity that can appreciate the security needs on one hand and the application on the other. So I think that’s really quite wise.

I feel like I can grill you on more and more of these questions but I think we’re running a little bit low on time, so I’ll just kind of jump to my typical final question, which is –

[00:27:02] Geoff Kershner: Yeah, you bet.

[00:27:03] Guy Podjarny: If you have one bit of advice or one tip or maybe even a pet peeve of something you wanted people to stop doing to give a team that is looking to level up their security foo, what would that be?

[00:27:14] Geoff Kershner: I’m going to give you a little bit of a vague answer. I’m going to say it depends. I think you need to find out what the organization and what the group or the team thinks about security and adapt your answer towards that. What I mean by that is if it’s something that they’re curious about or embrace or something like that, then it’s great. Then you can figure out a way to kind of collaborate with. If they don’t want to be thinking about security or have a bad experience, you need to convince them why it’s important. I think the biggest thing that’s important that I’ve learned over the years of working with many companies from consulting to prior to that, being more operational roles, I think if you come in with a fixed approach to security, that’s where you can fail. You have to understand where the organization is and what it needs and how they think about it.

When I came into Medallia, I was actually consulting. Medallia was a client of mine for a while, and one of the things that I loved about it is top-down at every level of the company they believe that security was important. I didn’t want to come into an organization as the CISO and say like, “Okay, we’re just going to hire [inaudible 00:28:22] sort of take care of that problem.” I wanted to feel that we believe that this is an important part of our journey as a company, and not every person completely had that mindset that is pretty darn close, so what I learned is you have to be open, you have to listen to them, and you have to adapt your security messages based on the position that the leadership and the teams have.

[00:28:45] Guy Podjarny: I love that. You have to sort of bring them along for the journey. You don’t take and just sort of do it for them. They need to be true [inaudible 00:28:49].

[00:28:51] Geoff Kershner: That’s right.

[00:28:52] Guy Podjarny: Geoff, this has been great. Thanks a lot for coming on and for sharing so much sort of experience and wisdom here.

[00:28:57] Geoff Kershner: You bet. It was a pleasure and I’d love – I’m happy to do this again.

[00:29:01] Guy Podjarny: Thanks everybody for tuning in, and I hope you join us for the next one.