Changing Culture

Guy Podjarny: Hello everyone, welcome back to the secure developer. Thanks for tuning back in. Today, we’re going to dig into changing culture here with a guest that has managed to do that a couple of times before in a variety of contexts. We have Nitzan Blouin, who is the engineering manager for product security at Spotify. Nitzan, thanks for coming on to the show.

Nitzan Blouin: Sure, thank you for having me.

Guy: Nitzan, tell us a little bit, before we dig into culture changes and other topics, tell us a little bit about what is it you do and your journey into security?

Nitzan: Okay, my official title is an engineering manager and I lead the product security team at Spotify. My journey is a little bit different and not very linear, which perhaps might be interesting. The majority of my career has actually been in test or QA. I prefer the word ‘test’ because it eludes to something that engineers can do versus something that someone who is not an engineer, AKA what used to be a QA, do. I’ve built five different QA departments from scratch in different organizations and different domains from e-commerce, to advertising, to a couple of gaming startups.

After the last gig, where I started a QA or test department in a mid-level organization, I had kind of a reckoning and I started to think about what’s next and I was a little bored to be honest with the test road. It’s something that I’ve done five or six times and I had a working model that was pretty easy to implement and I was looking for a challenge. I was talking to a former colleague and he was trying to tease that out of me and I heard myself say, “Yeah, I think security would be interesting.”

He was like, “Well, you need to talk to Dave Hannagan who is the head of security at Spotify.” And that’s how I ended up here and in security. Now, I am almost two years into the journey and to product security at Spotify.

Guy: Got it. I guess so far so good. You’re not regretting the decision?

Nitzan: Absolutely not. I mean, I think this may be a comfort for folks in security, so coming from the test world, actually security feels much easier. In test you really have to come up with all kinds of tricks and dances to get people to do what you ask them to do and, in security, it’s always great when you talk to a team and you say, “Hey, there is a vulnerability, there’s something you need to fix,” and they’re like, “Well, do I need to do this today or is it okay to wait for the next sprint?”

As someone in test you’re like, “Oh my god, this is easy!”

Guy: Yeah, always good to remember that there are actually areas that are even harder for them to mobilize engineering to do as they need to do. It’s interesting to think a little bit about this journey, from test to security, because you would almost expect more people to go through that journey, given that all of them are, at the end of the day, some aspect of quality of your product. Did you seek out or encounter, because you came from test into security, do you find yourself naturally uncovering the others’ route, people in the industry that might have come through that journey?

Do you feel that analogy holds true or is it just in concept?

Nitzan: Yes, I absolutely agree that for me, this is part of shipping quality code, right? You want the code that lasts longer, and that is more usable, and that is more safe and secure. The more work you can do upfront before you’re on production, the less chances you have of having incidents, of having bugs, and those are very similar.

Before I went into security, I’ve talked with a couple of heads of security or I happen to know just on a personal level, to see if I was really crazy doing this. I mean, I could have had a very smooth and easy journey just continuing in the same domain, and they were all very encouraging.

Citing exactly that, that the fact that I do have background in test is going to set me up for success and accelerated learning. Although I have not seen many others who have had the same journey, I do pick people for my team who have had similar journeys. I find that there is something about the mindset of being attentive to shipping quality code, and someone who thinks that that’s important also from an infrastructure perspective, and how do you scale that, and you don’t make that process manual and tedious.

I have two people on my team right now that actually have background in quality, anecdotally at different parts of their career, in addition to security and the two do seem to go hand in hand pretty well.

Guy: That’s good to hear. Hopefully we learn from one another. We’ll inspect maybe that analogy a little bit later on. Just to kind of close off that journey, tell us a little bit about your current role. You’ve got this engineering manager, product security, what does that mean within Spotify?

Nitzan: eah, Spotify is a fairly flat organization, we don’t have a ton of fancy titles. Our security team, which is called a tribe in Spotify jargon, has different squads and squad is the smallest operational unit. My team really looks at product security and more so on the automation. How do we bake security into our CICD pipelines and how do we make security easy for engineers?

We’re really looking to take the pain from engineers who have to do many other things other than making sure that their code is secure. How do we make that easier for them in a way that there is the minimum amount of friction possible?

Guy: Got it, let’s talk about some of these practices but beforehand, maybe the journey towards them. I know that when you, and even Dave for that matter joined Spotify, there was this notion of looking to level up or invest more in security, get better there. You might not have had when you arrived, the exact setup, right? People weren’t obviously, you came on to change something, you know? That wasn’t there before and you have this model that I found very interesting, or this perspective on changing culture and how do you do it. Walk us through it a little bit.

You arrive at Spotify, what’s the model then, and what were the steps that you did to start to tackling this challenge?

Nitzan: Yes, I arrived at Spotify and my manager, who is Dave Hannagan, who took a chance on bringing me on and saying, “Okay, there is a thing called product security and it’s a green field and you need to start up-leveling that.” Spotify is very strong in other areas of security but this one happens to be one where there was a lot of work to be done. As someone who is new to the domain, I had to figure out how to start taking actions that would also be meaningful.

As you mentioned, I have this mental model that I’ve used, that I’ve developed over time, when coming to an engineering organization and needing to change culture. I see a lot of the test work as essentially that, you changed engineering culture. You get engineers to take a step that they are not used to taking, or to use the tool they’re not used to using, or to follow a process that is not naturally part of their process. I developed this model, and I’ve tested it so far on startups which are very small scale, mid-level companies, and then Spotify was an enterprise.

The first ingredient or the first step is coming with a passion and with a vision for what you’re going to do. In terms of test, that was very straightforward, I’ve done it a couple of times. In terms of security, less so, right? But I knew that I needed to make a change and you need to have that grit and that stamina otherwise it’s going to be a tough journey. I hear the term, changing engineering culture, getting tossed around the industry a lot and I really wanted to demystify it a little bit. It’s not magic after all, it’s a very concrete process.

The first ingredient is really coming with a certain vision and with passion for what you’re going to do, because you’re going to need it, it’s not the short journey. You don’t go from zero to a hundred and one step, it’s an incremental change. It’s important to have that stamina.

And then the second step is really gathering some data and then this changes, depending on the scale of your company. If you’re in a startup, it’s a great use of time to just sit down and meet everyone and get their take on, what are the biggest problems you should focus on? In a mid-sized company, usually distributed over different time zones and locations and what not, a form can give you a lot of qualitative data and you can start looking at that. Usually I would leave like an open comment field, and when you look at that, you can see the people – my rule of thumb was people who write a comment that’s longer than a tweet, it means that they are invested.

Those people are flagged somehow and I know that I want to go back and talk to them and hear more than what a form would give me.

Guy: They have something to say.

Nitzan: Exactly. They’re invested, they have lots to say actually. I want to hear it from them. And then, in a company like Spotify, that’s an enterprise and often, there is form fatigue and you really don’t want to blast 4,000, 5,000, 10,000 people with a form that may or may not be relevant to them. Also, because you don’t know how, what’s the quality of the answers that you’re getting back.

What we did there at Spotify was we did a focused interview week, where we divided between 50 to 60 stakeholders for security from the engineering side of the house, from legal, from finance, from infrastructure, and of course, our customers. Those help to start mapping out the landscape and you see patterns very quickly, and that helps basically to inform your strategy. That’s the third step.

Guy: Yeah, let’s kind of unravel it specifically maybe in the context of Spotify. So, like you go back to that demystify piece or come with a certain passion, you know, towards it. What was the mission in your mind for Spotify? What was the aim, the goal that you were trying to establish?

Nitzan: I wanted to come out with a plan that would solve security as an engineering problem. Coming from the test world, you hear the word best practices quite a bit. I can tell you for a fact that nobody does anything because it’s best practices. People do something because they understand the why, and they understand the reward, and they understand this would save them a bunch of time later.

I wanted to drill down to what was the most important problem, and how can we solve it from an engineering standpoint? Meaning it’s automated, scalable, there is a very easy and quick feedback loop if things don’t work. That’s how I came to the problem versus coming more from a consultancy standpoint of, “I know and I will kind of tell you what to do, or what is the best practice.”

Guy: Yeah, that sounds very healthy and also, taking advantage of coming from a different field or different demand of expertise. Not having maybe that temptation to think that you have all those answers. In that data gathering, you explain how you’ve gotten a bunch of people together and asked them questions. What type of questions did you ask as you were gathering information about this change?

Nitzan: I think we did four questions that really established the baseline and they were very simple. I don’t remember exactly what the questions were, but what is the biggest problem that you see right now? And what is the biggest problem that you would want us to solve?

And again, the people that would show to come and talk to us, as stakeholders, had a lot to say. We need that just to plant one or two seeds and they would start talking and they would be very generous in offering their perspective and their opinion, and that’s really what we wanted. We wanted different opinions.

Guy: Got it. We’re past step one, you know, we’ve established this awesome mission of come up with a plan to solve security as an engineering problem. I love that quote. You go off, you talk to that set of stakeholders, you ask a bunch of questions. What’s the third step, what happens next?

Nitzan: the third step is really creating the strategy. Once you have the strategy of course, you hire the team to help you to start to implement the solutions, the prioritized solutions or the first step, and maybe start a little bit on the intermediate step. You start to look at the change of the culture, I would say, two years down the road. So you look at your end result and you kind of work backwards from that. Being a very agile organization, I also am not super methodical, I don’t have the two year vision fully fleshed out to the T.

Because I want to see where each step is going to take me, and sometimes there is something that you put in place that takes you way further than what you thought, and sometimes it’s the other way around, and sometimes it’s just according to plan. In my experience, you want to have a pretty clear 12 months vision, 18 is a little blurry, and two years is me without my glasses. Looking down the road. It’s a little fuzzy.

Yeah, that’s what we did at Spotify and at first it was just building the team and, in my case, one of the things that I did that’s I think is worthwhile noting, I really hired engineers. That was part of the interview process is having coding skills. I needed to have people in the team who could work side-by-side with infrastructure engineers, and then also work with our customers, who are going to be product engineers or future teams.

Guy: Just to clarify. When you say customers, you’re not referring to Spotify listeners, you’re talking about engineers and the team and sort of internal customers?

Nitzan: That’s a good clarification. I’ve been in the infrastructure world too long. When you’re come from infrastructure, your customers are Spotify engineers. I always say we have the toughest customers because Spotify engineers were blessed with having very senior engineers or very smart, so they also make up for pretty tough customers and we need to satisfy them.

Guy: When you talk about hiring people with engineering skills into this, was that driven by the work or maybe more on the empathy side? It’s more because they will need to be writing software, or is it more about interacting with those customers?

Nitzan: I would say, it’s a little bit of both and, again, this is a lesson I learned from the test days. If you ask teams to do things that you can’t show them how to do, they want to do them. Often, in order to get teams going, and this comes back to the engineering transformation or the changing the engineering culture, you need to be able to help them out with the learning curve. For that, you need hands on people to do that and we needed engineers because we’re looking at the infrastructure side of the house and we’re looking at automation.

We needed the security engineers to be able to work with the people who build our different build pipelines.

Guy: Maybe, if you don’t mind sharing some specific examples. So what have been some of these learnings, and now you have a bit more mileage behind you and not just having theorized but also executed. Can you share some examples of steps that worked well that people should mimic, or maybe even some that didn’t that people should avoid?

Nitzan: So I think we’re really got the great pointers from all the people we have talked with. The first thing that we learned was that there was a lot more appetite to hear more clear guidelines from security, and we’re careful of giving those and being very opinionated, so it was kind of a vote of confidence like, “Hey, you could be more opinionated and we will welcome it.”

Secondly, more specific in terms of Spotify, there was a need for better automation, and that’s how we partnered with Snyk as well, and that was in my book a low hanging fruit right there. Where you want to get a tool for engineers, in Spotify our security organization is a hybrid model of a centralized and decentralized. Most of Spotify’s engineering organization is decentralized. So security is a bit of an outlier but overall we do want engineers to be able to act on, for example, security vulnerabilities even though code there is not some centralized team that goes in and comes to adapt and works on resolving those.

Guy: Got it. How do you engage some of the practices? These are great tool choices, if I may say so, but the investment in security engineering is great. On the human level, how do you interact with the rest of the engineering organizations? Is it messages that get passed on? Are you aligned in some way? How do you tell them when they’re doing something correct versus just bashing them over the head? Can you give us some examples of the interaction models?

Nitzan: Yeah, we definitely don’t bash over the head at Spotify. We have different teams that use different methods. I would say when you come to working with customers, I am going to go back to that word, sometimes it is good to get the message from different directions. So we have a team that is working specifically on education initiatives, on providing guidelines, and so forth. I would say the more manual part of product security, and then we have the teams that are more focused on automation.

Sometimes you learn from a trial by fire, right? And I think coming from test this happens a lot. Nobody takes critical bugs very seriously and, until that goes somehow rolled out to the external users, and then people get really good about fixing those bugs. It is a similar process again in our case going back to that little system of mine. When we started interviewing people, it became very clear who are the strong allies that we can bring to the table, whether it is from a leadership perspective, whether it is from an architecture perspective and whether it is just strong engineers who want to come and sit at the table.

That is something really important because those early allies are just going to be amplifiers of your own message and of the change that you are trying to make, and they are going to help to spread it, and probably speak about it in a different way. It is very good to hear the same message coming from different directions, and especially from a direction you don’t expect.

People would expect me to talk about security but if an engineer and their team starts talking about security then, usually, there is more attention coming to that.

Guy: Right, you know I guess it is that form. I love that notion of using the forms, not just to gather that data but also to identify your allies. Do you consider that almost like an early version of the security champion’s program? Do you also think a security champion’s program is something you are doing or will consider later on?

Nitzan: So, that is an interesting question, because I have talked a lot with other companies in the space about security champions and investments and gains and opinions in general and I’ll be honest, I got very mixed recommendations around security champions. At Spotify, we have a model, it is called TC4X. We started with an initiative called TC4D, test certified for data-focused squads, that was an initiative that I was very involved in, and it was one of those tiny grass roots efforts that became a way of working right now at Spotify.

And it leans on Google test ladder and you really self-certify yourself levels one to three. I think three is usually the highest and now at Spotify we have this per data for mobile clients, for back end. I could see something like that, because Spotify believes so much in the sense of ownership, that each team has of their code, their product, whatever that product is. Whether it is an infrastructure product that’s offered to other engineers, or it is something that is offered to Spotify users. I would see a model like that working really well for Spotify but we haven’t started digging into that yet.

Guy: Yeah that is very interesting. So this is a self-certification type of element where teams or individuals like is it certification of the application of the software or is it a certification of –

Nitzan: So usually it would be a certification of – that’s the million dollar question by the way. When we created TC4D it was something that we spent a lot of time talking about, do you certify just the data pipeline? We have some teams that are on many, many data pipelines.

I wouldn’t say we nailed it down to a science. With security it’s a little easier because we are looking at services. So I think that that would be one scenario and again this is not something that exists for the security world.

What does exist for other domains is that our service has a way to self-certify then they get a badge. We have a great system called backstage, which is now open source, and then that badge is public and that also signals to someone who is relying on your service to say, “Hey, we invested in security and that doesn’t need to be a concern for you if you consume our data.”

Guy: I love that for a variety of reasons. I think we definitely have seen the world of open source how having, for instance, a disclosure policy or a security.md file in open source repositories actually correlates well with lower risk, with people that address this well when people know where to disclose the issue if there is a disclosure file then they would disclose it responsibly. They might even think to report a security flaw or a security concern they are seeing.

Similarly we’ve seen, actually at ComCast, I had a recent episode here. There was some great data demonstrating this type of iterative process, identify a 40 odd practices, and helped team – they had picked. In their case they tried to roll out first eleven then three others practices across the organization and see people adhere to them, but it is about that. It is about those bite-sized chunks.

I guess it even eludes a little bit into that paved road mindset you know from Netflix. Talking about that infrastructure maybe more on your team, because you are building these services and people can choose to use them, and they’ll get the badge, and they’ll be good. So there is definitely a lot to like about that path. They hope that when we talk at whenever post you’ve implemented we’ll hear some stories about it. Shifting gears this is fascinating, I’ve got a whole bunch of other questions but, another topic I want to make sure we talk about a little bit is the hiring.

So you’ve had the luxury and the pain here of needing to build up a team and hire for it. The world of security isn’t well-known for its diversity, and I am sure you have encountered that as well. How did you approach that? When you have hired you already gave us an inkling around this need for programming or for engineering expertise. How did you approach this type of hiring for diversity or their attributes that are important as you form your team?

Nitzan: Yeah that is a great question. So security is at, I think in general the last time I looked at the data was at 12% diversity for gender. That is – and so that is a very low. I think overall benchmark is 15 for diversity and more let’s say “progressive” organizations are at 20. Spotify is at 35 now, after investing heavily in diversity and developing strategies. So as a hiring manager I feel that is one of my responsibilities, and it is actually one of the joys of being a hiring manager, is that you can help uplift people from underrepresented groups.

In this case, for me the gender piece naturally comes to mind, but of course it is not the only determination, right? You pick the best person for the job at the end of the day and, lucky for me, I happened to build a team that is 75% female. This is the first time in my career that that has happened. Yeah, and it is wonderful. It is a fun team to be a part of. It is a very different dynamic in terms of our interaction but what I really looked for is – I think, and this might be a bold statement, and the more mileage on me in terms of security, but as a field in terms of people going to study to be a security engineer, or someone saying, “Hey I want to be a security engineer when I grow up.” I don’t think that that is very common or not common quite yet.

So what I really looked for was people with strong engineering backgrounds, primarily backend, because that’s the domain, and then with passion for security. I have one person in my team that has a masters in security and computer science. I have one person who has taken some courses in college. So there was a mix in terms of the skillset between people who came from tech background, people who came from security background. I like to mix also the regulated/unregulated, especially when it comes to security, because that brings a very different perspective into, how good does your solution needs to be? And someone who works at the bank or in a health organization has a different approach than someone who worked in another tech company.

So, I believe in building mixed teams and diverse teams, and take that onto different access in terms of experience. At the same time, I think also all of us who are privileged enough to be in a position of hiring people, we do have to think about underrepresented communities. How can we give people a chance? It does require more heavy lifting on the management side. If you take someone who is more junior then you would need to give them more attention in the beginning, but that always pays dividends, always.

Guy: Yeah and how I fully relate to the importance of this, and some of the social responsibility. Also, as the white man here, like, I think feel even doubly responsible than my group, if you will, has sort of not solved or contributed to the problem before, even more responsible to kind of help be a part of the solution. But also like indeed the challenge is often times successfully doing so, you can be committed to it conceptually.

So you gave us one example with how we can approach it, which is you can hire people that are more junior and then invest in them. Are there other tips or examples of practices that people should embrace to help them do better in terms of hiring diverse candidates?

Nitzan: Yeah, I think the other path that has worked well, specifically for my team, was really looking at the talent pool of backend and assuming that someone – I mean every hiring manager has this set criteria of what has to be there. In my case it was the coding skills and then you take that side-by-side and you say, “Okay, I maybe have a very strong backend engineer, who is passionate about security, but doesn’t know much. Can I train them?” And that has worked again for us.

So we knew we would have a strong engineer who can start executing on engineering tasks right away, and then we also knew that we would be able to level up the security knowledge and best practices etcetera, with specific trainings.

Guy: Yeah, I love that. So basically look at hiring pools, or the queues for other teams that, for one reason or the other – well, maybe they did fit that team and you might steal them away, but you know it could also be that they didn’t have any specific knowledge for that team but they were great candidates otherwise, and then tap into that resource, and bring them in. Skill them up in the areas that you need to build them up. Does that sound right?

Nitzan: Yeah exactly, and there are no perfect candidates, but I think what you need to make sure there is, is that spark or that curiosity or that passion to learn about security. If someone doesn’t care about security, this doesn’t work, right? So you need to have that base level. I think sometimes people get very hung up on finding a perfect candidate that ticks all the boxes and, at least in my experience, that doesn’t exist. Then you think about where can you compromise.

And if you do prioritize diversity, you often hear like, “Oh this pipeline is not diverse.” There is a lot of ways to make that pipeline more diverse and then to find again the best person for the job. You don’t have to limit yourself to diversity on the gender axis. There is race, there is age, and you build those diverse teams and those are the strongest teams because your customers, guess what, are exactly going to be a mirror of that team. If you are going to have a bunch of people who are just like you, you are probably not going to be able to solve for the problems that your customers are going to see.

Guy: Yeah very well said. So Nitzan this was excellent and you have already shared a whole pile of advice, but I still like to try to squeeze one more. I like to ask every guest coming on the show, if you have one bit of advice to give a team that is looking to level up their security-fu, what would that be? Something that they should start doing, stop doing, to get better when it comes to security?

Nitzan: That is a great question. What comes on top of mind to me, and this is probably tainted a little bit, but the different security organizations that I have talked to or experienced, is really talk to your customers. Don’t come with answers but come with questions and you will have a much easier time.

Guy: Yeah that is so simple and yet not done all too often. So that is a great piece of advice. Nitzan, thanks again for coming to the show. This has been really great, very insightful and keep up the great work that you are doing at Spotify.

Nitzan: Thank you so much, Guy. It was wonderful to be here.

Guy: And thanks everybody for tuning in and I hope you join us for the next one.

[END OF INTERVIEW]