Rolling up our sleeves and getting to work

Our infosec team was pretty much a greenfield project, and we basically had free hand to make technology choices for the team. Where and how should we host any services we might need? We worked with a few different teams to help us decide. Elastic already had a large Cloud team which was available for running Elasticsearch clusters, and we also had an internal infrastructure team responsible for Jenkins, some key business services, a company-wide Kubernetes platform, and more. We had some discussions with these teams to share knowledge and evaluate any limitations, and ultimately decided to run things ourselves. Some of this decision was for “separation of duties,” but also, at the time, what we wanted wasn’t available internally. In my head, this run-things-ourselves approach was temporary, maybe 1-2 years, with the goal of relying more on other internal teams to reduce our own workload and strengthen relationships with those teams.

After doing some research, we picked Kubernetes as the platform (despite having little to no experience with it). Kubernetes had a lot of positive inertia at the time both inside and outside the company, so it felt like a safe choice. We skipped a lot of the harder parts of Kubernetes by using a managed service, Google’s GKE. Kubernetes also helped us make our work visible. A few other teams were using it in production, and by choosing Kubernetes, we could lean on those teams’ expertise while also building strong relationships with those teams. A successful information security implementation, to me, would require strong relationships and trust.

We ran things large and small upon this infrastructure — several Elasticsearch clusters, TheHive for incident response,  and a few other small tools. The outside inertia on Kubernetes accelerated our efforts to run larger log collection clusters, where we initially wrote much of our own deployment pieces (kubernetes yaml, etc), but quickly switched to communal efforts around helm charts and operators.  I helped shape the Elasticsearch Kubernetes operator as the first production user.

Early adoption of the Elasticsearch Kubernetes operator was important because it added business value in a number of ways. First, we had some logging clusters receiving a few terabytes of data per day, and our success paved the way for other users to succeed in operating moderately-sized clusters. Second, it helped us build a strong relationship with the developer and product teams. Third, our use case was focused on information safety, so our Elasticsearch clusters were configured with security enabled, encryption, retention policies, backups, and single sign-on. The result? I shared our team’s success story to many customers and users who attended our conferences. We can tell our story in full truth that yes, this new product works, and yes, we use it in production on critical systems. This is possible because this work was visible and valuable to the whole organization.

The fun parts

I love making little tools — shell scripts, web apps, command line tools! Do you like making little tools as much as I do? If so, keep reading.

At Elastic, our compliance group within infosec helped steer a course towards a variety of standards certifications. One early effort was for something called “SOC 2” — a comprehensive effort to define and document how we work with respect to safety practices and a 3rd party team to audit what we say we do vs what we appear to do. It’s a good milestone for any company to achieve this.

We finished our first SOC 2 process, and I learned that in order to send customers our SOC2 report, each report (a PDF) had to be watermarked uniquely each time. While this may sound pretty simple, it was actually a fairly tedious and manual process that required at least two people to manage due to the volume of the task. My experience in operations and automation drove me to write a small tool to perform these steps — a small script to do the watermarking steps and a web app protected by single sign-on to make it accessible. Once we succeeded, our team wrote a how-to on the wiki for how to use the tool. The old manual process took several days and used a ticketing system for requests, and this new process was self-service and completed with just a few clicks on a website. I love little tools like this. The outcome is important, since sharing these reports is part of the sales process, and dramatically reducing time cost will help smooth out sales.

A second tool was one to enable single sign-on to Kibana. Managing user accounts is tedious, so it was common for folks to create one account and share the password for each cluster. Setting up SSO (single sign-on), at the time, required opening a ticket with our helpdesk team to make configuration changes on our identity provider, Okta. As you can imagine, Elastic runs a lot of Elasticsearch clusters internally — some for production, some for testing, some for experiments, so access control was somewhat of an operational nightmare. I reached out to a few folks around the company — those with expertise in SSO, those in IT, etc, and standing on their shoulders, wrote a tool to help folks automate setup of SSO on their own Elasticsearch clusters. This added value, again, the same two ways: reducing load on the helpdesk team while also empowering individuals to perform an otherwise complex task. It also strengthened infosec’s relationship with our IT team. Beyond that, it helped us achieve infosec goals to reduce use of shared accounts and took a step towards making things safer-by-default.

Food for thought

I’d like to leave you with some thoughts and questions you can ask yourself to see if this type of transition is the right one for you.

  1. Do you like investigating and troubleshooting?

    Incident Response might be an area of interest for you! Much like the process for piecing together a timeline for an outage or troubleshooting a failed service, incident response uses similar investigative skills, technical writing, and interviewing.

  2. Do you like learning?

    Information security is a cross-functional effort! Depending on the team’s size and scope, you may work with, and learn from, folks from your company’s legal, marketing, sales, IT, customer support, software engineering, office admin, HR, and other teams to learn about their challenges and their role in supporting the business, things you may not be exposed to in other engineering functions.

  3. Maybe you like making content and giving presentations and training?

    Besides working on Logstash, my early years at Elastic had me writing and delivering product training. I rather enjoyed it! I’ve also given a number of presentations at various conferences on a variety of topics. This experience put me in position to write and deliver the “What is Infosec?” sessions for Elastic’s new hire onboarding. Have you given talks before? Do you want to? Think about any mandatory training you’ve ever received, especially for information security. Was it fun? Relevant? Could you deliver the same lessons in a more engaging way?

  4. Do you like technical writing?

    Writing documentation and creating content  is a wonderful skill which can have a big impact on your coworkers. Have you read your company’s information security policies? Are they endless documents full of dry legalese that probably nobody understands? You could be the person to improve that!

One of the things I’ve learned has been that information security is not strictly defense of data but more broadly that data can cause harm, so it is important to take care in choosing what to store and what to destroy. That old adage: the safest computer is the one that is turned off. The safest data may be data you avoid storing at all, and these are some of the lessons learned from working on projects such as GDPR and others, that brought me new perspectives.

Some of the learning can be a bit tiring or at least strangely amusing. There are so many short abbreviations and acronyms. FedRAMP, FIPS, SOC2, CSA Star, PCI-DSS, HIPAA, and more, and each of them have their own unique terms and topics. It’s turtles all the way down.

I’ve also found that infosec loves spreadsheets. Sometimes this love is too strong. Spreadsheets with hundreds of rows and dozens of columns and some individual cells contain entire essays longer than even this article!

I’ve learned that compliance is not security. Compliance is a fascinating and valuable industry of standards, lawyers, auditors, and storytellers.

I’ve enjoyed my transition to infosec, and maybe infosec can be a place for you! If you’re already in infosec and have friends or peers who are interested, consider helping open the door for them.