A Sysadmin moves to InfoSec

Technology-related advent calendars have provided me with much inspiration over the years.


Perl and PHP advent calendars gave me the incentive to start SysAdvent in 2008, and I’m honored to participate in this year’s SecAdvent, which I’m told is inspired by SysAdvent. Sometimes things come full circle.

This is my story about how I joined an infosec team after being a sysadmin and a software developer for a while, which afforded me a different perspective when making this transition. If you are comfortable making a computer go beep and boop, from anything between shell scripts, to network technology, to cloud services, you might find a good & fulfilling job opportunity in information security that you may not have previously considered.

Where it All Started 

I’d operated systems under various titles over the course of the years: SRE, sysadmin, system engineer as I liked the mix of duties — capacity planning, tool making, troubleshooting, maintenance, etc. One part of operations that always fascinated me was logs, and that’s how I ended up at Elastic through my work on Logstash (started in 2009 and was acquired by Elastic in 2013), which I led until 2018.

After almost ten years leading Logstash, I wanted to explore something new.

Around the same time, there was a new worry for many companies, GDPR, and it had a deadline — many people remember the date May 25, 2018 as an ominous one. At Elastic, meetings tended to be open, and I liked to learn what other groups were working on. With GDPR topics, discussion about logging came up frequently – what needed logging, how long to retain records, etc. I like logs, so I volunteered to join the infosec team to help run some Elasticsearch clusters and help us solve our logging-related infosec challenges.

I had no idea what I was getting myself into. 

I was largely a newbie to infosec. I was familiar with popular hacking topics that flow out of DEFCON each year, and I had helped with Shmoocon’s Hack or Halo and HackFortress for a few years, but I had never worked near corporate infosec.

Did you know that newbies have a superpower? 

Newbies are positioned to identify the weird, the confusing, and the unknown. Experience normalizes things, and before that happens, you are in a perfect position to question the way things are and whatever feels confusing. After normalization, confusing and difficult tasks are often left without improvement, and that drives me to say, “If a user has a bad time, it’s a bug” — to empower folks to report the weird and confusing.

Building the Team

So, in early 2018, I joined the Elastic Infosec team as its third member. While it was good timing for me to join, it wasn’t exactly smooth sailing. Some internal groups like the Elastic Cloud team had dedicated infosec staff, but the formation of a company-wide infosec team was fairly new to Elastic. My teammates came and went as we figured out what “Infosec at Elastic” meant: what roles/responsibilities were needed, what department would infosec be in, how is success measured? I suspect this scenario plays out at other companies as they find a need to focus on information security.

As promised, my first order of business was to set up log collection for a few key systems. This team was still nascent at the company and very small, and I found myself simultaneously having to own or support many different areas of infosec: operations, incident response, policy reviewer, infosec advisor, customer security question-answerer, and vendor safety evaluation. It became clear that we’d need significantly more hands to properly handle these different tasks,  and, after 18 months, the team had grown fivefold from 3 to 15, where each team member had different focus areas.

I felt successful on this infosec team, and I’m going to share some of the reasons why. For yourself, if you are considering joining infosec, maybe this will help you think about what success might mean for you. 

When I think about what set me up for success on this team, a few things come to mind. 

  • First, my main responsibility was to manage log collection and operating other infosec-related services. I have a lot of experience doing this as a sysadmin and other operations roles I’ve held, so strong operations skills and experience helped.
  • Second, I had a mentor who had been doing infosec for much longer and also had an ops background. She helped me learn about different infosec areas, recommended books, and helped me get onboarded. 
  • Third, my past involvement with other teams at Elastic gave me confidence to ask weird questions and also be a bridge between infosec and other teams. 

Another mindset which I feel helped me be successful was this:

Can we have infosec provide business value?

From outside infosec, I’ve seen plenty of infosec organizations act as gatekeepers which exist only to tell you “no” when a question is asked. These groups were invisible until things went wrong. I learned in operations that invisible work was a path without success, I wanted to find an approach in infosec where my team could do work which could enable and empower and also could be celebrated (which means it can’t be invisible work).

“If you are comfortable making a computer go beep and boop…you might find a good & fulfilling job opportunity in information security that you may not have previously considered.”

Rolling up our Sleeves and Getting to Work

Our infosec team was pretty much a greenfield project, and we basically had free hand to make technology choices for the team. Where and how should we host any services we might need? We worked with a few different teams to help us decide. Elastic already had a large Cloud team which was available for running Elasticsearch clusters, and we also had an internal infrastructure team responsible for Jenkins, some key business services, a company-wide Kubernetes platform, and more. We had some discussions with these teams to share knowledge and evaluate any limitations, and ultimately decided to run things ourselves. Some of this decision was for “separation of duties,” but also, at the time, what we wanted wasn’t available internally. In my head, this run-things-ourselves approach was temporary, maybe 1-2 years, with the goal of relying more on other internal teams to reduce our own workload and strengthen relationships with those teams.

After doing some research, we picked Kubernetes as the platform (despite having little to no experience with it). Kubernetes had a lot of positive inertia at the time both inside and outside the company, so it felt like a safe choice. We skipped a lot of the harder parts of Kubernetes by using a managed service, Google’s GKE. Kubernetes also helped us make our work visible. A few other teams were using it in production, and by choosing Kubernetes, we could lean on those teams’ expertise while also building strong relationships with those teams. A successful information security implementation, to me, would require strong relationships and trust.

We ran things large and small upon this infrastructure — several Elasticsearch clusters, TheHive for incident response,  and a few other small tools. The outside inertia on Kubernetes accelerated our efforts to run larger log collection clusters, where we initially wrote much of our own deployment pieces (kubernetes yaml, etc), but quickly switched to communal efforts around helm charts and operators.  I helped shape the Elasticsearch Kubernetes operator as the first production user.

Early adoption of the Elasticsearch Kubernetes operator was important because it added business value in a number of ways. First, we had some logging clusters receiving a few terabytes of data per day, and our success paved the way for other users to succeed in operating moderately-sized clusters. Second, it helped us build a strong relationship with the developer and product teams. Third, our use case was focused on information safety, so our Elasticsearch clusters were configured with security enabled, encryption, retention policies, backups, and single sign-on. The result? I shared our team’s success story to many customers and users who attended our conferences. We can tell our story in full truth that yes, this new product works, and yes, we use it in production on critical systems. This is possible because this work was visible and valuable to the whole organization.

The Fun Parts

I love making little tools — shell scripts, web apps, command line tools! Do you like making little tools as much as I do? If so, keep reading.

At Elastic, our compliance group within infosec helped steer a course towards a variety of standards certifications. One early effort was for something called “SOC 2” — a comprehensive effort to define and document how we work with respect to safety practices and a 3rd party team to audit what we say we do vs what we appear to do. It’s a good milestone for any company to achieve this.

We finished our first SOC 2 process, and I learned that in order to send customers our SOC2 report, each report (a PDF) had to be watermarked uniquely each time. While this may sound pretty simple, it was actually a fairly tedious and manual process that required at least two people to manage due to the volume of the task. My experience in operations and automation drove me to write a small tool to perform these steps — a small script to do the watermarking steps and a web app protected by single sign-on to make it accessible. Once we succeeded, our team wrote a how-to on the wiki for how to use the tool. The old manual process took several days and used a ticketing system for requests, and this new process was self-service and completed with just a few clicks on a website. I love little tools like this. The outcome is important, since sharing these reports is part of the sales process, and dramatically reducing time cost will help smooth out sales.

A second tool was one to enable single sign-on to Kibana. Managing user accounts is tedious, so it was common for folks to create one account and share the password for each cluster. Setting up SSO (single sign-on), at the time, required opening a ticket with our helpdesk team to make configuration changes on our identity provider, Okta. As you can imagine, Elastic runs a lot of Elasticsearch clusters internally — some for production, some for testing, some for experiments, so access control was somewhat of an operational nightmare. I reached out to a few folks around the company — those with expertise in SSO, those in IT, etc, and standing on their shoulders, wrote a tool to help folks automate setup of SSO on their own Elasticsearch clusters. This added value, again, the same two ways: reducing load on the helpdesk team while also empowering individuals to perform an otherwise complex task. It also strengthened infosec’s relationship with our IT team. Beyond that, it helped us achieve infosec goals to reduce use of shared accounts and took a step towards making things safer-by-default.

Food for Thought

I’d like to leave you with some thoughts and questions you can ask yourself to see if this type of transition is the right one for you.

  1. Do you like investigating and troubleshooting?

    Incident Response might be an area of interest for you! Much like the process for piecing together a timeline for an outage or troubleshooting a failed service, incident response uses similar investigative skills, technical writing, and interviewing.

  2. Do you like learning?

    Information security is a cross-functional effort! Depending on the team’s size and scope, you may work with, and learn from, folks from your company’s legal, marketing, sales, IT, customer support, software engineering, office admin, HR, and other teams to learn about their challenges and their role in supporting the business, things you may not be exposed to in other engineering functions.

  3. Maybe you like making content and giving presentations and training?

    Besides working on Logstash, my early years at Elastic had me writing and delivering product training. I rather enjoyed it! I’ve also given a number of presentations at various conferences on a variety of topics. This experience put me in position to write and deliver the “What is Infosec?” sessions for Elastic’s new hire onboarding. Have you given talks before? Do you want to? Think about any mandatory training you’ve ever received, especially for information security. Was it fun? Relevant? Could you deliver the same lessons in a more engaging way?

  4. Do you like technical writing?

    Writing documentation and creating content  is a wonderful skill which can have a big impact on your coworkers. Have you read your company’s information security policies? Are they endless documents full of dry legalese that probably nobody understands? You could be the person to improve that!

One of the things I’ve learned has been that information security is not strictly defense of data but more broadly that data can cause harm, so it is important to take care in choosing what to store and what to destroy. That old adage: the safest computer is the one that is turned off. The safest data may be data you avoid storing at all, and these are some of the lessons learned from working on projects such as GDPR and others, that brought me new perspectives. 

Some of the learning can be a bit tiring or at least strangely amusing. There are so many short abbreviations and acronyms. FedRAMP, FIPS, SOC2, CSA Star, PCI-DSS, HIPAA, and more, and each of them have their own unique terms and topics. It’s turtles all the way down.

I’ve also found that infosec loves spreadsheets. Sometimes this love is too strong. Spreadsheets with hundreds of rows and dozens of columns and some individual cells contain entire essays longer than even this article!

I’ve learned that compliance is not security. Compliance is a fascinating and valuable industry of standards, lawyers, auditors, and storytellers.

I’ve enjoyed my transition to infosec, and maybe infosec can be a place for you! If you’re already in infosec and have friends or peers who are interested, consider helping open the door for them.

Related Posts

Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.