By definition, startups need to be able to adapt quickly. The ability to move and adjust quickly is crucial if we are to have any hope of out-competing larger, established players.
On the face of things, this would make us a perfect fit for an in-house DevOps team. DevOps and CI/CD are meant to increase rapid development, testing, and deployment. DevOps was designed to remove traditional barriers between development and operations teams – making sure they’re collaborating across the software’s lifecycle to identify opportunities for improvement, then integrate changes quickly and seamlessly.
However, startups are also – to be honest – cost-sensitive creatures, especially in challenging financial climates. And this is why during the early development stages of our product, prior to GA, we did not choose to have a dedicated DevOps team - or even a full time DevOps engineer. And yet, we were able to implement DevOps practices for over a year before we hired for a full time DevOps role.
Becoming 'DevOps People' Without a 'DevOps Person'
The success we had adopting DevOps practices came down to 4 practices we implemented in the team:
- Engineers learned to speak DevOps - In the early days, everyone was DevOps and DevOps was everyone. All engineers were aware of and able to contribute to DevOps-related issues. The lack of DevOps actually made us all far more aware of hidden cloud costs that we might not have all paid attention to if there was a dedicated DevOps team.
- The “Bus Factor” – This is going to sound a little morbid, but the basic idea was that, lacking a single DevOps point of contact, we always had to ensure that too much information was never concentrated in the hands of one person. That way, if that person got “hit by a bus” (metaphorically speaking, of course), the loss to the project would be minimal. Teams without DevOps can’t rely on one person.
- Lowering the on-call burden – DevOps teams are the ones who get called at 3am on Sunday because production is down. It’s just part of the game. But without DevOps, we learned to rotate the burden of on-call during off-hours and weekends, which lowered the burden on us all while positively impacting our team spirit and morale.
- Professional DevOps teams, like professional teams in any field, work with established tools and methodologies. This is a good thing. That said, engineers are constantly trying to find new and better ways. During the time where we filled the DevOps role, we discovered a lot of new ways of doing things that we might not have found had we had a dedicated DevOps team.
But Then Things Changed...
As I mentioned, we had to move on from our split DevOps responsibility and bring in a full time DevOps professional.
We knew it was time to bring on dedicated DevOps pros when we got to the point where we had too many product and feature tasks and couldn’t keep up with the core infrastructure tasks. As we began to implement more feature requests, we found ourselves spending less and less time on infrastructure. This is a problem because versions and features still continue being released, whether or not the infrastructure is ready to support them.
For example, just before GA, our team needed to make absolutely sure that our infrastructure was stable, that the product was monitored and we had full visibility of issues. To make this happen, we had to put a hold on new features and focus solely on infrastructure. Lesson learned: it was time for DevOps.
The Takeaways
Not having DevOps is perfectly OK…until it’s not. One big takeaway here is that you need to watch scale carefully between handling infrastructure and handling features. Once it tips too far towards features, it’s time to consider DevOps. The other takeaway is that you need to consider everything in terms of dev velocity versus costs. Engineer time is expensive, on one hand, but so is slower dev velocity. The question is, where is the breaking point for your organization?
