Hacking SaaS #23 - SaaS Developer Toolbelt
We discuss the right tools for SaaS developers, data migrations, K8s complexity, KV vs RDBMS and other tough decisions that SaaS developers make.
Did you know, the first Hacking SaaS was published on July 12, 2022? This is the 1 year anniversary for this newsletter!
Most of the content here is based on conversations in the SaaS Developer Slack. Join us and take part in the conversation.
Also, a quick reminder that if you are a SaaS developer, or interested in what SaaS developers talk about, and you happen to be around Mountain View, CA next week, you should join our meetup and hear how SaaS developers use generative AI.
But this post isn’t about generative AI or even the one year birthday of this newsletter. It is about the ways that SaaS Developers choose which tools to use. Lets dive in!
Which tools should SaaS developers use?
Jeffrey Sherman asked a philosophical question:
Do you see yourself more as someone who provides a service or someone who writes software?As an example of the difference, and the origin of my curiosity, I was talking to a developer who is against using no-code tools like Zappier to solve problems because they wouldn’t be using their software skills.
The community was unanimous: “Engineers solve problems. You can use any tool that solves the problem”.
Aaron Kimball wrote an in-depth answer, and then published it in his blog. He started by defining “What is SaaS?” and used this to also define the role of a SaaS developer. This is a pretty big deal!
what exactly is it you’re trying to accomplish with a SaaS business?
I really think the “S stands for Service” part of SaaS needs to be more prevalent.
[….]
So to the question of “what tools ought a software engineer feel comfortable reaching toward,” I think you use what you need to use so that you can “do what it takes to make [customers] happy.”
you don’t have to write a “weapons grade” bulletproof API to do the job, it just needs to get done; choosing the right tool for the job to dispatch it expediently is, I think, what good engineering is all about.
I love every bit of this. Although, I do think customers value bulletproof APIs too.
Relational DB or KV Store?
This was published by a former colleague of mine, and it discusses a key decision for SaaS developers - the datastore. Jos goes into a lot of history and interesting stories, so worth reading even if your mind is made up.
Keeping History
Jorin is a new member of the SaaS Developer Slack, and he shared his blog, where I ran into an excellent article discussing the challenges regarding storing historical data, different approaches to the problem, how this impacts schema evolution and how to handle more nuanced retention concerns. This is one of the problems that pretty much every SaaS developer ran into, and Jorin provides a great overview.
One of the most obvious uses of historic data is implementing requirements for auditing and version control in critical applications.
Having historic data also becomes invaluable for understanding and debuging complex processes. It is not only helpful for developers but also for users trying to understand the processes of the bigger system the software is used in.
From a technical perspective building solid, fault-tolerant, distributed software that interacts with external systems becomes more feasible once you have a log of change events. You can replay the events at any time to recreate the state of different sub-systems which is a big help keeping everything in sync without running into race conditions.
Complexity, K8s and Microservices
Colt decided to bring a controversial tweet and some 🍿 to the SaaS developer community:
And Lucas Stephens gave an in-depth response, that started by separating “use K8s” decision from “have 150+ microservices” decision. And rightly so - they are independent and you can make one mistake at a time.
Two parts to this:
Kube
Microservices
In my experience people who criticize Kubernetes for its complexity forget to tell you about all of their custom internal tooling they've spent years building to replicate the same functionality. Or they compare Kubernetes to something fundamentally more limited in scope (like Ansible + systemd) & think "wow why would I ever use Kube". As a whole, there's not a better open-source out-of-the-box orchestrator than Kube; alternatives are either proprietary or have less functionality (cough cough Nomad cough cough).
These people also forget that Kube is meant to be the platform's platform - you're supposed to customize & build APIs around your cluster to give you what you need. The power of Kubernetes is that it is basically an open-source API for deploying things - that's really powerful for establishing community standards & projects, as well as hiring.
As for microservices - I definitely agree with "stick to a boring architecture for as long as possible". And a boring architecture is definitely possible with Kube.
Most people don't realize that adopting a services-oriented architecture requires you to get conscientious about your organizational structure - you need to have clear boundaries (APIs) between engineering teams, clear definitions of ownership, decoupled deploys/releases, good internal tooling, proper monitoring & tracing across services + segregated oncall - splitting out functionality into multiple processes without doing this just yields more pain as you end up operating a distributed monolith.
Conway's law always applies, and frankly one of the initial motivations behind SOA was to give independent engineering teams autonomy & freedom to move faster. Because at the end of the day, scaling a monolith across hundreds or thousands of developers is hard from an organizational perspective; some companies choose to bite that bullet and go all in, but whatever you do execution is key - half-measures are what kill you. I am generally very skeptical of people who use bad implementations of an idea as evidence against the usefulness of the idea itself.
I can attest to the pain that just splitting up functionality can cause if not accompanied by the right org structure and tooling.
Patterns of Distributed Systems
Rauan Mayemir said “Clear your weekend” and dropped a new book, by Unmesh Joshi, that was published on Martin Fowler’s blog: Patterns of Distributed Systems. As the title suggests, Unmesh explains core distributed systems concepts in the format of design patters, and he also links each concept to real world systems where the pattern is used. I think it works quite well!
Pattern structure, by its very nature, allows us to focus on a specific problem, making it very clear why a particular solution is needed. Then the solution description enables us to give a code structure, which is concrete enough to show the actual solution but generic enough to cover a broad range of variations. This patterns technique also allows us to link various patterns together to build a complete system. This gives a nice vocabulary to discuss distributed system implementations.
What follows is a first set of patterns observed in mainstream open source distributed systems. I hope that this set of patterns will be useful to all developers.
Learning from other’s mistakes
Robin Moffat shared good read with the community: InfluxDB decided to close down services in two regions. Mistakes were made, data was lost. They did the right thing and published a detailed post mortem and the improvements they’ve made as a result. Great learning opportunity for the rest of us.
Advice on Advice
Over on the community YouTube, I talked about giving and receiving advice:
YouTube and Twitter are full of “things developers should never do”. There's endless demand for simple advice that applies in all situations. And thats not a bad thing. If there's a simple solution that works 80% of the time, this is useful information. More useful than just "it depends".
But advice givers and advice getters can do better. The best advice doesn't just solve an immediate problem. It tells you when to ignore this advice, and what to do if the problem isn't solved.
I share a very old story about a time I needed help sizing a connection pool, the advice I got, and the advice I now wish someone gave me.