What every SaaS Developer should read on vacation - 2023 edition
Last Hacking SaaS for 2023! Continuing the tradition of ending the year with recommendations for long form, deeper and hopefully inspiring content.
Happy holidays to all SaaS Developers! The winter holidays season is a time where the frantic business cycle slows down for a bit (as long as you are not in retail or finance). We can all pause and take a breath. It is a good time to reflect on everything we’ve shipped and learned in the last year - and get inspired for the next year.
Here at Hacking SaaS this is the time of year when I share longer and deeper content. One that require a bit more time to dive into, time that is hard to come by in the normal day to day rhythms. And who knows, maybe something will inspire you to try a new idea or a new project in the new year.
So grab hot cocoa, cuddle up, and start reading!
Engineering inspiration
Last year, Werner Vogel re-published a note on service-centric architecture that was used as a basis for large scale re-architecting at Amazon in 1998. Reading it today is like listening to the Beatles today - easy to think “its nice pop music, but what’s the big deal” until you realize that the reason pop today sounds so similar to the Beatles is that they invented it all those years back. So worth reading for a look at where all these micro services and workflow systems originated.
This talk is not actually long, but one of the most inspiring talks I’ve heard about engineering. If you want to start 2024 ready to change the world, worth listening to it:
Proper benchmarks can be very involved and time consuming. Especially when analyzing and trying to make sense of the results (if you just need some numbers for marketing, life can be a lot easier). The good news is that you don’t always need proper benchmarks. If you just need to know “is my architecture doomed”, there is an easier way.
Transactions
I’ve spent the last few weeks writing a primer or a refresher (depending on your age…) on transaction isolation levels both in SQL92 and Postgres. The blog has a resources section with some great and timeless content. You can spend your entire holidays week on those alone.
My top 3 resources on transaction isolation are:
Microsoft’s famous critique of SQL92 transaction isolation levels
How transaction anomalies can be used to exploit web applications
The first two are just core fundamental reading on databases. The latter is practical, readable and by one of my favorite researchers.
Speaking of favorite researchers, the day after I published my blog, Kyle Kingsbury and Peter Alvaro published a great blog on isolation levels in MySQL. Much more rigorous and detailed than mine, so really fun read for us nerds.
Daniel Abadi wrote about the difference and relationship between consistency guarantees and isolation guarantees. If you want to go deeper into this topic, I highly recommend his clear and insightful writing. His explanation refers to an older blog that he published about “time travel anomalies” that are possible even with serializable isolation, especially in distributed databases. Those are interesting to learn about in their own right.
SaaS
Cost efficiency is at the core of building SaaS - both the business and the underlying architecture. Bill Tarr is a principal engineer at AWS’s SaaS Factory and a member of our community. Earlier this year he published a fascinating blog on the topic of cost efficiency and how awareness of costs should be part of the culture.
If you haven’t read it yet, Jack Vanlightly wrote a very deep analysis of Serverless model vs BYOC model. It is a must-read for anyone operating or buying infrastructure SaaS. Which likely means all of us. While I agree with everything Jack writes about the benefits of the Serverless model (this is why Nile is a Serverless Postgres for SaaS, and not just Postgres), it is hard to deny the strength of the BYOC trend. Which makes me think that there has to be strong current that is driving the industry in the BYOC direction despite the benefits of Serverless. AWS credits model, perhaps.
And a short-but-sweet one on how to correctly handle situations where an outage of a SaaS provider is affecting the availability of your service.
VLDB 23
I talked quite a bit about the Kora paper from Confluent (which my former team published). I still think it is an excellent overview of many core techniques in building cloud native systems.
Related to transactions is Epoxy: ACID Transactions Across Diverse Data Stores. Worth reading just based on the list of authors (Michael Stonebreaker, Peter Bailis, Matei Zaharia), but the entire premise of implementing ACID across 4 different data stores using Postgres as the transaction coordinator is just super sweet. My only quibble here is that they ran 4 real-world benchmarks, and not a single one of them is for SaaS / multi-tenant workload. This is so 2010.
Another intriguing paper from the same conference is one about schema evolution in MVCC databases (nearly all modern databases, that is). Schema modifications are still considered costly in very large databases, with the potential for holding locks and halting all processing for long periods of time. The paper shows a promising alternative.
If you’ve been a long time reader of this newsletter, you know about my love-hate relationship with CRDTs - they are useful building blocks for distributed systems, but they are far from trivial to reason about. Joseph Hellerstein and his team published Keep CALM and CRDT On, where they introduce a framework for reasoning about CRDTs in the context of query guarantees, and lay out a direction for building safer and easier to use databases on top.
I haven’t had a chance to read as many VLDB papers as I’d like. If you find other “must read” - let me know!
Thats it for now, and looking forward to all of us becoming even better SaaS Developer next year!