Debugging

Everyone loves a good debugging story. Especially one that involves a well-known highly-reliable bit of infrastructure, Postgres for instance, that suddenly crashes with segmentation faults. If you love diving deep into how the software we all rely on actually works, this is a must-read.

The article, from DataDog engineering, has so many interesting details - both of the investigation and of the software involved. It starts with a seemingly simple query that causes a mysterious crash, dives into Postgres JIT (little understood, but fascinating bit of optimization) and from there to LLVM internals and ARM64 ABI.

Debugging is clearly a powerful skill for software engineers. In the age where LLMs write code, it may be the most valuable skill for a human engineer. Yet I always found it challenging to both improve and to teach. I mentored engineers who were amazing at debugging and others who seriously struggled, and I have to admit that I never managed to help someone become significantly better at it. I worked hard to improve my own debugging skills, but what improvements I made were either due to discovery of better tools (Wireshark, strace, ltrace, etc) or a mindset shift that came from watching someone else debug.

But there is hope! David A. Wheeler reviewed David J. Agan's “Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems”

The review gives a nice taste of the book, which sounds exactly what I wish I read about 25 years ago, and what I’m going to give any junior engineer I’ll ever have the pleasure of mentoring in the future.

State of Auth

Web authentication is one of these things that look simple, until you start looking into it. On the surface, you enter your username and password, the app looks it up in some database and either lets you in or gives you an error. How complicated can it possibly be?

But then you start looking and find so many details. It involves cookies (which must be secure and served from the right domain), sessions, JWT (which is considered evil but also mandatory everywhere), tokens, oauth and its friends oidc and pkce, SAML, TLS of various versions and if you get unlucky, possibly Kerberos. This is obviously a partial list.

To make things more fun, a lot of the web is built with Javascript, so naturally there are many Javascript auth frameworks. And because Javascript ecosystem is even more trend-driven than others, there’s a new and arguably improved auth framework out every 6 month.

In the video below, Theo does a great job demystifying web authentication and then summarizing the state of things in 2025.

Once you understand the basics, you may want to dig a little deeper. For example, this writeup on a vulnerability in Google’s implementation of OAuth. The complexity of the protocol means that even the biggest and most reliable implementations have some holes. And of course, discussions of vulnerabilities in protocols are almost as fun as debugging stories.

Software Design

We all love logs, and we all know that logs are at the heart of basically every system out there. But it is still worth revisiting the implications of building applications around a log of events. Especially how it reduces coordination costs and locking.

This blog, by the Restate team (formerly Flink founders), is a clear and practical writeup on the use of event log for coordinating distributed applications. The hacker news discussion is good too.

And if you want to read more on how logs can be used when building applications, the classic Turning the Database Inside Out, is always worth a look - whether its a first or tenth look.

About AI

One of my goals with this newsletter in 2025 is to help us all stay on top of AI news without getting distracted by rampant hype.

DeepSeek has been getting a lot of hype. It is an open source model out of a Chinese hedge fund that beats all other open source models and is competitive with OpenAI models from few month back. The hype is driven by few things:

• It uses novel training methods. Especially reinforcement learning. Reinforcement learning isn’t new, of course (it goes back all the way to Pavlov’s experiments!), but using it to improve LLM performance as replacement for fine-tuning.

• It is trained with different but effective methods of reasoning - multi-expert and chain-of-thought.

• It was (controversially) very cheap to train. Orders of magnitude cheaper. So cheap that it sent NVidia stock crashing.

To get ground truth, I suggest reviewing the paper which explains all the interesting new methods they used, as well as this balanced review - which points out both what DeepSeek does well and also where it is likely misleading folks and causing excessive hype.

Everyone is still trying to make agents happen. Google has a good writeup on a new “chain of agents” model that may outperform both agent frameworks and RAG. And the always brilliant Jack Vanlightly collaborated with Sean Falconier to summarize the current state of agents.

In other news, there is evidence that use of LLM can cause “brain rot” or degraded performance in software engineers. I’ve noticed that anecdotally, but there is research that points in that direction too. This short blog discusses the problem and suggests some techniques for using LLMs while still keeping your brain intact.

And if you are confused by how LLMs can be so smart and so stupid at the same time, this excellent article dives into “how LLMs think” and explains the current confusing state of things:

And… thats it! Happy February everyone.

Colt McNealy, long time active member of the SaaS Developer community published another post in his series on integration patterns: Retries and Dead Letter Queues. His blog is becoming a really great resource for anyone building microservices, especially of the event-driven kind. He also re-did the LittleHorse website over the holidays, so you may want to check it out and admire the chocolate-first design.

I restrained myself from too much AI for the holiday-reading post, but everyone is saying that 2025 is the year of the AI Agent, so we need to keep an eye on those. There’s a very comprehensive paper from Google called “Agents”, Agent design patterns from Anthropic, and if you are worried about AI Agents taking your job, this nice writeup on “70% problem” has good perspective. Last, but not least, this 2 hour video will catch you up on literally everything important about AI. I suggest watching while properly caffinated - it is super high pace and has 0 fluff.

On to security! This writeup from Wiz on how they found a serious vulnerability in a security tool is entertaining and educational. The issues they discovered - parsing bugs and prematurely stripping signatures are not uncommon. Its good to have an explicit example of how these can be exploited - in case someone asks why spend time on those small details.

And lets finish off with some engineering organization reading. I normally don’t link to LinkedIn posts, but this writeup on the various roles that principal engineers play in an organization is a great mental framework. And this blog on refining engineering strategies is pure gold. It has great patterns for working on engineering strategy - an activity that is as important as it is neglected.

Thats it for today! Don’t forget to leave a comment and let us know what topics you love most, which links were the most useful and what you want to see more of.

Technical Deep Dives

I’m not sure if this is a classic that I missed or the world’s most under-rated paper, but Out of the Tar Pit is one of the best software engineering papers I’ve read, and I’m surprised it took 18 years for me to find it. The paper builds of classics such as No Silver Bullet. It expresses the (well supported) opinion that complexity stems from managing State and that object-oriented programming is the wrong approach, and then proposes a combined Functional-Relational model as a way to reduce complexity in software systems. If you read it and say “this is just common sense and how most people build software these days”, I’ll happily agree. This paper is a great read about the reasons and thinking behind modern software design.

Marc Brooks is one of my favorite bloggers. His blogs are always technically grounded, deep and inspiring. He wrote about 20 blogs this year (how???) and it is hard for me to recommend just one. They are all amazing. The series on Aurora DSQL is obviously interesting, but so is the one on Transaction Coordination, TCP NO_DELAY and the one about leaving CAP behind. Its the holiday, so I suggest just reading everything.

If you prefer long content in video format, I highly recommend this Youtube collection of SaaS talks from re:invent. It should keep you busy for few days and keep you up to date on SaaS-related tech content - lots of AuthN, AuthZ and multi-tenancy. If you only have time for one talk, I recommend Bill Tar’s Next Generation SaaS. Another great video for SaaS developers, this time from KubeCon, is Daniel Bryant’s Platform engineering for developers and architects

My own best writing in 2024 is pg_karnak: Transactional schema migrations across tenant databases - where I dive deep into an important part of Nile’s architecture and teach you quite a lot about Postgres extensions, hooks, locks, transactions and two-phase commits.

I tried to keep this blog light on AI, because I figured everyone else is already covering that. But you may be interested in reading On the Measure of Intelligence - this is the paper behind the now famous ARC benchmark, which tries to measure the “common sense” intelligence of AI. This paper and benchmark drove a lot of the buzz around OpenAI O3 model, so its a good read in order to understand what the buzz is about and move beyond it.

Everything that Joe Hellerstein writes is great, and Looking Back on Postgres is especially great. It talks about the evolution of Postgres, and the design choices that made it not just an awesome database but a flexible data platform in its own right. If you believe that learning history helps in building the future, you should read this one.

To complement the look into the past, Modern Hardware for Future Databases goes over recent advancement in networking and storage, and how it will change the design of databases.

And of course, I can’t write an annual summary without highlighting my favorite VLDB papers of the year:

1. The Clickhouse architecture paper

2. Resource Management in Aurora Serverless - which won the best industry paper aware

3. Petabyte Scale Row Operations in Data Lakehouse - I enjoyed it because I know and admire many of the authors, but if you are working on modern data analysis systems, this paper is both interesting and important.

Soft Skills

Writing is a key skill for developers, and blogging is a fun way to improve this skill while sharing your ideas with a wider audience and advancing your career. I wish more developers blogged. If you are even thinking about blogging, Writing for Developers, is a must-read. It is basically “design patterns” for blogs, with a lot of references and ideas. I really like the short-snippet structure, since you can get a lot of the book without committing to weeks of head-down reading. I really hope this book will inspire more of you to blog - and if it does - please comment and link to your blog!

Technical leadership is hard, and perhaps even harder if it involves people management. Mistakes you’ll make as a new manager covers the most common mistakes new managers made (I made most of them and invented some new ones in my first 6 month as a manager). Decision-making Pitfalls of Technical Leaders by Chelsea Troy (possibly the best engineering blogger) adds few more traps. After reading all the mistakes, come back and read Chelsea’s advice on How not to flub technical decision making as a group.

About a decade ago, Sam Altman turned 30 and shared some life advice. It is a bit shallow, but always good to read a reminder to drink water, hang around smart people and do work that matters to you.

One of the blogs that inspired me the most this year is Lego vs Woodworking mindset. It talks about the ways software engineering and startups can be hard and can feel very inefficient - and how the inefficiency is a feature rather than a bug. The main thesis is that you can’t explore and you can’t be creative efficiently. And that efficiency isn’t the appropriate approach to every problem.

2025 Trends and Predictions

I will start by observing that, in my experience, pronouncements that a technology is “dead” rarely pan out. While Jensen Huang may be right that kids these days shouldn’t learn to code, the demise of software engineering profession is unlikely to arrive in 2025. Along the same lines, I think this time next year, LLMs will still be going strong, SaaS will do fine, cloud computing and Serverless will still be in use, RAG will continue to supplament LLMs for many business use-cases, and everyone will still complain that data engineering is a pain.

The trends that I’m most interested in, going into 2025, are:

LLM validation and observability

The essential non-determinism of LLMs make this both a serious challenge and a must-have. Without solid data collection and analysis, it is easy to fool yourself into believing your model / product is much better than it really is. On the other hand, it is also hard (and scary) to take advantage of optimizations if you don’t have a trusted system that makes sure the quality bar remains high. I believe that this time next year, we’ll look at AI validation and observability the same way we look at unit tests today - an essential tool for continuous improvement process.

LLMs are scaling down and out, not up

There is increasing talk that the ability to build larger and larger models is hitting a wall - OpenAI’s missing GPT-5 is often cited as evidence, as well as the question of whether we ran out of training data and the exploding cost of harder (and energy). With a big question mark around bigger LLMs, we are seeing fast improvements in few critical areas:
- Inference is becoming faster and cheaper. This trend is driven both by the ability to train small models with good performance and the ability to shrink trained large models without reducing quality. Either way, models are trained once and then endlessly used for inference, so anything that makes inference faster, cheaper, less energy-demanding is going to accelerate the adoption of AI with fewer negative externalities.
- LLMs are being used as part of complex systems of inference. Whether it is multi-agent system, response validation workflows, multi-step reasoning, etc - we are looking at using LLMs in more complex architecture that can make-up for some of the inherent limitations.

Relational Databases at Scale

This one isn’t a surprise - I’m interested in this every year. This space just never stops advancing. The cycle keeps shifting between scaling out, with distributed architectures, and scaling up with compute/storage decoupling and improved containers and hardware. It looks like we are firmly back in the distributed architectures side of the cycle. But this time around there is renewed effort in making distributed tradeoffs legible for application engineers.

And, thats a 2024 wrap! I’ll see you in the new year! Don’t forget to send me interesting things to read, and share what you inspires you for 2025.

AI News: Same results at 25% of the cost

If you are not familiar with quantization, you should be. AI models are all about large vectors and matrices. Typically, the numbers in these matrices are 32bit floating points. Quantization is the process of replacing these with 16 or 8 bit floating point. If the model that uses 16bit or 8bit performs (in AI terms) as well as the 32bit version, you just saved 50% or 75% of GPU, memory, network and storage requirements. This is a huge deal, but obviously depends on whether the quantized models deliver.

Now there’s a paper shows that shows:

We present a comprehensive empirical study of quantized accuracy, evaluating popular quantization formats (FP8, INT8, INT4) across academic benchmarks and real-world tasks, on the entire Llama-3.1 model family…. Our investigation, encompassing over 500,000 individual evaluations, yields several key findings: (1) FP8 weight and activation quantization (W8A8-FP) is lossless across all model scales,

FP8 quantization is lossless! Essentially meaning that you can replace all your Llamas with much smaller Llamas that use 25% of the resources and yield the same results.

If thats not huge news, I don’t know what is.

OSS Language Models

It is common to refer to language models as either proprietary (ChatGPT or Claud for example) or OSS (Meta’s Llama for example). But what makes an OSS Model actually “open source”? Models are neural networks - there is source code involved, but also weights and training data. The OSI (open source initiative) has been discussing this exact topic and has some definitions for us, but it looks like the debate is very far from over.

You can find a good writeup by Joe Brockmeyer on LWN and another, more opinionated, take by Stephen OGrady at Redmonk. Both are great reads and link to many interesting resources on the topic.

pgvector 0.8.0 is out

Postgres’ vector extension, pgvector, released version 0.8.0 with much awaited improvements and optimizations. I blogged about the new features (including a quick demo). If you are curious about pgvector in general, I also recorded an introduction video and blogged about some of the misconceptions around pgvector.

Integration Patterns

Colt McNealy, well known in the SaaS community as the founder of LittleHorse, is working on a multi-part blog series about data integration patterns.

So far he covered Saga, Transactional Outbox and Queueing patterns. Highly recommended if you are new-ish to the world of architecture patterns, and also if you are an experienced architect looking for a friendly resource to refer to.

Data Formats

I just recently learned about the Vortex file format. This blog post explains the format and compares it to Arrow. I learned quite a bit from it - both about Arrow and about Vortex. And if you are interested in learning more about file format internals, this is a short but interesting blog about parquet pruning as implemented in DataFusion.

Bluesky!

Last but not least, I’m becoming more active in the new social network - BlueSky. It now has a lively tech community and many interesting conversations going on at all times. Bluesky is built around an open protocol, which means everyone can hack their own ecosystem for this social network. The best way to get started is by following a starter pack or two. For example Cloud Native pack, Infrastructure Engineers pack or Distributed Systems pack. And of course if you joined, give me a ping and say “hello” - I love making new friends or finding new ways to connect with old friends.

So here are some new things I’ve learned this week, that I think you should know as well. And if you are not an AI fan, skip all the way to the end where I share general SaaS content that you may enjoy.

Quantization

I learned about it few weeks back, and I thought I was the last person to find out about quantization and how important it is. But then I keep running into other people who don’t know about quantization, and I keep spreading the word.

Simply put - LLMs (and neural networks in general) are essentially collections of very very large vectors and matrixes. Typically, the data type representing the numbers on these vectors and matrixes are 32bit floating point numbers. Quantization is the process of replacing these 32bit floating point numbers with 16bit or 8bit such that the model performance won’t be affected (or with minimal impact).

Sounds simple, but matters a ton - you can run a model using a quarter of the memory, fraction of the CPUs (or GPUs) and fraction of the latency. With minimal degradation. This is huge in time and money saved. Or may allow you to use models that you simply couldn’t use before.

Hugging Face has a good summary about quantization. If you browse models in Hugging Face, you’ll discover that popular OSS models also have quantized variants. When I built the Sales Insights example and deployed on Modal, Charles Frye suggested that I try replacing Llama 3.1 8B that I’ve been using with an FP8 equivalent from Neural Magic. This was really nice of him to suggest - my app became faster and my Modal bill went down.

Quantization isn’t just useful for models. It is also great for vector embeddings. If you store vectors using quantized types, you can store more dimensions in less space. Again speeding up your vector similarity search. In pgvector (Postgres vector store), using halfvec type allows you to index vectors with 4000 dimensions (instead of the previous 2000 limit). And turns out that in most cases, the 16 least significant bits don’t have much impact on finding nearest neighbors.

Jonathan Katz has a wonderful blog with benchmarks that show the power of quantized types in pgvector. I was so surprised by how few people know about such a powerful feature that I included it in my own pgvector blog.

Matryoshka Learning

Another way to have smaller vectors is to use fewer dimensions. Reducing the number of dimensions in a vector is known as feature extraction and there are several well known techniques for this (PCA, t-SNE, UMAP). But these aren’t always trivial to implement.

It turns out that you can train a model to successfully output vectors in multiple sizes, maximizing the information captured at each size. This technique is called Matryoshka Learning (after the set of Russian Dolls that are nested within each other).

The original paper is worth reading, or if you are in a hurry, Nomic (who implemented such models) wrote a short summary.

ColPali - document similarity search

Searching for relevant PDFs is both a basic example of RAG (“Chat with PDF” is a common beginner example that can be done in 5 lines of code with some frameworks) but also annoyingly complex and challenging to do well. This gets increasingly difficult if the PDF has images, tables, etc.

To do it well, you need to extract the text, extract images, tie images to text in a meaningful way, split all this into meaningful chunks and then generate vector embeddings and store them somewhere. There are some OSS libraries for some of this, but it still isn’t simple and quality isn’t always there.

ColPali is a new approach - instead of extracting text, chunking and embedding it, what if we could embed images of pages directly? It sounds like “this will never work”, but reports say that it works quite well (Caveat: I didn’t try).

You can read the announcement, slightly longer writeup on how this works by the author, and a good blog about the RAG use-case.

Not-AI but worth reading

QCon presentation titled “Building SaaS from Scratch using Cloud Native Patterns” is bound to be interesting to my readers. The Control Plane / Data Plane patterns should be familiar by now. Resource Service / API may be a new name on a familiar pattern, or perhaps it is brand new. Overall, well presented and quite interesting.

As you may know, I’m a big fan of combining analytical real-time data in SaaS products. I also happen to believe that the analytical data has to live in a seperate data store from the OLTP data that drives other parts of the site. But what do you do when you need some of the OLTP data for analytics? This is a bit inevitable and can require some annoying real-time ETLing. TinyBird made this a bit easier with their Postgres integration. Worth checking out if thats your stack.

1. Everyone always rolls their eyes when I say “the documentation is really good”. But… the documentation is really good. You can become quite proficient by reading nothing else. The documentation has one huge advantage over almost every other resource - it has versions that match releases of Postgres. You can even view different versions of the same page! Not all information remains true or relevant over time, and it is sometimes difficult to match blog posts with the version of Postgres they reference. With the official docs, it is very easy. Besides, the docs have something for everyone - beginners and advanced, developers and admins, performance and tools, internals and extensions. Did I already say that the docs are really good?

2. I’m a nerd, and as such, I’m endlessly curious about how things work under the hood. You don’t really *need* to understand Postgres internals, but it does not hurt either. There are two good internal books: The Internals of Postgres by Hironobu Suzuki and Postgres 14 Internals by Postgres Pro. They cover different aspects in different ways and both are fun to read.

3. Too busy to read internal books? Want to learn the important bits in about 30 seconds and an easy cheatsheet? Jeremy Scheneider’s Postgres Happiness Hints is for you.

4. If you are especially interested in database administration, I’ve heard good things about Postgres 16 Administration Cookbook. Although, with all the managed and serverless Postgres around, perhaps administration is going out of style?

5. If you are using Rails (quite popular for SaaS) and even if you are not, High Performance Postgres for Rails has good advice for developers using Postgres. The author sometimes hangs out on the SaaS Developer Slack, if you have questions.

6. Nile’s documentation has a short but hopefully useful section on Postgres with basic information and lots of examples. We also have a section on pg_vector, the popular extension for vector search.

7. If you are into blogs, you should know about the Planet Postgres aggregator - one feed with (almost) every Postgres blog. Most of my favorite bloggers appear there on a regular basis. I enjoy reading Shayon Mukherjee, Andrei Lepikhov, Jeremy Scheneider, Peter Eisentraut, Jonathan Katz and probably a bunch more that I don’t remember at the moment. Which is why the aggregator is so helpful.

8. If you are into Youtube, my favorite channels are Hussein Nasser, who covers many developer topics but has a lot of very engaging Postgres videos, and PostgresTV, which is more of podcast style and can be harder to follow but has great guests and conversations.

9. Depending on where you live, your local meetup can be a great resource. You can learn a lot from the presentations (usually monthly) and even more from chatting to people and making friends. San Francisco has virtual meetups, so you can join even if you live a bit farther away. You can find local events in the community events page and user groups page.

10. And last tip: You need to upgrade.  Why? depesz can tell you at

    https://why-upgrade.depesz.com/ - just enter your current version and the one you are considering, and you’ll get plenty of good reasons.

With all those resources, I hope this made your learning journey a bit easier. Did I miss your favorite resource? Comment and let us know!

In-person SaaS Community Meetup

Since the last newsletter, we’ve had our second-ever in-person meetup. Thanks to AWS SaaS Factory for supplying the space and the Pizza. It was great meeting the community and more than a few new faces.

If you’d like to know when the next meetup will happen, your best option is to join our Slack or community meetup. We also post announcements on Twitter and Linkedin, but you never know what the algorithms will do.

We opened the event with Madhukar Thumma from AWS SaaS Factory, who introduced us to the SaaS Builder Toolkit. This OSS framework lets you SaaSify your application in just a few lines of code. It integrates with various authentication, authorization, and billing services, including AWS services and third parties. They are also looking for additional third-party vendors to integrate with, so feel free to join our Slack and DM Madhukar Thumma and/or Bill Tar about SBT.

Then, it was time for the main speakers of the evening, and thanks to Bill Tar, we even had an impromptu iPhone-based recording of the talks.

We had Colt McNealy talk about his approach to observability and how to use workflow engines to get better insights into your business processes—especially when they go wrong. The talk included a live demo and very funny examples.

Kshtiji and Arsh from Orb discuss the challenges surrounding usage-based billing and how Orb evolved its architecture to tackle one of the biggest challenges—timely usage alerts.

It was a fantastic community event, and we hope to see you there next time!

Postgres Conference - Silicon Valley

I spent last week in PostgresConf SV - the largest Postgres conference in North America. Conferences are a good way to keep in touch with what’s new in the ecosystem and what topics are on people’s minds.

Few of my event highlights:

• AWS’s Jignesh Shah gave a talk about AWS’s contributions to the community. In one of the slides, he showed which EC2 machines are recommended for Postgres. He also mentioned the trusted language extensions, which allow loading extensions (in certain languages) to AWS’s managed Postgres.

• I was surprised and impressed by the wide range of deployment options I’ve heard mentioned by the audience - from the old-school installations in traditional datacenters, all the way to containers on K8s in cloud vendors.

• Not surprisingly, there were many genAI related talks. Some focused more on getting good performance out of pg_vector, but most taught the audience to build an end-to-end solution where Postgres was a key part of a larger system.

• I gave two talks. Both were well attended and I also enjoyed the deep and thoughtful questions from the audience. First talk was about schema upgrades (also known as “migrations) and covered some well known issues, as well as Nile’s approach to distributed upgrades. Second talk was an overview of transaction isolation, demonstrated with cats.

Catching up on SaaS news!

In case you missed it, there was a fun little security scare at the beginning of the month, where a version of SSH with a backdoor was almost shipped in popular Linux distributions. Luckily, the sophisticated backdoor was caught in time by Postgres core committer Andreas Freund. Both the backdoor and the discovery are fascinating. I recommend reading the original bug report, the high level summary (with diagrams) from ArsTechnica, and the deep dive with many links from Sam James.

If you haven’t been following HashiCorp vs OpenTofu, you may want to do so - especially since so many SaaS companies use Terraform for deployments. Few month back HashiCorp re-licensed Terraform to a more restrictive business license. Since a few cloud deployment providers rely on Terraform code to some extent, this caused immediate concern and the formation of a Terraform fork called OpenTofu. Recently, Matt Assay published an article accusing OpenTofu of essentially stealing Terraform code. HashiCorp sent cease and desist to OpenTofu and OpenTofu wrote a detailed breakdown denying all allegations. The New Stack has a good article covering the situation.

Vercel, the company behind the popular Javascript framework NextJS and one of the leading frontend hosting vendors, has taken a step back and removed the server-side rendering at the edge functionality. Their developer advocate explains the reasons, essentially - it makes no sense to render at the edge unless your database is at the edge. And unfortunately, most databases are not.

Nile, the company I founded, is moving a tiny bit closer to the edge by introducing new regions and an improved SDK that supports connecting to any database in any regions using just the credentials. I recorded a short video explaining how to use the new SDK and demonstrating why and how to pick new regions for your SaaS.

New and exciting distributed testing company, Antithesis, is offering a framework for determenistic simulation testing. They recently published a video of an old talk, explaining what they do, how they do it and why it is valuable.

Oldie-but-goodie blog from Netflix on CPU isolation between containers and optimizing container placement. It is interesting to read, but worth noting that if you are running on ARM servers (and Graviton on AWS is a great deal), then everything it says about NUMA is irrelevant. There is no NUMA on ARM processors.

If you are in an engineering leadership position, this article on giving tough and honest feedback may be interesting and perhaps even contain useful ideas.

Inspiring article on what it takes to build something of high quality, something users truly love.

On SaaS Developer YouTube - Cost Optimization

On the SaaS Developer YouTube, I interviewed Adam Shugar, founder of DashDive. We talked about his journey to build DashDive - SaaS for optimizing cloud costs. We discussed the close relationship between observability problems and cost optimization problems (and no, I’m not talking about the latest invoice from DataDog). We also talked about technology trends, whether or not a startup should use K8s, his experience with YCombinator and cost optimization advice.

Now on to SaaS news!

News from Re:Invent

I know, I know, re:Invent is *so* 2023. But I finally got recommendations for the best parts.

Bill Tarr, SaaS advocate at AWS SaaS Factory joined the SaaS Developer YouTube to discuss the SaaS Factory, Re:Invent news, per-tenant cost tracking and much more.

Bill also wrote a blog post with the re-invent announcements we should care about. I highly recommend reading it and following the links. I’m sure at least one announcement will be a game changes for every reader.

Observability plays a key part in all SaaS architectures, not to mention the observability companies that are part of this community. If you are curious about observability news from re:Invent, AWS conveniently summarized things in one blog.

And if you want to combine the two hottest SaaS topics, costs and observability, the spicy Charity Majors talks about the ever increasing costs of observability. I’m definitely feeling it, and I suspect many of you do as well.

Bill also recommended few good re:Invent talks that were published on YouTube. The first is by the Amazon Prime Day team - one of the highest scale operations on the planet. Prime day is like Black Friday, only orders of magnitudes worse. One day, tons of traffic and failure is not an option. They talk about techniques they use to avoid overload and collapse of their service.

The second talk is by Bill himself. Maybe not quite as exciting as Prime Day, but a lot more useful for us, normal SaaS developers. Bill talks about architecture pitfalls for SaaS, and I can’t imagine a more relevant topic.

OK, we are done with the videos for this newsletter. The rest is text.

CIDR 2024

CIDR, one of the largest database conferences, and one with a strong reputation for combining insights from academia and industry. It took place in San Francisco in January and while I’m not even close to catching up, I did find few noteworthy talks for y’all.

MotherDuck published a paper on their cloud architecture and their client. MotherDuck is “DuckDB as a service”, which is pretty interesting because DuckDB itself is an embedded data store. Their architecture of choice is Serverless store with shared storage layer and dedicated compute that scales up and down - similar in spirit to the Aurora architecture. They touch in passing about rebalancing concerns and challenges in allocating compute resources that align with query plans, but unfortunately don’t go into the depth I’d love to see.

The always fantastic, Pat Helland, gave a talk with the provocative title “Scalable OLTP in the Cloud: What’s the BIG DEAL?” I’m going to spend a lot of time thinking about this paper. Pat Helland tries to understand the theoretical limits on scaling OLTP systems, argues that the limits are a result of the expected transactional semantics, and suggests that the key to the solution is cooperation of the app and the DB. Ideas around app/DB cooperation have been raised by various research groups for the last few years, and it is exciting to see them evolve.

Murat, one of my favorite bloggers, wrote a short blog that clarifies Pat Helland’s and provides perspective. You may want to read it before reading Pat’s paper - Murat’s blog helps orient around the paper.

Murat also blogged about my favorite topic - Postgres. He discusses Postgres origin story and some of the key features that he sees as the sources to its success.

InfoQ on the changing face of Serverless

I am endlessly fascinating by the evolution of Serverless products and architectures. From the controversial FaaS, through the “everything is serverless” marketing hype phase… I feel like we are finally getting to a point where the serverless landscape offers practical solutions to real problems faced by SaaS developers - elasticity, flexible costs, low latency, geographical distribution.

This article by Birgim Ibryam, discussed the changes in the serverless ecosystem. He sees a trend toward “hyperspecialization” of serverless - a move away from the broad offerings of the large cloud providers. He talks about the basic primitives of serverless architectures and how they are currently in the process of unbundling by large number of providers. And he helpfully includes a map of the space.

And related, InfoQ published my presentation from last May where I discuss different serverless data stores, and the architecture patterns behind them. 7 month is an eternity in cloud databases, so this talk is worth revisiting.

Marketing to developers

For obvious reasons, I’m interested in how different companies market to developers - an audience that is famously resistant to marketing. Sentry is one of my favorite observability vendors - great developer experience all around at a very fair price. They discuss their journey and lessons learned. Starting with the awesome title (marketing lesson in itself):" You Suck at Marketing.

Some companies see Product Hunt as a critical part of launching new products. Other’s see it as a distraction with just marketers marketing to themselves. No matter how you see it, if you look at the most successful products launched there - you can find some SaaS-y gems. They announced their 2023 Golden Kitty awards and it is worth a Quick Look.

And you can’t mention marketing to developers without mentioning Hacker News. Let’s close off this newsletter with a data scientist who tried to optimize for hacker news success. You judge if this has real insights or if this is closer to astrology.

2024 - Lets Go!

Fireship, a popular tech YouTube for web developers has a short and entertaining trends and predictions video:

I’m not sure I’d sign off on all the predictions there (I gave up on predicting anything crypto-related a decade ago), but it is safe to predict that the AI trend will keep going strong.

A trend that Fireship didn’t mention is Postgres. In 2023 Postgres made it to the top of the DB Engine ranking, making it the “DB of the Year”. Not for the first time either, Postgres was the DB of the year 4 times in the last decade.

Jonathan Katz, one of the few top core contributors to Postgres wrote a fantastic blog post on the direction the Postgres community is taking in 2024 and beyond, along various key dimensions: Availability, performance, developer features, security, extensions and community. Great writeup and well worth reading.

The discussion in Hacker News is also fantastic and well worth reading. It has a lot of wishlist items - things folks would like Postgres to do. This includes industry-standard replication formats, extensible searching, better handling of connections, better availability options and more. One comment, responding to “why would anyone care about 1s downtime” is especially good:

I run a service that sells theatre tickets and, far more critically, a service to check if a ticket is valid or not.

Tickets are checked, by scanning a QR code, as people walk though the door. Not only do people hate sitting in their seat waiting (or worse, standing in line waiting)... you also need to pay a couple hundred employees to stand around and do nothing while waiting - ballpark cost of wages might be $5 for every second you can cut from the process of people walking through the door. On top of that wages cost, people waiting for the show to start don't buy drinks at the bar. A lot of events make more money on drink sales than ticket sales... often a high percentage of ticket sale revenue goes to whoever has a copyright claim to the show. They often get a cut of drink sales too, but it's a smaller cut.

This continues for many more paragraphs, going into the nitty gritty details of how latency sensitive every aspect of show biz can be, and how this is handled today. If you love knowing these nerdy behind the scenes bit, you’ll love it.

Elsewhere, folks are wondering if “just enough availability” is the current trends, as CTOs with tight budgets discover they’d rather not pay for all those 9s. Kelly Sommers shared the observation on Twitter, and a bunch of folks agreed.

Costs of Cloud

I interviewed Ryan Worl from WarpStream, and he shared his insights on how data-intensive applications can become significantly cheaper to operate by being smarter about how they leverage S3:

This aligns nicely with an older article I’d read that did an analysis and discovered that cloud costs are not dropping as the author expected.

If anyone has more recent data, I’d love that. I think this matches what I’ve seen as well - the big cost savings in the cloud come from watching the new cloud products (and new features in existing products), discovering those that offer better price/performance for your use-case and jumping on those. Examples include moving from the standard EC2 machines to the graviton based ones (or even AMD), moving from GP2 to GP3 storage, replacing NAT with EC2-based solution, and apparently finding good ways to use S3 is up there as well.

I think “cut costs” is on everyone’s new year resolution list (or on the normal todo list). I hope to share more great suggestions and learnings from the community. It is near impossible for any one SaaS developer to follow all the new cloud offerings and infrastructure SaaS to figure out the best deals - but together we can do it!

Here at Hacking SaaS this is the time of year when I share longer and deeper content. One that require a bit more time to dive into, time that is hard to come by in the normal day to day rhythms. And who knows, maybe something will inspire you to try a new idea or a new project in the new year.

So grab hot cocoa, cuddle up, and start reading!

Engineering inspiration

Last year, Werner Vogel re-published a note on service-centric architecture that was used as a basis for large scale re-architecting at Amazon in 1998. Reading it today is like listening to the Beatles today - easy to think “its nice pop music, but what’s the big deal” until you realize that the reason pop today sounds so similar to the Beatles is that they invented it all those years back. So worth reading for a look at where all these micro services and workflow systems originated.

This talk is not actually long, but one of the most inspiring talks I’ve heard about engineering. If you want to start 2024 ready to change the world, worth listening to it:

Proper benchmarks can be very involved and time consuming. Especially when analyzing and trying to make sense of the results (if you just need some numbers for marketing, life can be a lot easier). The good news is that you don’t always need proper benchmarks. If you just need to know “is my architecture doomed”, there is an easier way.

Transactions

I’ve spent the last few weeks writing a primer or a refresher (depending on your age…) on transaction isolation levels both in SQL92 and Postgres. The blog has a resources section with some great and timeless content. You can spend your entire holidays week on those alone.

My top 3 resources on transaction isolation are:

• The famous Serializable Snapshot Isolation (SSI) paper

• Microsoft’s famous critique of SQL92 transaction isolation levels

• How transaction anomalies can be used to exploit web applications

The first two are just core fundamental reading on databases. The latter is practical, readable and by one of my favorite researchers.

Speaking of favorite researchers, the day after I published my blog, Kyle Kingsbury and Peter Alvaro published a great blog on isolation levels in MySQL. Much more rigorous and detailed than mine, so really fun read for us nerds.

Daniel Abadi wrote about the difference and relationship between consistency guarantees and isolation guarantees. If you want to go deeper into this topic, I highly recommend his clear and insightful writing. His explanation refers to an older blog that he published about “time travel anomalies” that are possible even with serializable isolation, especially in distributed databases. Those are interesting to learn about in their own right.

SaaS

Cost efficiency is at the core of building SaaS - both the business and the underlying architecture. Bill Tarr is a principal engineer at AWS’s SaaS Factory and a member of our community. Earlier this year he published a fascinating blog on the topic of cost efficiency and how awareness of costs should be part of the culture.

If you haven’t read it yet, Jack Vanlightly wrote a very deep analysis of Serverless model vs BYOC model. It is a must-read for anyone operating or buying infrastructure SaaS. Which likely means all of us. While I agree with everything Jack writes about the benefits of the Serverless model (this is why Nile is a Serverless Postgres for SaaS, and not just Postgres), it is hard to deny the strength of the BYOC trend. Which makes me think that there has to be strong current that is driving the industry in the BYOC direction despite the benefits of Serverless. AWS credits model, perhaps.

And a short-but-sweet one on how to correctly handle situations where an outage of a SaaS provider is affecting the availability of your service.

VLDB 23

I talked quite a bit about the Kora paper from Confluent (which my former team published). I still think it is an excellent overview of many core techniques in building cloud native systems.

Related to transactions is Epoxy: ACID Transactions Across Diverse Data Stores. Worth reading just based on the list of authors (Michael Stonebreaker, Peter Bailis, Matei Zaharia), but the entire premise of implementing ACID across 4 different data stores using Postgres as the transaction coordinator is just super sweet. My only quibble here is that they ran 4 real-world benchmarks, and not a single one of them is for SaaS / multi-tenant workload. This is so 2010.

Another intriguing paper from the same conference is one about schema evolution in MVCC databases (nearly all modern databases, that is). Schema modifications are still considered costly in very large databases, with the potential for holding locks and halting all processing for long periods of time. The paper shows a promising alternative.

If you’ve been a long time reader of this newsletter, you know about my love-hate relationship with CRDTs - they are useful building blocks for distributed systems, but they are far from trivial to reason about. Joseph Hellerstein and his team published Keep CALM and CRDT On, where they introduce a framework for reasoning about CRDTs in the context of query guarantees, and lay out a direction for building safer and easier to use databases on top.

I haven’t had a chance to read as many VLDB papers as I’d like. If you find other “must read” - let me know!

Thats it for now, and looking forward to all of us becoming even better SaaS Developer next year!

Now, let’s get back to our regular hacking.

Collaboration without CRDTs

Collaboration is still a major trend in SaaS. Both because this is how humans want to work and because if you do it right, it can lead to significant growth in your active users - a key SaaS KPI.

The problem with implementing collaboration was always that to minimize latency, you don’t want to synchronize and queue up every single mouse movement or keystroke that each user makes. Ideally, you want to let each user work as if they are completely alone and then merge the results—kind of like how git handles commits.

Until very recently, the gold standard for merging results was using CRDTs. And good papers were written about applying CRDTs in various real-world scenarios. My favorites are Martin Klepmann’s Local First and Christopher Meiklejohn’s Red Wedding Paper.

However, despite all attempts at friendly SDKs, CRDTs remained challenging to reason about and use in practice. Recently, smart engineers started asking whether anyone uses them in real life and, if not - what the best collaborative SaaS does instead.

“Collaboration with no CRDTs” is a great TLDR on the topic - summarizing when you need CRDT and what you can use instead, with tons of real-world examples. The Hacker News discussion is surprisingly informative as well.

Intelligent Routing

The holy grail of SaaS performance is to be as fast as running the same software locally on your machine. If your app is in `us-west-2` and your user is in Ipswitch, UK, it is unlikely to happen due to network roundtrip latency.

With solutions like Cloudflare Workers, you can deploy your application to the edge. Companies like Vercel are trying to make this deployment super easy.

But… if your database is still on `us-west-2`, did you solve anything? The unintuitive answer is that you probably did. If not, you may be able to make a few changes in your app architecture and get significant improvements in the latencies your users perceive.

Th3o has a short and simple video explanation of the problem and why you don’t need to move everything to the edge right away.

Netflix has an in-depth InfoQ presentation on massively reducing latencies without storing data on the edge - by using CDN as a proxy.

AWS Re-invent

It was last week, but I only partially caught up. Topics that showed up on my radar are:

• Low latency S3

• Limitless Aurora

There was probably a lot more! Help me catch up by sharing the best Re: Invent announcements in the SaaS Developers Slack.

New on SaaS Developer Youtube

Where do these identifiers come from?

Either from sequence generators (for example, database identity columns) or various types of UUID, random 4-bytes that are virtually guaranteed to be unique. Each method has its pros and cons, which can get somewhat nuanced.

Since changing identifiers for existing data is challenging, picking the method of ID generation is one of the earliest “one-way doors” when developing new services.

What SaaS Developers should know about UUID

Not surprisingly, this topic pops up in the SaaS Developer Slack regularly and leads to lively conversations. Here are some of the community’s favorite resources on the topic of UUIDs.

Andrew Atkinson recommended:

• Haki Benita’s “Unconventional ways to index UUIDs in PostgreSQL”

• Going deep on UUIDs and ULIDs

• UUIDs vs. Serial for Primary Keys - What’s the right choice?

Lucas Stephens recommended:

Segment’s blog post on the history of UUIDs. Segment also has an OSS library for generating sortable UUIDs (although today, you should just use UUIDv7, which is standard and implemented in most languages).

The blog post explains the importance of the library - sorting and order can be desirable traits in IDs for application and optimization reasons. Historically, this was a challenge with UUIDs. Thankfully, there are now many variations of sortable UUIDs.

Lalit Pagaria recommended:

Instagram’s blog on generating unique identifiers in a sharded Postgres system. It has one of the clearest comparisons of the pros/cons of various solutions. Instagram’s post included a link to a blog about Twitter’s Snowflake - a mostly monotonous ID generation service.

Specifically for Postgres users:

• An overview of the UUID standard and its versions and how to generate different types of UUID in Postgres.

• A performance concern about UUIDs that applies to MySQL but not Postgres. And you may also want to read a related discussion that popped up when I posted this on Twitter the other day.

My highlights from the Twitter discussion:

Many, many, many folks mentioned that UUIDv7 and ULID are both better choices than the common v4, since they are semi-monotonous and sortable.

A lot of folks shared a great blog by Vlad Mihalcea, where he clearly explains some of the issues with UUIDs and recommends TSIDs instead. TSIDs are not only sortable, they are also twice more compact, so they are more efficient in memory.

A repeated misconception seemed to be that UUIDs are basically very large strings. This is fake news. On almost every system, UUIDs are 16 bytes, not 36 characters.

Little-known fact is that modern CPUs and modern compilers can compare UUIDs in a single operation, so sorting and indexing is as efficient as using numbers. For the extra nerdy, you may want to check the conversation where Jeremy Schneider and I exchange assembly code that proves this.

Stephen Garland clarified that UUIDs still have performance implications when inserting and shared links to his benchmark script, data generator, and results.

Richard Banffy explained one under-rated benefit of UUID:

Since a UUID will probably be needed anyway (because one doesn’t simply expose a sequential PK), it makes sense to not have a sequential PK in the first place.

While not as critical as “thou shalt never save plain-text passwords to a table” it could result in involuntarily disclosing data such as insertion order of inclusion or gaps with hidden records. A number of things can be inferred from sequential ids.

Postgres Performance for Rails

Andrew Atkinson took a Rails web application that was struggling with load and optimized it to handle over 9000 HTTP requests per second with an average latency of 35ms end to end. Handling a much higher load on a smaller RDS instance with lower latencies.

He then shared his expertise by writing a book: "High-Performance Postgres with Rails." You can watch my video interview with him below or read the blog post he wrote following our conversation.

Andrew often hangs out and answers questions on the SaaS Developer Slack, ping him if you want to discuss Postgres, Rails or maybe even get a discount on his book.

As I reviewed the community activity in the last 3 month, looking for individuals who stand out, one of the things I looked for is relevant content in the community Slack. Relevant content can be questions, opinions, links to blogs or tools. Developers linking to their own blog/podcast is a welcome contribution, but links to content for marketers, sales people or lawyers is not. Most off topic conversations are actually pretty much on-topic, but not my question about favorite music!

Between Feb and May, we had roughly 192 relevant messages in the community. Between May and August, I roughly counted 335. I felt a lot more energy and activity in the community recently, and the numbers support this feeling. Interestingly, the number of active people didn’t change much (~150 active members, and ~40 actively posting), but the most active members are a lot more active.

It is great to see the community becoming more active, more engaging, more useful and more fun. There are many people who contributed to this, but two stand out:

Colt McNealy

Colt is by far the most active community member. He initiated some of the most interesting conversations - about K8s, micro services, open source and many discussions about the tradeoffs of the different SaaS deployment models. He also actively responded to other’s questions and opinions on a huge variety of topics. Always bringing curiosity, perspective and facts to the discussion.

Colt recently launched his startup LittleHorse, where he is building the future of micro services orchestration on top of Kafka Streams. He shared his journey and his passion for performance and Kafka Streams in the SaaS Developer YouTube:

Daniel Chaffelson

Daniel is currently a data engineer at TinyBird , serverless real-time analytics company. He has deep experience in the data space, but seems to know almost everything. In the SaaS developer community, you can find him contributing to conversations about Python best practices, Serverless technologies, enterprise security requirements, SaaS deployment models, design patterns, micro services architecture and a lot more.

He doesn’t just have a wide area of expertise, he also had a very wide-ranging career. Daniel was a software engineer, an SRE, a founder, solutions architect, director of field engineers and a sales engineer. Having worn the engineering hat, the manager hat and the customer-facing hat, he brings valuable perspective to every conversation. SaaS is never *just* about the technology, but it is never *just* about the business either.

5 Good Links

Speaking of heroes, the story of VaccinateCA, is a fantastic story of engineering, entrepreneurship, hacking and getting shit done. And it ended up doing enormous good for millions of people.

Charity Major wrote the ultimate guide to communicating when trust is low. Charity’s writing is always great - practical and no BS. Just acknowledging that some relationships require a bit more care in communication, without any judgement or blame about this situation, is a good thing in our industry.

Great war story from Yugabyte on how they diagnosed and solved a mysterious issue with high latency IO. Always great to learn new tricks from someone else’s problems.

Every SaaS has an authentication problem. Kent Dodd asked on Twitter:

Anyone here have an end-to-end test that involves social login (OAuth) that actually goes through the full flow? Pretty sure the answer is no, but if so, how do you get around captchas?

The responses are enlightening, and worth keeping in mind next time you encounter a broken authentication flow.

Gitlab wrote about day-2 operations, what it takes to smoothly evolve and maintain large systems, and the ways Gitlab handles those challenges. I love the concept and enjoyed reading about the tools and techniques Gitlab uses.

I only recently noticed how many people see July as another January. Even though there are no official holidays or events, you can feel some echoes of January in the air. Nordstrom has their semi-annual sale. People take time off and reflect. And businesses publish trend reports.

DevOps and Cloud InfoQ Trends Report – July 2023

InfoQ publishes several trend reports, but the DevOps / Cloud report is probably the most relevant for this community. One thing I love about how they report is that they split technology by adoption stage. This way, the intelligent engineer can immediately find the technologies that match their risk tolerance.

If you are truly cutting edge, perhaps you already have a platform engineering team.

If you are an early adopter, you probably have a dataOps team, a service mesh, and handle documentation as code. Where do you fit? Let us know in the comments or on the community Slack. I’m betting 90% of us are early adopters.

Interesting to note that Serverless Databases were included as “over the chasm.” I think we are just at the beginning of the adoption cycle. Many services call themselves “serverless databases,” and perhaps some are seeing adoption by the early majority (I doubt it). But in my opinion, very few deliver an actual serverless experience, and we are just starting to explore what “serverless database” even means.

If you are interested in hearing me and Ram discuss how Serverless Databases will evolve, check out our video:

2023 Stack Overflow Developer Survey

The Stack Overflow report may be the ultimate trend report. The survey asks the largest number of developers, “What’s cool? What do you actually use?”

For me, the most significant thing this year is the rise of Zig. It came out of seemingly nowhere and is now at the top of the “highest-paying tech” list. Probably biased due to small numbers, but I’ve been hearing more and more good things about Zig this year. Many developers prefer it over Rust for system and high-performance development. Keep an eye on this language, and maybe jump on the hype train early and snag one of those high-paying jobs.

Postman’s State of APIs

Postman’s annual survey of the API landscape is highly relevant for SaaS developers. This year it is even more relevant with the inclusion of API monetization and generative AI APIs.

The APIs technology section is a helpful way to track the quick rise of GraphQL and the slow decline of SOAP. They currently have about equal adoption, but if you are working on a new product, it is pretty clear which one you’ll adopt.

The tools section can be a useful way to check if you are missing out on anything interesting that you should be using. But… this year, I think they are missing a tool! I recently adopted Thunder - a VS Code extension that offers a small part of Postman functionality and has proven convenient for checking simple things. I use it to validate examples in my docs.

Data Engineering Landscape

Since a good portion of our community is building data SaaS, you may have already seen LakeFS’s state of data engineering report. If not, check it out… and make sure you are included. It is fun to quibble over how they drew the boundaries, but having a map is essential in this rapidly changing and highly disaggregated space.

How fast is the space changing? The report is about a month old, and one of the companies on the map has already been acquired since the report was published. Congrats!

How to write a blog

Our community is all about sharing, and blogs are a great way to share thoughts and ideas about SaaS. This week, Robin Moffat shared a blog he wrote for engineers who want to blog. I encourage all of you to practice and share your writing on the SaaS Developer Slack so that I can feature your blogs in this newsletter.

First In-Person SaaS Developer Meetup

Thank you to everyone who attended our "Generative AI for SaaS Developers" event. It was a great success! We had ~50 attendees. We exceeded the space capacity and had to move to a larger space. We enjoyed the pizza and drinks; we got SaaS-y swag. We had good conversations, and we learned a lot about AI.

The presenters did a fantastic job educating us about the use of generative AI. Unfortunately, we had equipment failures and did not record the presentations. I anticipate more events in the future, and we will do better with recordings next time.

Meanwhile, here are a few photos:

Most of the content here is based on conversations in the SaaS Developer Slack. Join us and take part in the conversation.

Also, a quick reminder that if you are a SaaS developer, or interested in what SaaS developers talk about, and you happen to be around Mountain View, CA next week, you should join our meetup and hear how SaaS developers use generative AI.

But this post isn’t about generative AI or even the one year birthday of this newsletter. It is about the ways that SaaS Developers choose which tools to use. Lets dive in!

Which tools should SaaS developers use?

Jeffrey Sherman asked a philosophical question:

Do you see yourself more as someone who provides a service or someone who writes software?As an example of the difference, and the origin of my curiosity, I was talking to a developer who is against using no-code tools like Zappier to solve problems because they wouldn’t be using their software skills.

The community was unanimous: “Engineers solve problems. You can use any tool that solves the problem”.

Aaron Kimball wrote an in-depth answer, and then published it in his blog. He started by defining “What is SaaS?” and used this to also define the role of a SaaS developer. This is a pretty big deal!

what exactly is it you’re trying to accomplish with a SaaS business?

I really think the “S stands for Service” part of SaaS needs to be more prevalent.

[….]

So to the question of “what tools ought a software engineer feel comfortable reaching toward,” I think you use what you need to use so that you can “do what it takes to make [customers] happy.”

you don’t have to write a “weapons grade” bulletproof API to do the job, it just needs to get done; choosing the right tool for the job to dispatch it expediently is, I think, what good engineering is all about.

I love every bit of this. Although, I do think customers value bulletproof APIs too.

Relational DB or KV Store?

This was published by a former colleague of mine, and it discusses a key decision for SaaS developers - the datastore. Jos goes into a lot of history and interesting stories, so worth reading even if your mind is made up.

Wednesday Wisdom

The unloved relational database

Here is something that I have been wondering and worrying about for some time: The popularity of the distributed key/value (K/V) store at the cost of relational databases. Compulsory video here. At the start of my career, relational databases were slow, expensive, and often just not available. Consequently, almost nobody used them. Instead, we used a hos…

Read more

3 years ago · 1 like · Jos Visser

Keeping History

Jorin is a new member of the SaaS Developer Slack, and he shared his blog, where I ran into an excellent article discussing the challenges regarding storing historical data, different approaches to the problem, how this impacts schema evolution and how to handle more nuanced retention concerns. This is one of the problems that pretty much every SaaS developer ran into, and Jorin provides a great overview.

One of the most obvious uses of historic data is implementing requirements for auditing and version control in critical applications.

Having historic data also becomes invaluable for understanding and debuging complex processes. It is not only helpful for developers but also for users trying to understand the processes of the bigger system the software is used in.

From a technical perspective building solid, fault-tolerant, distributed software that interacts with external systems becomes more feasible once you have a log of change events. You can replay the events at any time to recreate the state of different sub-systems which is a big help keeping everything in sync without running into race conditions.

Complexity, K8s and Microservices

Colt decided to bring a controversial tweet and some 🍿 to the SaaS developer community:

And Lucas Stephens gave an in-depth response, that started by separating “use K8s” decision from “have 150+ microservices” decision. And rightly so - they are independent and you can make one mistake at a time.

Two parts to this:

1. Kube

2. Microservices

In my experience people who criticize Kubernetes for its complexity forget to tell you about all of their custom internal tooling they've spent years building to replicate the same functionality. Or they compare Kubernetes to something fundamentally more limited in scope (like Ansible + systemd) & think "wow why would I ever use Kube". As a whole, there's not a better open-source out-of-the-box orchestrator than Kube; alternatives are either proprietary or have less functionality (cough cough Nomad cough cough).

These people also forget that Kube is meant to be the platform's platform - you're supposed to customize & build APIs around your cluster to give you what you need. The power of Kubernetes is that it is basically an open-source API for deploying things - that's really powerful for establishing community standards & projects, as well as hiring.

As for microservices - I definitely agree with "stick to a boring architecture for as long as possible". And a boring architecture is definitely possible with Kube.

Most people don't realize that adopting a services-oriented architecture requires you to get conscientious about your organizational structure - you need to have clear boundaries (APIs) between engineering teams, clear definitions of ownership, decoupled deploys/releases, good internal tooling, proper monitoring & tracing across services + segregated oncall - splitting out functionality into multiple processes without doing this just yields more pain as you end up operating a distributed monolith.

Conway's law always applies, and frankly one of the initial motivations behind SOA was to give independent engineering teams autonomy & freedom to move faster. Because at the end of the day, scaling a monolith across hundreds or thousands of developers is hard from an organizational perspective; some companies choose to bite that bullet and go all in, but whatever you do execution is key - half-measures are what kill you. I am generally very skeptical of people who use bad implementations of an idea as evidence against the usefulness of the idea itself.

I can attest to the pain that just splitting up functionality can cause if not accompanied by the right org structure and tooling.

Patterns of Distributed Systems

Rauan Mayemir said “Clear your weekend” and dropped a new book, by Unmesh Joshi, that was published on Martin Fowler’s blog: Patterns of Distributed Systems. As the title suggests, Unmesh explains core distributed systems concepts in the format of design patters, and he also links each concept to real world systems where the pattern is used. I think it works quite well!

Pattern structure, by its very nature, allows us to focus on a specific problem, making it very clear why a particular solution is needed. Then the solution description enables us to give a code structure, which is concrete enough to show the actual solution but generic enough to cover a broad range of variations. This patterns technique also allows us to link various patterns together to build a complete system. This gives a nice vocabulary to discuss distributed system implementations.

What follows is a first set of patterns observed in mainstream open source distributed systems. I hope that this set of patterns will be useful to all developers.

Learning from other’s mistakes

Robin Moffat shared good read with the community: InfluxDB decided to close down services in two regions. Mistakes were made, data was lost. They did the right thing and published a detailed post mortem and the improvements they’ve made as a result. Great learning opportunity for the rest of us.

Advice on Advice

Over on the community YouTube, I talked about giving and receiving advice:

YouTube and Twitter are full of “things developers should never do”. There's endless demand for simple advice that applies in all situations. And thats not a bad thing. If there's a simple solution that works 80% of the time, this is useful information. More useful than just "it depends".

But advice givers and advice getters can do better. The best advice doesn't just solve an immediate problem. It tells you when to ignore this advice, and what to do if the problem isn't solved.

I share a very old story about a time I needed help sizing a connection pool, the advice I got, and the advice I now wish someone gave me.

And if you are in the SF bay area, don’t miss our first in-person meetup. We have great presentations planned about adding generative AI features to SaaS apps. Frank Greco, Head of AI at Superblocks, will share how Superblocks uses AI to transform the Developer Experience. Antje Barth and Chris Fregly from AWS will share stories and advice based on their experience working with hundreds of developers building AI products.

I summarized one particularly interesting conversation from the SaaS Developer Slack this week. In addition, I linked to many other interesting Slack conversations. Let me know what you think of this new format.

Event Sourcing for SaaS Applications

Moshe Eshel asked:

I've been seeing event sourcing all around for years, but have never seen an actual production implementation. Seems like a neat idea but can't figure how a business application actually functions this way

Lucas Stephens explained the use case but warned that it ain’t for everyone:

Event sourcing is typically paired with CQRS - you have two data stores: one that can be optimized for writes and one that can be optimized for reads.

As an example, at my current company, we offer a "proxy-workflow engine" as a service: so we proxy a significant portion of our customer's traffic & need to operate on that traffic quickly without impacting latency. We have two separate data stores as a result - the write-optimized data store, which is very difficult to query but generally has 10-20ms inserts, and the read-optimized data store, which is treated as eventually consistent & populated by a data pipeline, and is much more usable for analytical queries and is what the UI of our application points to.

With event sourcing, your event store essentially becomes the write-optimized store in this system. Then you build read-optimized projections of that data in separate data stores - this typically means writing a stream processor to populate those new stores. The obvious trade-off here is that you do double the work of maintaining two databases. I don't recommend CQRS or event sourcing unless you absolutely need it. Personally, I think the main benefit of event-sourcing (auditability) can still be achieved with your traditional RDBMS model.

we don't do event sourcing at my company, only CQRS - the concepts are closely related but separate). The hard part of event sourcing is that you're putting all of your business logic into a stream of events, which is cognitively harder to think about and change - I've seen event sourcing fail more than it succeeds, so I'm a bit biased.

But Moshe was looking for something more specific:

I'm looking specifically for a working example of event sourcing as the root of the architecture (exactly as you explain in the second part). It's not only consuming streams and processing to populate a data store but also the entire application running on top of a stream - constantly scanning it from start to ... (endless?)

So Lucas went into more details:

You could model pretty much any system as a stream of events; you just have to decide when and if it's appropriate.

A specific case where I've seen it done (and it ultimately failed) was when I worked at a life insurance tech startup. We modeled the entire flow of initiating & managing a policy as a stream of events, so every single creation/modification/deletion of the policy itself became something like PolicyHolderAddressChanged or PolicyBeneficiaryAdded instead of calls to a generic UpdatePolicy endpoint.  To query for the current policy, services would either have to get all events from the event store, order them, and compute the state in memory or query a "projection" showing the current state. We tended to opt for the projection route for performance. With a few exceptions, mostly everything is communicated via publishing an event to a queue. We chose event sourcing because we knew we needed audited history for the policies. We built & modeled that entire system using protobufs for the event definitions.

Why I ultimately consider it a failure (this startup recently shut down at the beginning of the year) is that it simply took way, way, way too much time to engineer the system like this. And from a compliance perspective, we would have been okay simply creating triggers on our database tables that inserted versions of rows into some "history" table. We had real problems when we needed to add new events to the domain or update the schema of existing events - which are inevitable changes in any system. This meant that every single projection needed to be replayed & rebuilt, and this was especially expensive for the projections we built for our analytics team, as they needed the most generic view of the data.

As our data grew, this only became more difficult. I do remember we ran several event-storming sessions to model the business before we started eng work, so if you're thinking about it, it might be useful to do this exercise.

Going into the project, I wasn't against the idea, but now that I've seen how the sausage is made, I'll always be a bit scarred from it. What is really ironic to me is that it seems most appropriate for systems with infrequent writes (fewer events, less need for projections), but the added complexity and extra work are especially not worth it in those situations. It can be argued that an unstructured event store (NoSQL) is easier to scale than your traditional RDBMS - but there's so much exciting stuff happening in more "horizontally scalable" SQL databases like Vitess, Cockroach, and Clickhouse that I don't feel compelled ever to choose event sourcing.

Not to mention, event sourcing is done so sparingly there's not a lot of open-source tooling or experience amongst devs that can help you out with things like replaying events, building projections, etc. Whereas there is a ton of tooling for the traditional RDBMS model

Worth noting that from my (Gwen) conversations with many startups, it is very common to start with exactly the simple pattern that Lucas described - triggers and a history table. It scales surprisingly well, especially if the history table is designed correctly.

Lucas’ experience convinced Moshe, and there was agreement all around.

But then Daniel Chaffelson entered the conversation:

I've seen event sourcing done in payments processing - specifically servicing the 'instant payments' standard where latency is <2s. It makes sense for this to be an event-driven system because its high volume, transactions have a limited number of pathways through the process, generally live for a very short amount of time, and tables can be materialized of the various outputs over the streams as they happen.
Ultimately though, this is just a core feature of a larger banking system, not the entire system itself, so I don't know if it really answers your question. I guess you could ask what % of a system needs to be event sourced for it to be the main architecture.

Also, I would agree if you were thinking you need event sourcing because of some kind of scale problem, technology like ClickHouse (and others as mentioned)  is solving a lot of this kind of thing without making your architecture structurally complex. We operate CH in a use case over billions of daily rows with a latency that pegs under 500ms at a couple of hundred qps on average that can spike from 5-20x that on a busy day. It requires operational finesse to maintain, but the architecture isn't complex and development over it is quick and easy.

Moshe pointed out the distinction between event sourcing and event-driven architectures. He also highlighted how both Daniel’s examples cleverly worked around the key disadvantages of event sourcing while enjoying many of the benefits:

It sounds like two separate cases.

The first you described sounds like event streaming, but on a small time scale (so there is no persistence of events or, rather, events have a short TTL). Indeed the operation and state are done/processed as stream always: Queries are aggregating the events, and there is no other data in the system - views are temporary and either aggregated in place or recreated. This is a very valid use case. Thinking of the tradeoffs, it throws away the biggest problem with event-based (data accumulation and how to deal with it) and enjoys all the benefits (one example is any stream processor that might use time windows and such).

The second part sounds to me like Event Driven architecture. Communication is done via events, but they just drive the state, which is managed internally by each component/service (in a DB?) according to their preference. It isn't event streaming. I like this architecture because it again preserves the advantages and "throws away" the problem: Events are easy to pass and provide a rich and contextful interface, but we don't have to aggregate many small pieces every time we want an answer. Instead, we use SQL/Whatever on a strong Data Store.
You can still store the events aside and replay them if needed. For example, when you want to correct something. This is a nice backup strategy that can help deal with data corruption due to a bug in handling events. I built such a system in a previous company. Sometimes our processing logic was changed due to either a bug or a client request to use a different aggregation rule. and we could delete the stored data, replay the original events, and recalculate - which was heavy, but our support loved it!

Daniel agreed and added interesting details:

I agree. It goes back to your original point that there are a lot of streaming use cases around, some on a truly massive scale, but very few actual event-sourcing architectures.

Also, perhaps it's useful to observe that the first case was building a payments system from the ground up to be a streaming ledger, whereas typically they are migrating from a mainframe or other database-as-ledger approach. And the second is taking a problem that the existing RDBMS couldn't scale to, and reworking it into a Kafka + OLAP solution.

Replay and archiving to cold storage, amongst other things, are part of what make the managed service so valuable as you note. We find customers get a lot of value from our data engineers optimizing their queries specifically for CH performance, which would otherwise require a larger and more expensive internal team if every customer had to do it for themselves, and would be probably an order of magnitude more expensive if you had to hire the unholy trinity of Kafka + Flink + Cloud developer to service it.

Ultimately I think if event sourcing was such a great solution to particular problems, 'streaming is the answer, what was the question' companies like Confluent and StreamNative would have more use-cases for it on their homepage

Moshe circled back with good news about the project that started the discussion: Event sourcing was proposed as a high-level futuristic idea, but didn’t seem to be a likely future direction for their product architecture.

Recommended reading / watching from the community:

Robin Moffat shared a rather amusing blog about the various stages SaaS companies go through when it comes to deploying and running their applications.

Colt shared an epic blog about the many ways SaaS products can be delivered. He went into more details and nuance than one typically sees when discussing the topic. Highly recommended.

And I talked about transaction isolation levels on the SaaS Developer Youtube channel:

More good Slack conversations:

Unfortunately, I don’t have the bandwidth to summarize every great discussion here (ping me if you want to volunteer?). So here are links to other great conversations. They will be gone by September, so grab them while you can.

Colt started a conversation about the incentives of cloud vendors to contribute to K8s and other OSS projects. Mitch, Aaron Kimball, Lucas Stephens, Buchi Reddy, and Moshe Eshel all shared interesting viewpoints.

Lucas shared the Kubernetes documentary, which I haven’t watched yet, but looks like a must-see:

This forked off to two conversations about (what else) cloud pricing:

Aaron Kimball made the case that cloud vendors sell mostly RAM and networking since CPU and disk space are rarely the bottlenecks. Shikhar’s workloads turned out to be CPU-bound. And in another thread, Moshe Eshel made a convincing case that cloud vendors don’t sell resources at all. They charge for resources, but they sell elastic capacity - the ability to get 1000s of machines in a click.

Our first strategy was to test out services on AWS, and once we learn the workload pattern we can cost optimize by moving to our DC. However, the capacity and stability eventually turned to moving everything to the cloud...

Jeffrey Sherman asked Do you see yourself more as someone who provides a service or someone who writes software? Moshe Eshel, Colt, Daniel Chaffelson, and Aaron Kimball all had great stories that illustrate the difference. The community is in wild agreement that developers solve problems and deliver value. Writing code is just a part of the job.

Moshe Eshel asked if anyone has experience with Yugabyte. Rauan Mayemir shared what he learned while investigating their solution, Daniel Chaffelson shared thoughts on the space and Colt added internal architecture information (and admitted that his friendship with Yugabyte leadership makes him biased).

Also, check out questions and conversations about alerting , EDR, customer dashboards , cloud marketplaces, and push vs pull in control plane architectures.

Did I mention that if you didn’t join the SaaS Developer Slack you are missing out ?

Most of the links here are worth an hour of your time, in a nice quiet spot, with your favorite beverage. I’d also put some light jazz for the atmosphere.

If you prefer lighter, shorter, more practical, or more beginner-friendly content - I promise to make it up to you in the next edition.

Relational Databases

Massively Scaling MySQL Using Vitess (Video, slides, and transcription or presentation)

The Snowflake Elastic Data Warehouse (Paper)

Cockroach DB architecture in 5 parts + their cloud architecture (Documentation + a course)

Yugabyte architecture (26 articles in 9 sections)

AlloyDB for PostgreSQL under the hood: Columnar engine (Blog)

Amazon Aurora:

• Design considerations for high throughput cloud-native relational databases (Paper)

• On avoiding distributed consensus for I/Os, commits, and membership changes (Paper)

Google’s Spanner / F1: (all papers)

• Google’s Globally-Distributed Database

• A Distributed SQL Database That Scales

• Becoming a SQL System

• Online, Asynchronous Schema Change in F1

• Large-scale Incremental Processing Using Distributed Transactions and Notifications

Non-relational and possibly non-databases

Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service (Paper)

The Cloud-Native Chasm: Lessons Learned from Reinventing Apache Kafka® as a Cloud-Native, Online Service (Paper)

Schema-Agnostic Indexing with Azure DocumentDB (Paper)

Amazon S3:

• Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3 (Paper)

• Building and Operating a Pretty Big Storage System (My Adventures in Amazon S3) (Video of presentation)

• AWS re: Invent 2019: Beyond eleven nines: Lessons from Amazon S3 culture of durability (Video of presentation)

On SaaS Developer Youtube

My interview with one of our SaaS Developer Heroes - Jeffrey Sherman.

We discussed his “Never Rewrite” philosophy and how it applies even to early-stage companies building their MVPs. We also discuss the critical design decisions engineers have to make in the early days of building the product: Tabs or spaces, pooled tenants or siloed, Micro-services or monolith... And who makes these decisions? And Jeffrey shared funny stories related to these decisions and other common scenarios when building SaaS.

Let’s start with the breaking news - The first preview release of Postgres 16 is now available. Highlights include:

• Quite a few performance improvements

• Notably - COPY from files should be 300% faster

• Logical replication from standby

• Client-side load balancing

Client-side load balancing is especially interesting since Postgres currently has a lively ecosystem of proxies. There’s a Hacker News discussion of the new release in general and whether the new client load-balancing will replace the proxies or not.

And speaking of Postgres proxies, I found an interesting new one, PgCat. It is written in Rust, and in addition to load balancing and connection pooling, it also handles failover scenarios. In general, I like it when tool names include cats.

Over in the MySQL world:

The company formerly known as Facebook published a blog describing their Raft-based distributed MySQL.

We enhanced MySQL and made it a true distributed system. Realizing that control plane operations like promotions and membership changes were the trigger of most issues, we wanted the control plane and data plane operations to be part of the same replicated log. For this, we used the well-understood consensus protocol Raft. This also meant that the source of truth of membership and leadership moved inside the server (mysqld). This was the single biggest contribution of bringing in Raft because it enabled provable correctness (safety property) across promotions and membership changes into the MySQL server.

On Data Modeling:

Hacker News is discussing when to use indexes. Shockingly, it sounds like the answer is “It depends.” Still a good discussion with a lot of interesting viewpoints and anecdotes.

Knowlo’s data modeling blog post describes how they tried to create a reasonable data model for their MVP without spending too much time on it. They started with their two MVP use cases and went from there. I found the data modeling process with DynamoDB and the help from AI quite interesting.

Knowlo’s build-in-public process is quite interesting in general. I ended up reading more posts and cheering for their young SaaS.

Ease of use vs. flexibility

When you build a platform, you need to balance two opposing requirements:

• Making things easy for beginners or developers with simple requirements. Abstracting away something they shouldn’t think about.

• Allowing advanced users to implement complex or uncommon requirements while benefiting from the rest of the platform.

It was interesting to read how these two play out in Remix, as an enthusiastic developer struggled as their requirements evolved.

In the community Slack: How to authenticate in gRPC?

Good discussion in the community Slack on the best way to authenticate in the gRPC protocol.

I was wondering if anyone could critique how we are thinking of providing authentication for a grpc-based SaaS (maybe it should be called PaaS) product. Basically, we expose an API that does useful stuff over gRPC, and we have an admin console which you can access using an SSO provider of your choice. Here's my thinking:

1. We'll go with mTLS because that's the only non-google-specific out-of-the-box grpc authentication mechanism: https://grpc.io/docs/guides/auth/

2. The client libraries require CA Cert, private key, and public key for authentication. We will distribute all three securely through the dashboard (which users login to using their SSO + MFA)

3. Cert rotation is pretty simple, just provide a new private key and public key which can be downloaded from the admin console.

4. When we want to revoke credentials (eg. if the customer has a leak), what we do is create a new CA cert + private key + private cert combo, and configure the server to accept requests signed by the new CA rather than the old one. This would be a very rare occasion.

What do you think? Join our Slack to respond, or let us know in the comments.

Scalability and Multi-tenancy

Shayon wrote a great blog post on his approach to scalability issues and how his team at Loom applied it to their database as the company hyper-scaled.

He also joined our YouTube to discuss database scalability, performance, and multi-tenancy:

“Growing this garden is quite easy. You mow the lawns, ensure everything is appropriately watered, trim here and there, and maybe a bit of mulch or fertilizer.”
Pause.
“And you need to do this every single day for 500 years”.

The story’s moral is that this is how you become an expert. You need to dig a bit deeper, understand a bit more, and learn something new. And to do it every single day of your entire career. Joining a community of developers who share knowledge and learn from each other is not a bad way to do this.

In the last quarter, two individuals have stood out in the quantity and quality of their contributions to the SaaS Developer Community. They are worthy of a big shoutout and our appreciation.

Jeffrey Sherman

Jeffrey Sherman drives scaling and performance at ActiveCampaign through his role as a Staff Engineer. In addition to hands-on coding, he teaches problem decomposition, iterative delivery, and his philosophy of never doing greenfield rewrites.

If you hang out in the community Slack or follow this newsletter closely, you may be familiar with his blog https://shermanonsoftware.com or his podcast https://www.neverrewrite.com/podcast.

One of his earliest conversations in the community was around scalability - especially how you ensure production upgrades won’t cause issues to your biggest customers. Jeffrey’s LinkedIn tagline showcases his customer-centric view of performance engineering: “I help SaaS Companies keep their biggest clients from churning.”

Lalit Pagaria

Lalit is the founder of Oraika, an AI feedback analytics startup. He has an extensive professional background working with companies such as Brocade, Ola, and Careem (Uber). His expertise covers diverse areas, including technical architecture, natural language processing (NLP), identity solutions, infrastructure security, and cost optimization.

Lalit does not (yet) have a blog, but identity, security, and cost optimization are some of the hottest topics on the SaaS Developer Slack - and Lalit generously shared his advice and experience.

As an ML expert, Lalit shares useful lines with practical suggestions, thought provoking ideas and exciting tools - both in the community and on Twitter. Very few people are at the crossroads of ML, identity, security, and costs, which means that Lalit has a unique angle on cutting-edge technology. If you haven’t yet read the article he shared about Prompt Injection, this is a great time to do so.

You, too, can be a Slack Developer Hero.

We’ll select the top contributors every quarter—those who actively share ideas, problems, concerns, suggestions, and excellent links with the community. And we’ll show our appreciation with public shoutouts, interviews, exclusive swag, and whatever else I can think of.

Meanwhile, say kudos to Lalit and Jeffrey!

Few More Good Links

Atlassian shares how they re-architected their SaaS product and made it more scalable.

Josh Long, a rather famous Java community champion discusses the Rise of Postgres.

While OtterTune discussed in detail why Postgres MVCC is the worst. If you don’t know what’s MVCC and why it matters, don’t worry - the blog has one of the best explanations.

New on the SaaS Developer YouTube

Last week we published an interview with Chinmay Soman, who is not (yet?) a community hero, but is occasionally active on our community Slack. We discuss the many use-cases of real-time data in SaaS, how to pick the best technology for each use-case and the best way to handle conflicting requirements.

What makes communities work so well as a way to accelerate learning is the people who share their experiences. Look at this chain of events: In 2019, someone attended a meetup and shared how they solved a problem, In 2020, Sachin applied what he learned to a similar problem and in 2023, Sachin shared his learnings with 300 attendees of RTA Summit. Those who benefit from the community experience pay it forward by sharing their own experience.

SaaS Developer Heroes

To encourage more sharing in the SaaS Developer Community, we (Nile) announced the SaaS Developer Hero program for recognizing and rewarding developers who contributed above and beyond to our community. You can read more details here and if you have questions, we can discuss them in the community Slack.

Now, true to the spirit of the community, let’s look at new content from members of our community. If you find these blogs insightful, ping the authors on our community Slack and tell them! If you disagree or have questions, start a conversation - I’m sure the entire community will benefit from the discussion.

What kind of developer are you?

We are all SaaS developers, but it doesn’t mean we are all the same! Help us understand the community with one simple question:

Multi-tenancy Architecture with Kong API Gateway

Rick Spurgeon shared his very in-depth blog on how to use Kong API Gateway in multi-tenant architectures. He starts with high-level ideas around the control/data plane model and then goes into specific deployment models and how they match different types of multi-tenant architectures. I learned that Kong approaches multi-tenancy as an authorization problem, which is a common pattern and definitely addresses the data protection aspects of the problem (but not the “noisy neighbor” aspects).

In Kong Enterprise, multi-tenancy is supported with workspaces. Workspaces provide isolation of gateway configuration objects while maintaining a unified routing table on the data plane to support client traffic.

When pairing workspaces with RBAC, Kong Gateway administrators can effectively create tenants within the control plane. The gateway administrator creates workspaces and assigns administrators to them. The workspace administrators have segregated and secure access to only their portion of the gateway configuration.

Performance Isolation in multi-tenant DB at Cloudflare

Kong addresses multi-tenancy as an access control problem and solves it with RBAC. But multi-tenancy requires not just access isolation; it also requires performance isolation. You don’t want one misbehaving tenant to create noise for everyone, especially since some workloads are more critical or more sensitive to latency.

I invited Vignesh, engineering manager at Cloudflare and author of the blog post Performance isolation in a multi-tenant database environment to discuss the topic on the SaaS Developer Youtube:

User Defined Fields

Jeffery Sherman blogs quite a bit and even has his own podcast. A while back, he wrote a series of blog posts on how to handle user-defined fields in CRM. CRM isn’t the only SaaS product that has this problem; it is actually quite common: JIRA also supports use-defined fields, and most expense reporting apps allow for custom fields. I think every SaaS Developer should be familiar with the three patterns for addressing this problem and their tradeoffs.

Pattern 1, generalized columns in a database, spanned the dawn of time until the rise of NoSQL around 2010.

Pattern 2, NoSQL, began around 2010 and continues to today.

Pattern 3, JSON in a relational database, began in the late 2010s and combines the best of the two approaches

I’d add that pattern three also has some of the drawbacks of both approaches, and the right answer is always “It depends.”

Software Complexity

Complexity isn’t specific to SaaS and is a common theme of conversation between developers. Is this solution simpler? From what perspective? Are we bringing in needless complexity or solving an inherently complex problem?

Felix GV has a slightly pessimistic take - that after a certain point, you can’t reduce complexity, just move it around.

His writing reminded me of the classic essay - Worse is Better, which discussed the complexity of implementation vs. the complexity of APIs.

Trends in Observability and Alerting

Anjul Sahu shared Grafana’s Observability Survey in our community Slack. Mark Roddy couldn’t believe that 52% of the companies use six or more observability tools. Shahar Glazner, Jeffery Sherman, Rauan Mayemir, and Ken Finnigan jumped in to discuss what they observed in the observability space and how one defines an observability tool.

Shahar Glazner shares the opinion that companies have many observability tools and that this is one of the top problems in the alerting space. He joined our channel to discuss:

Real-Time Analytics in SaaS

I (Gwen) gave a keynote at Real Time Analytics Summit, discussing the use of Real-Time Analytics in Modern SaaS. I went over the history of SaaS, gave examples of cool SaaS products that are built on top of real-time data, and some of the architectures that power these products.

Lots of videos this week :) I hope you enjoy watching them over the weekend!