Hacking SaaS #37 - Debugging, Authentication, Design and AI
Grab-bag of interesting software engineering updates for your reading (and viewing) pleasure
Some weeks hacking SaaS has a theme - Database updates, Re-invent news, AI catchup. This isn’t one of these weeks. This is a week where we have something interesting on a wide variety of topics. I hope you’ll find one or two bits that you’ll find interesting and perhaps even useful.
Debugging
Everyone loves a good debugging story. Especially one that involves a well-known highly-reliable bit of infrastructure, Postgres for instance, that suddenly crashes with segmentation faults. If you love diving deep into how the software we all rely on actually works, this is a must-read.
The article, from DataDog engineering, has so many interesting details - both of the investigation and of the software involved. It starts with a seemingly simple query that causes a mysterious crash, dives into Postgres JIT (little understood, but fascinating bit of optimization) and from there to LLVM internals and ARM64 ABI.
Debugging is clearly a powerful skill for software engineers. In the age where LLMs write code, it may be the most valuable skill for a human engineer. Yet I always found it challenging to both improve and to teach. I mentored engineers who were amazing at debugging and others who seriously struggled, and I have to admit that I never managed to help someone become significantly better at it. I worked hard to improve my own debugging skills, but what improvements I made were either due to discovery of better tools (Wireshark, strace, ltrace, etc) or a mindset shift that came from watching someone else debug.
But there is hope! David A. Wheeler reviewed David J. Agan's “Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems”
The review gives a nice taste of the book, which sounds exactly what I wish I read about 25 years ago, and what I’m going to give any junior engineer I’ll ever have the pleasure of mentoring in the future.
State of Auth
Web authentication is one of these things that look simple, until you start looking into it. On the surface, you enter your username and password, the app looks it up in some database and either lets you in or gives you an error. How complicated can it possibly be?
But then you start looking and find so many details. It involves cookies (which must be secure and served from the right domain), sessions, JWT (which is considered evil but also mandatory everywhere), tokens, oauth and its friends oidc and pkce, SAML, TLS of various versions and if you get unlucky, possibly Kerberos. This is obviously a partial list.
To make things more fun, a lot of the web is built with Javascript, so naturally there are many Javascript auth frameworks. And because Javascript ecosystem is even more trend-driven than others, there’s a new and arguably improved auth framework out every 6 month.
In the video below, Theo does a great job demystifying web authentication and then summarizing the state of things in 2025.
Once you understand the basics, you may want to dig a little deeper. For example, this writeup on a vulnerability in Google’s implementation of OAuth. The complexity of the protocol means that even the biggest and most reliable implementations have some holes. And of course, discussions of vulnerabilities in protocols are almost as fun as debugging stories.
Software Design
We all love logs, and we all know that logs are at the heart of basically every system out there. But it is still worth revisiting the implications of building applications around a log of events. Especially how it reduces coordination costs and locking.
This blog, by the Restate team (formerly Flink founders), is a clear and practical writeup on the use of event log for coordinating distributed applications. The hacker news discussion is good too.
And if you want to read more on how logs can be used when building applications, the classic Turning the Database Inside Out, is always worth a look - whether its a first or tenth look.
About AI
One of my goals with this newsletter in 2025 is to help us all stay on top of AI news without getting distracted by rampant hype.
DeepSeek has been getting a lot of hype. It is an open source model out of a Chinese hedge fund that beats all other open source models and is competitive with OpenAI models from few month back. The hype is driven by few things:
It uses novel training methods. Especially reinforcement learning. Reinforcement learning isn’t new, of course (it goes back all the way to Pavlov’s experiments!), but using it to improve LLM performance as replacement for fine-tuning.
It is trained with different but effective methods of reasoning - multi-expert and chain-of-thought.
It was (controversially) very cheap to train. Orders of magnitude cheaper. So cheap that it sent NVidia stock crashing.
To get ground truth, I suggest reviewing the paper which explains all the interesting new methods they used, as well as this balanced review - which points out both what DeepSeek does well and also where it is likely misleading folks and causing excessive hype.
Everyone is still trying to make agents happen. Google has a good writeup on a new “chain of agents” model that may outperform both agent frameworks and RAG. And the always brilliant Jack Vanlightly collaborated with Sean Falconier to summarize the current state of agents.
In other news, there is evidence that use of LLM can cause “brain rot” or degraded performance in software engineers. I’ve noticed that anecdotally, but there is research that points in that direction too. This short blog discusses the problem and suggests some techniques for using LLMs while still keeping your brain intact.
And if you are confused by how LLMs can be so smart and so stupid at the same time, this excellent article dives into “how LLMs think” and explains the current confusing state of things:
And… thats it! Happy February everyone.