Hacking SaaS #4 - Mostly about Data

A new data store, semantic data layer, self-serve streams, control-plane events, state of APIs and how to optimize your SOC2 audit.

Discussions on SaaS Developer Slack

Shomik Ghosh shared “a great article by the Fly.io team on the SOC2 process, why it’s important, how it’s a PITA and how they got it done as efficiently as possible.” Shomik’s own blog, Software Snack Bites, covers wide range of topics that can interest engineers at startups. Well worth checking out.

Felix GV let us know that his LinkedIn team will open-source Venice at the Strange Loop conference. Venice is a highly anticipated data store optimized for high throughput ingest and low latency serving, for years now.

Yuval L asked about “feature flags as a service” and shared his requirements:

• Single pane of glass in the form of a Web UI - MUST

• Preferably a Web API integration

• Creating tenants / segments and assigning them with a specific feature flags configuration - MUST

• Supports Role based operations - optional

• Complies with the highest security standards - MUST

Mathew Dodge suggested LaunchDarkly and Lalit Pagaria suggested two OSS projects - Unleash and Flagsmith.

Other SaaS Development News:

The data community is buzzing with discussion about “semantics”. The Modern Data Stack is notoriously fragmented, and the shared meaning of business entities, relationships, metrics and events can become lost or conflicted. “Semantics Layer” is proposed as a solution. DBT are gearing to announce their semantics layer, Chad Sanderson revealed the Semantics Data Warehouse, and discussed his ideas with Marc Grover at the Stemma blog.

We learned how Riskified turned their Spark-based stream processed architecture into a self-serve system, and how they later added automatic capacity management with no impact on their users.

One would expect that control-plane events from EKS (AWS’s managed K8s) would automatically find their way into Cloudwatch (AWS’s log and metrics product). They do not, but this blog by AWS explains how to get events from EKS control plane to Cloudwatch. EKS managed service is really just the control plane, and you need to set up the system out of 30+ other AWS resources. AWS has templates for a recommended K8s deployment.

Postman released their State of APIs 2022 report. Few highlights:

While only 8% of respondents identified as API-first leaders, this small, elite group excelled on almost every metric.

The number-one factor in deciding whether to consume and produce an API is how well it integrates with internal apps and systems.

One of the top obstacles to producing APIs is new this year: a lack of API design skills. This skills gap may be contributing to an overproliferation of microservices, which is creating its own problems.

SaaS Developers on Youtube:

We talked to Oz Katz. CTO of Treeverse and LakeFS about “Git for Data” and its many use-cases. Oz shared his view of the data engineering world, wise advice and great war stories.

Thats it for this week!