The Activity Schema is Worth a Look

Right before we took the newsletter and blog offline for our repositioning, we had a call with Cedric Dussud from Narrator.ai. Narrator is a business intelligence tool that specialises in something called an Activity Schema.

I'm here to tell you that the Activity Schema is worth a look.

What is it?

The Activity Schema is a data modeling approach, like Kimball's dimensional data modeling (which was created back in the 90s), or Data Vault (early 2000s). As far as I can tell, there aren't that many data modeling approaches around, and Narrator's is probably the only new one that is worth paying attention to.

What do I mean?

Let's take a step back. Kimball's dimensional data modeling was developed to give business intelligence teams three things:

• Flexibiility
• Extensibility
• Performance

We talk a lot about dimensional data modeling in terms of flexibility and extensibility, but a huge chunk of the Data Warehouse Toolkit was dedicated to performance considerations, especially that of a RDBMS.

And for good reason: the predominant database architecture of the time was the RDBMS. The star schema was designed to take advantage of the star join, which was a relatively performant query to do in an RDBMS. (A star join is specifically the Cartesian product of all the dimensional tables, which is a good fit with relational database paradigm — as you might imagine.)

Today, however, we mostly use columnar data warehouses to do business intelligence.

Star joins aren't performant in a columnar data warehouse — and they don't actually match the nature of the underlying technology.

So why are we so insistent on using Kimball dimensional data modeling?

Well, the reason is simple: dimensional data modeling gives you two other things — flexibility and extensibility. But if it's no longer performant (or even recommended) to do star joins ... should we even be sticking to it?

I think we shouldn't. And I know I'm not alone in this — increasingly, people have been dumping tables into columnar data warehouses without strict data modeling.

Well, the Activity Schema is a data modeling approach that is flexible, extensible, and performant — because it's been designed with columnar data warehouses in mind.

The core idea is simple: you have one main table, called the activity stream table. That table has just 14 columns.

Every action that a user does is modeled as an event, called an 'activity'. That activity can pretty much fit into the 14 columns. This is conceptually very different from dimensional data modeling. Think about it — everything is an event, placed in time:

• Customer upgrading a plan? That's an event. Insert it into the activity stream table.
• Prospect hits your website? That's an event. Insert that into the activity stream.
• Customer churns? Or files a ticket? Or makes a purchase of a Godzilla t-shirt from your store? Or you update the inventory of Godzilla t-shirts because your supplier finally got back to you?

All events. You get the idea.

The magic is in the way you query this table. They call this 'borrowing'. Think: you're performing joins, but across different activity types within the same table. And you're doing in through time.

Say we wanted to group all 'submitted_support_ticket' activities by the last product purchased. Because all activities can be joined together over time, it's a fairly straightforward query join in the last completed_order before each submitted support ticket and select its 'product' feature.

This means that you're taking advantage of the nature of columnar datastores: quering down a single table is wonderfully fast.

Now, there are a few other tables — like a table to store extra metadata, and a table to store enrichment data. But by and large, most things are done in the activity stream table.

And here's the kicker: according to Dussud, any model you can think of creating in the dimensional data modeling way, you can also do in the activity schema method. They say that they've worked with clients for a few years now as a consultancy, before starting the product company. Dussud says that watching their clients realize that this is possible is still one of their favorite things to do.

Ok, I'm not sure if this explanation makes sense. But the idea has been in the air for some time; see, for instance Eric Bernhardsson's observation:

Spotify made a (imo) brilliant architectural decision early and made it super easy for any service to emit machine readable events with a proper schema. This was the basis of the whole analytics stack. I think this is rare though? Do we need better tooling for it?

— @bernhardsson

https://twitter.com/bernhardsson/status/1435648096155881482

If you want to read more, click here to read the full spec.

---

Insights From Elsewhere

How to Live with Chronic Imposter Syndrome — I really like Eugene Yan's work, and I think this is one of those things that is more widespread in data than in other domains.

Many of us are not from a technical background. And somehow we find ourselves in a data analyst/data professional role.

Yan's piece is a good reminder that some of the best people in the world also suffer from imposter syndrome. And that's ok.

The Data Mesh Article — This is apparently the canonical article that introduced the concept of a data mesh. I ... don't yet know what to think about it. So I'm going to tell you about a theory I have.

Leans in conspiratorially Here's my theory. I think that organizations that are data driven will eventually evolve their way into an effective data stack. Data is a core capability. It's important to them. The CEO hugs his dashboards to sleep. So they invest in it. They hire good people. They evolve their tools.

But if an organization isn't data driven, then, well — nothing happens. Good people leave. The infrastructure rots. And no amount of complex data mesh domain driven architecture book writing self service Thoughtworks consulting buzzword soup will make a whit of difference.

To a software consulting company, everything looks like a software problem. But perhaps the problem is ... being data driven?

Go collect some $#*(&% data — Randy Au, making the case that collecting data is a skill, and if you want to grow as a data scientist, you should probably collect data.

(This is, after all, the same writer who brought us 'data cleaning is analysis!')

Did I mention I like Randy's writing? I do.

---

That's it for this week! If you enjoyed this newsletter, I'd be very appreciative if you forwarded it to a friend. And if you have any feedback for me, hit the reply button and shoot me an email — I'm always happy to hear from readers.

As always, I wish you a good week ahead,

Warmly,

Huy,

Co-founder, Holistics.

PS: If you've not seen it already, we've got a guidebook to bring you up to speed on the ins-and-outs of a contemporary analytics stack — get your copy here.