Applying The Principles of Agile to Data Modeling

This is a third post in a series on Agile Data Warehouse Design, a 2011 book about data modeling. Read the second post here.

In February 2001, a group of 17 software engineers gathered in a ski resort in the Wasatch mountains, and created The Agile Manifesto. The manifesto is reproduced in its entirety in an appendix in Agile Data Warehouse Design, and it reads as follows:

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The Twelve Principles of Agile Software

We follow these principles:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity—the art of maximizing the amount of work not done—is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

As of 2020, the Agile Manifesto is 19 years old. Its influence in software is palpable; it’s impact on software development already established by the end of its first decade.

These were the ideas that Corr and Stagnitto took and applied to the practice of data modeling.

Where Did Agile Come From?

The software people who attended that 2001 meeting weren’t there by accident. For months before the conference, they were throwing about ideas on mailing lists and in wikis, debating the differences in their approach to software development. You could say they were a bunch of ‘gurus’ — people who were proselytizing a lightweight approach to building software, though under a set of different banners with different names.

Who were these 17 people? They were IT consultants and senior software engineers; engineering leaders and internal software champions. They built internal software for corporations.

It is tempting to describe agile as the answer to all software development process problems. It is equally easy to become dogmatic about the ideas. The original agile principles were meant to be a set of common statements that everyone in the group of 17 professionals could agree on. The fact that these individuals were mostly corporate software builders wasn’t lost on the rest of the world.

A year after the Agile Manifesto was published, software entrepreneur Joel Spolsky wrote a perceptive blog post titled Five Worlds — in which he argued that not all forms of software development worked well with agile:

Here’s an important thing to know. Whenever you read one of those books about programming methodologies written by a full time software development guru/consultant, you can rest assured that they are talking about internal, corporate software development. Not shrinkwrapped software, not embedded software, and certainly not games. Why? Because corporations are the people who hire these gurus. They’re paying the bill. (Trust me, id software is not about to hire Ed Yourdon to talk about structured analysis.)

The blog post is worth reading in its entirety. It serves as a reminder that ideas need to be evaluated with their full context in mind.

Agile as Applied to Data Modeling

Is it reasonable to apply agile principles to data modeling? As you may have guessed from my previous post, I believe that it is. The business intelligence use case is similar to internal software development, and Corr and Stagnitto have had some success teaching their ideas to a generation of dimensional data modelers.

More interestingly, if we take the original principles of agile and adapt them to data modeling, we get:

The Twelve Principles of Agile Data Modeling

• Our highest priority is to satisfy the business person through early and continuous delivery of valuable, modeled data.
• Welcome changing requirements, even late in the data warehousing project. Agile processes harness change for the business’s competitive advantage.
• Deliver modeled data frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business stakeholders and data professionals must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a data team is face-to-face conversation.
• A completed data model — with ETL and attached business intelligence tool — is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, analysts, and business users should be able to maintain a constant pace indefinitely.
• Continuous attention to data engineering excellence and good design enhances agility.
• Simplicity—the art of maximizing the amount of work not done—is essential.
• The best models, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

At the beginning of this series I said that you may view Agile Data Warehouse Design as two books pretending to be one: on the one hand, it is a collection of design techniques and modeling processes intended to be used at the design stage — adapted for the databases and OLAP cubes of the day; on the other, the book is a serious attempt to apply the principles of agile to the activity of data modeling.

It is this second book that is timeless. Regardless of whether you use Kimball-style dimensional data modeling in your work today, and irregardless of the tools you use to get data to your business users, it would be valuable, I think, to reflect on your current practices in view of this list of twelve principles.

Corr and Stagnitto’s book is merely one implementation of these ideas. They did their work in 2011. But anyone can take these principles and adapt them to their practice of business intelligence. You merely have to realise: the questions you might ask today are no different from the questions they must have asked back then.