Data Hierarchy of Needs

In psychology, there is a famous construct created by Abraham Maslow called the hierarchy of needs. Put simply, it says that people must first satisfy their basic needs before they can progress to focusing on more nuanced goals.

It’s often shown as a pyramid where each need builds on top of the previous one.

The goal, of course, is to reach the top. By satisfying everything below it, you too can reach your true potential. Self-actualisation means being the best you can be.

Data stack pyramid

I have talked to hundreds of companies investing in their data infrastructure. Over time, I began to think about it like Maslow’s hierarchy of needs. There is a clear pattern of foundational pieces put in place that lead up to more and more value being created.

With the explosion of interest in improved data stacks, companies have been working their way up through these stages. Some of the more forward thinking organizations are even reaching their own version of self-actualisation.

Warehouse Storage Needs

The foundation of Maslow’s hierarchy of needs is about physiological needs like having enough food, water, and sleep. In the world of the modern data stack, the parallel is having a data warehouse that can act as a foundation for the work to be done.

Over the last few years, tools like Snowflake and BigQuery have become the go-to solution in this space. They have hit an attractive performance/price point through their use of columnar storage and separation of data and compute. At Grouparoo, we are also seeing an uptick in requests for Clickhouse, an open source data warehouse.

Like the core human needs, the ability to easily read and write warehouse data is fundamental. The rest of the steps are all built on this base by having all the key data in one accessible place.

Extract and Load Needs

The next area of concern for our human needs concerns safety. We are not ready to start being our best selves if we don’t have a safe place to live. In our data stack, what we need is some actual data.

The first thing many companies set up is a process to replicate their transactional product database into their warehouse. This often accounts for the core business cases. In an e-commerce company this includes customers and purchases tables.

Other data is also loaded into the warehouse from more remote sources. For example, a data engineer might load in data about purchases and returns from Stripe, their payments vendor. This can be custom code or via ELT providers like Fivetran and Airbyte.

This stage loads the raw data into the warehouse. Once this is done reliably, our humans can start to focus on what it means to their business.

Transformation Needs

At this stage, humans are starting to attend to their mental needs. How can we be better understood? In our data stack, we have similar goals. How can we take this big bucket of data and make better sense of it?

The growing trend of analytics engineering with tools like dbt mirror the physiological parallel of focusing inward. Analytics engineers take in data from the world around them and rearrange it in a way that makes it usable. For example, they would combine all the data about each customer in order to determine in which cohort where they belong.

With our house in order, we can move on to continually analyzing changes in the data to understand if we are meeting our business goals.

Reporting Needs

In Maslow’s hierarchy of needs, this stage is about self-esteem and gaining a sense of accomplishment in our lives. The same is (ideally) true of the dashboards of data that we use to track our business progress.

Data analysts and operational teams use business intelligence tools like Looker or Tableau to report on the transformed data in the warehouse. By rolling up all the data and standardizing on what success looks like, an organization can focus on moving the numbers. The eCommerce company, for example, will want to know if the cohort of churned customers is growing or not.

We are creating more and more value from our data. At this stage, our organization can make better decisions and understand their impact through analysis. As we reach towards the top of the pyramid, we look to put the data itself into the action.

Operationalize

Self-actualization means reaching one's full potential. What does that look like in our data stack? In a growing number of organizations, it means automatically acting on the results of our data activities in order to improve the customer experience and the business.

When team members start to see the value in the reports being generated, they inevitably start to look for ways to feed that into the real operations. A report of customers that are predicted to churn soon is helpful, but what if we could automatically send them an email with a coupon when they reach that precipice?

Data teams are now becoming involved in this productization of their data stack through this process called Reverse ETL. With this approach, they can close the loop by putting their data into action in the Marketing, Sales, and Support tools used by the company.

This action lies at the top of the pyramid because it is the highest leverage activity. All of this investment in data storage, loading, transformation, and analysis culminates in automated impact.

Reverse ETL with Grouparoo

Grouparoo is an open source Reverse ETL tool that makes it easy to act on your data. It bridges the gap between your data warehouse and any tool where its contents can be put into action.

Many of us on our own personal journey to reach self-actualization. It can certainly be quite challenging. However, with Grouparoo, our data stack can reach that point of enlightenment. At least that’s something.