The skills chasm of the data analyst career

The Skill-mix

The best data analysts aren't people who are great at SQL and data modeling and all the other technical things ALONE. The best data analysts — the ones who excel — are usually the ones who are best able to understand and communicate with the business end of things.

They understand the organization. They're easy to talk to. They know how to give their business colleagues the data they need, without too much struggle. (Don't make the business person struggle to spell out the metrics they need!)

When you look at your career like this, you'll begin to see that the data analyst career path really consists of a few skill buckets:

1. Technical data analytics itself — meaning, the ability to write good SQL, do modeling, manage data quality, and all the things we think about when we talk about the technical side of the business.
2. Business problem solving — the ability to understand, troubleshoot and identify what it is that your business colleagues need, even especially when they don’t know what they want.
3. Communications/stakeholder management  — doing all the above in a manner that your business colleagues feel comfortable throughout the entire process.

In most BI organizations, the typical entry-level analysts start out working on ad hoc requests. Business asks for reports, they write queries, run numbers, do Excel formatting, and pass the reports to their business colleagues. It’s the true English-to-SQL translation job. Here’s where their technical analytics skills have the most use.

As they rise through and acquire more knowledge, they’ll try to get into the business table, consulting people on how to utilize data the right way.

Yet here’s where I see most analysts struggle.

“They’re not smart enough”, and “they don’t understand what I’m saying”, Those are common complaints I hear. If this happens long enough, the business users end up abandoning working with BI altogether, relying on instincts to make decisions.

The BI leaders I spoke with, as much as they want to expose their junior analysts to business departments early, understand this risk of upsetting business. So they ended up pulling the analysts back doing mundane technical work, or over-extending themselves to train their team.

Technical analytics skills are not enough

Just to clarify, I’m not saying that technical skills aren’t important. They are. But they’re just not enough.

I’ve seen plenty of cases where a data analyst (well, technically a data engineer) with advanced SQL knowledge couldn’t answer a simple, naiive data question from a business user. He looked confused, and the business colleague looked equally puzzled. As if they’re both speaking 2 different languages!

I’ve also seen the same case handled by a seasoned data analyst. Upon hearing the question, the analyst simply asked a few clarifying questions (Socratic style), the business person quickly realized an epiphany with their business, thanked him, and didn’t really need the number she requested at all.

So, why are so few data analysts get good at business problem-solving? Let’s explore a few reasons.

The difficulty of learning (and teaching) business problem-solving skills

If you look at the skill-mix above, you’d soon realize that:

• Technical analytics skills (SQL, Python, statistics) are easy to teach, measure and test for.
• Business problem-solving is the exact opposite: They are difficult to learn and teach, hard to measure, and time-consuming to interview for.

You can pick up a book on SQL, Python, take an online course. With some hard work, it’s relatively easy to excel SQL writing all by yourself.

But you can’t learn (or teach) business problem-solving the same way. Why? It’s the business context! The heart of business problem-solving is to identify the right contextual information. Picking the right contexts into the picture is already more than half of the problem-solving process itself.

To learn this effectively, you’ll need to be exposed directly to a real-world business environment, and learn from doing real work under a supervision of a mentor.

(You see this in other study fields like product management as well. That’s why it’s very important as a student to get real-world exposure through internships and co-ops).

Because of this, the BI leader, when asked to design a data analyst interview assessment, scrapes together some SQL quizzes on a sample Northwind dataset. Feeling it’s not enough, he slaps in the last question asking the candidate to analyze the sample data provided based on some hypothetical assumptions.

Mind you, I don’t blame the BI leaders. I’m guilty of this myself, so I understand how difficult it is to truly prepare an interview to test for analytical business problem-solving and communications skills. I’m simply stating the facts for what it is.

This leads to the second reason…

The wrong emphasis of aspiring analysts

So business problem-solving is hard to teach, and technical skills are much easier. What do you think it leads to? An over-supply of technical analytics (SQL, Python) courses, and an undersupply of the former on the education front.

Naturally, what gets repeated get remembered: Entry-level analysts will tend to over-index technical skills and under-index business problem-solving.

Surprise! So now, not only the important data skills hard to learn and teach, the people entering the field also have the wrong impression about what’s important. They’ll just need to excel technical-wise to perform well.

On top of that, the illusion of data analyst being a junior role doesn't help.

This collectively creates the false expectation of the analyst job, letting people think the analyst career is an easy ride. (It's not, it's the exact opposite!)

Sooner or later, they’ll face this realization (and reality) a few months into the job, when they’ve already invested hard-earned money and effort in pursuing.

Ok... so how?

I’m not done yet... But this post/newsletter is getting long enough. So I’ll end here.

In subsequent posts, before we get into a solution, I'll also argue that: Not only it's hard to become (and therefore find) good data analyst, when the good data analyst rises in seniority, s/he will likely leave the data analyst job. The talent crunch happens on both sides (how exciting 🙃).

But let's leave it there.