A solution-driven mindset

You might think learning Spark, Kafka or AWS is more important, but you’d be wrong. 

Data Engineers who get the best jobs see solutions where others are stuck on what can’t be done. 

Here is one of the hardest, most important lessons I learned. 

Let’s look at what it takes to change your mindset.

Understanding the problem, really understand the problem. 

The etymological roots of the word understand can be interpreted as to stand under or in the midst of.  I take it to mean that if you want to grasp a concept or problem, you have to get your hands dirty – engross yourself in it

Too often we data engineers want to start coding right away. When I engage with a new client, I’ve taught myself that meeting with the stakeholders and sponsors is a first point of departure. Think Design Sessions and recorded interviews. It’s hard to resist the temptation of thinking you know what people need. As humans, we don’t enjoy admitting ignorance. Ignorance is also the basis for the Scientific Method. 

After interviewing those who will use my product, I replay the recordings and write down observations on a Miro board. If a user says we don’t trust the reports, for example, I’ll make a sticky note: 

The accuracy of the data is in question

I use these questions to validate my assumptions and create action times.

You can’t solve a problem without understanding it first

A viable solution is one within reason

“Oft expectation fails, and most oft there where most it promises; and oft it hits where hope is coldest, and despair most fits.” ― William Shakespeare

Under-promising and over-delivering are another of my operating secrets. Swapping those around is catastrophic. 

A recurring theme is that most users don’t understand the limitations of technology-driven solutions. Take real-time data as an example. Having real-time data might sound like a great idea but

• Is it worth the price of additional complexity and expenses?
• What questions will we be able to answer with real-time data that we can’t with data that’s 2 hours old?

If your users are happy with your list of constraints, you’re in a good place. If you don’t have a list, make one. 

Rellentlsly divide-and-conquer. MVDP as much as possible

If you start with the problem your customer has, what is the smallest, working, version of your solution? Data projects fail because teams try to boil the ocean. I’ve been guilty of scope creep many times. Here’s why a Minimum Viable Data Product(MVPD) is a good idea: 

• It forces you to be agile. If your product is not what they want, you can easily change it
• It gives you a foundation for a production-grade solution 
• It helps you to focus on the most atomic part of the solution
• It adds scalability to your solution
• It helps with thinking in terms of repeatable patterns
• Much more…. 

Granted, the engineering part of Data Engineering is easier said than done, but it’s worth learning to think in this way. 

To build an MVDP:

If your users want a report of daily sales figures, start with the report: 

• What platform will the report be on?
  • What is the most aggregate level at which we can display the data? 
  • Which of the graphs are most important and do they need first? 
• What data do I need to generate the report?
  • What are the most important tables needed for the report? Rank them and get the important ones delivered first
• Where is the data located?
  • Some source systems are typically more important than others. How do you get access to the important ones? Remember we are starting with the report and need to answer in light of it. 
• How do I join the data to deliver a usable report?
  • In an MVPD world, we’d focus on the absolute required transformations to make meaning out of the data from the source systems. The counter-example is trying to combine everything into one big table and include a small subset of the columns and data in the end report. There is a place for data exploration, data lakes etc., but don’t let the technology overshadow the importance of satisfying the user requirements. 
• What are the security constraints and how do we isolate data sources to only the relevant detail for the report?
  • What sign-off do we need? The smaller the dataset required the smaller the attack surface for malicious actors. MVPs tie in nicely with this consideration. 

Validate potential solutions. Get feedback. 

I have a barebones report. My report delivers only the vital graphs. I transformed only the necessary data, from the minimum number of source systems. Now it’s time to speak with my users. I’ve heard it said many times that, once you present someone with inaccurate data, it’s hard to win their trust again. When presenting someone with a small piece of a solution, I remind myself of this fact. 

The number of wrong assumptions I made that get shattered by user feedback is always surprising. 

I know it’s okay though. Building a trusting relationship with my stakeholders is more important than being a know-it-all.

Once I receive feedback, I can change where required or leverage my work to add more insights to the report. Both paths are part of the process. I’ve learned not to be precious about my code. If I need to redo everything to solve my client’s problem, that’s perfectly fine.  An MVP is easy to scratch and re-develop.

Optimism in a sea of fatalism

Optimism is another requirement for a solution-driven mindset. Matt Ridley speaks about Rational Optimists and I concur.

Studies show optimism can be developed and there are many benefits to it. (Learned Optimism by Martin Seligman is one of the books I still need to read.)

As a problem solver, the danger of pessimism is developing a sense of systemic helplessness – fatalism. Thinking in terms of a way forward, a belief in the future – optimistically – is thus by definition what it means to think of a solution. 

When I land on a project, things are normally not working as expected. People feel frustrated, angry and full of blame.

I force myself to listen, paying attention to the fundamental problems and thinking of a way forward.

Listening is the first step, codifying what I hear next, and coming up with solutions is the last. Throughout, if I don’t stay optimistic, the process will not work. 

Beyond client relations. Be solution-focused everywhere

If you go to your boss with a problem, you’re just putting more on their plate. Instead, I am learning that presenting a potential way out helps everyone. I am not perfect at it yet, but I know it’s worth learning the habit and have seen it pay off. Use this approach everywhere and be amazed at the results. 

Summary

By definition, we data engineers can’t add value if we don’t focus on the solution. The sentence seems almost simplistic. The challenge lies in navigating the waters to a feasible way out. Listening attentively, following a systematic approach and codifying everything has served me best. Optimism breaks barriers.