Hi fellow data professional!

In a previous newsletter, I mentioned an idea that I wanted to explore deeper.

At the risk of double-quoting a la The Office’s Michael Scott quoting Wayne Gretzky (“You Miss 100% Of The Shots You Don’t Take - Waynze Gretzky - Michael Scott”), here is the idea.

“To be marketable as a candidate, you don’t just want to show how you can go from A to B (requirements->pipeline). You need to go from A to C (requirements->pipeline->scale/support).”

You might be asking “Isn’t requirements->data pipeline what data engineering is?” And, yeah. Well, it depends.

It’s true that data engineering is becoming less about low-hanging fruit like ETL development because vendors are increasingly pushing codeless solutions and orgs are supercharging existing senior engineers with LLMs to do most of this work.

But what really determines whether the role you’re going for is doing traditional pipeline development is the stage of maturity of the team you’re joining.

Many teams that would take on and commit to training a junior engineer are likely in a more mature stage of data infrastructure and are looking to scale and extract value from existing systems.

I joined a data team in its infancy and have grown with it as the data org has reached maturity. That means my job is more A to C. I build some pipelines but really I work with lines of business directly to leverage a lot of our existing data.

And when I do build pipelines, it’s not like the old (again, 4+ years ago) days.

Instead of writing Python *gasp* by hand and spending an afternoon sifting through Stack Overflow, I feed starter code and API docs to an LLM. Then I test, QA and submit my work.

Steps that used to consume 1-2 sprints (about a month) can be executed within a work week.

Anyone who is purely building pipelines without understanding how to help refine and support them through their life cycle is not going to be a marketable candidate.

If you’re not currently working on a team as a data engineer, when you build a pipeline for practice, for school or for a freelance assignment, I encourage you to think critically about the output of your build and how it actually fits your desired use case.

This means

• Anticipating potential bugs and incorporating error handling
• Defining an appropriate downtime time frame - how long would it take you to get this pipeline up and running if there was an outage?
• Creating a fixed schema that allows your tables to scale from millions to billions of rows
• Developing a data dictionary that explains all the attributes in a given table

These are the “non-code” to-do items I spend the majority of my time addressing.

At a junior level it’s unlikely you’ll be given systems design questions in an interview, but that doesn’t mean you should squander an opportunity to present your larger vision for the data you’ve mined and refined.

Demonstrating an ability to go from A to C prevents you from being candidate choice B.

​Discover how I recently went from A to C at work by diagnosing and future-proofing an error related to the most basic SQL command.

Thanks for ingesting,

-Zach

​Medium | LinkedIn | Ebooks