[ETR #16] Better Data Solutions In 5 Steps


Extract. Transform. Read.

A newsletter from Pipeline

Hi past, present or future data professional!

When you apply to data analysis, data engineering or data science jobs, you likely consider factors like company name, culture and compensation. Caught up in the excitement of a fresh opportunity or compelling offer you’re neglecting an important part of your day-to-day reality in a new role: What stage of data maturity the organization is in. If you’re looking for experience building something new from the ground up, you likely won’t find it in a company that has a years-old established cloud infrastructure. If you’re inexperienced, you might also feel lost in a company that is still conceptualizing how it is going to establish and scale its data infra.

While I personally arrived at a team and organization in its mid-life stage, I’ve had opportunities to discuss, examine and advise those who are considering how they can make an impact at an earlier-stage company in both full-time and contract roles. This compelled me, after a transatlantic flight, to compile a framework you can use to conceptualize anything from an in-house data solution to full-fledged infrastructure.

Phase 1

Discovery - Extensive, purposeful requirements gathering to make sure you are providing a solution and, more importantly, a service, to an end user.

Phase 2

Design - You can’t begin a journey or a complex technical build without a road map; take time to make a wish list of must-have data sources and sketch your architecture before writing line 1 of code.

Phase 3

Ingestion - Build your pipelines according to best practices with a keen eye on cost and consumption; expect this to take 6-12 months depending on your work situation.

Phase 4

Downstream Build - Going hand-in-hand with requirements gathering, consider how your target audience will use what you’ve built; might it be better to simplify or aggregate data sources in something like a view?

Phase 5

Quality Assurance And Ongoing Tasks - Even though your pipelines and dashboards will be automated initially, nothing in data engineering is 100% automated. Components will break. You’ll be expected to fix them. And assure it doesn’t happen again.

These 5 phases aren’t meant to be strict rules for building data infra. But they should get you thinking about how to build something purposefully so you can spend your time dealing with angry code–not stakeholders.

Dive into the framework here.

Here are this week’s links:

Until next time–thanks for ingesting,

-Zach Quinn

Pipeline To DE

Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.

Read more from Pipeline To DE

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s the most wonderful time of the year… performance reviews. Depending on your outlook on your job/org, that “wonderful” could be sarcastic. However, if you’ve landed on your manager’s nice list, this can be the time to recap your achievements to maintain credibility and work toward the next rung on the corporate ladder. If you’re on a small team serving demanding stakeholders it’s possible...

Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! This holiday season will be a little less bright thanks to my lack of personal GitHub commits. Like you, I began 2024 full of ideas and motivation that, let’s be honest, was depleted by the end of Q1 when I was cranking out enough code at work that would please even notorious code volume stickler Elon Musk. Despite my lacking output, I managed to hunker down to create useful “bespoke” (a.k.a....

Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! When I worked at Disney there was one line (aside from “Have a Magical Day”) that was borderline beaten into us: “We are all custodial employees.” The line meant, of course, to keep areas under your purview neat and presentable (“show ready” in Disney-speak). Using the same logic, I’d like to emphasize that while the various data roles (data analyst, data scientist, data engineer, etc.) have...