Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #16] Better Data Solutions In 5 Steps
Extract. Transform. Read.A newsletter from PipelineHi past, present or future data professional! When you apply to data analysis, data engineering or data science jobs, you likely consider factors like company name, culture and compensation. Caught up in the excitement of a fresh opportunity or compelling offer you’re neglecting an important part of your day-to-day reality in a new role: What stage of data maturity the organization is in. If you’re looking for experience building something new from the ground up, you likely won’t find it in a company that has a years-old established cloud infrastructure. If you’re inexperienced, you might also feel lost in a company that is still conceptualizing how it is going to establish and scale its data infra. While I personally arrived at a team and organization in its mid-life stage, I’ve had opportunities to discuss, examine and advise those who are considering how they can make an impact at an earlier-stage company in both full-time and contract roles. This compelled me, after a transatlantic flight, to compile a framework you can use to conceptualize anything from an in-house data solution to full-fledged infrastructure. Phase 1 Discovery - Extensive, purposeful requirements gathering to make sure you are providing a solution and, more importantly, a service, to an end user. Phase 2 Design - You can’t begin a journey or a complex technical build without a road map; take time to make a wish list of must-have data sources and sketch your architecture before writing line 1 of code. Phase 3 Ingestion - Build your pipelines according to best practices with a keen eye on cost and consumption; expect this to take 6-12 months depending on your work situation. Phase 4 Downstream Build - Going hand-in-hand with requirements gathering, consider how your target audience will use what you’ve built; might it be better to simplify or aggregate data sources in something like a view? Phase 5 Quality Assurance And Ongoing Tasks - Even though your pipelines and dashboards will be automated initially, nothing in data engineering is 100% automated. Components will break. You’ll be expected to fix them. And assure it doesn’t happen again. These 5 phases aren’t meant to be strict rules for building data infra. But they should get you thinking about how to build something purposefully so you can spend your time dealing with angry code–not stakeholders. Dive into the framework here. Here are this week’s links:
Until next time–thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! I learned one of the most important personal branding lessons in the basement of Arizona State University. I was seated at my desk in the Post Office/Writing Center as my coworker, a fellow writing tutor, reviewed my resume. “The content is good, but I won’t remember this. There’s no branding.” She thought for a second. “You know what? Change the font color to navy. Your brand is now blue.” I laughed but she was serious and the interaction imprinted on me not the...
Hi fellow data professional! This edition almost became an apology because I’ve been on a tight deadline and pre-baby morning wake up thinking/writing time has become GSD (get sh!t done) hour. Long story short: I got brought in late to a time-sensitive project that required me to speed through a planned pipeline migration. As a recovering news junkie (aka journalist), I used to live and die by deadlines. But, given the unpredictability of data-oriented work and internal deliverables, it’s...
Hi fellow data professional! For years, the opening of The Simpsons, specifically Bart writing lines on the chalkboard, has been incredibly relatable to me. Not because I’m up to mischief (none I’ll admit to here, anyway), but because I spend most days writing the same three lines of SQL over and over again. If you've ever been paranoid about a table's content, you might know what I'm talking about. It’s the aggregate COUNT(*) grouped by a date field, ordered by date DESC. The output of that...