Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
You Can Build A Data Pipeline In <90 Min.
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourceHi past, present and future data professional! Since today is a U.S. holiday, I won’t take much of your time; the good news is that, when conducted efficiently, building a data pipeline doesn’t have to take days, weeks or months. In fact, you can build a data pipeline in as little as 90 minutes. Accelerating pipeline development depends on a thorough read of the documentation, a familiarity with your scripting language’s requests library and patience dealing with pesky data structures. If you think, during this time, engineers are heads-down, you may have watched The Social Network too many times; personally, I like a little external stimuli while coding, which is how I ended up building a full dashboard during another American pastime–a baseball game. My secret? Distilling data with clean views, which I recommend over bloated source tables for both aesthetic and performance reasons. Even optimizations like views have their limitations, leading to optimization ceilings. The best way to break through, aside from stubbornness, is a combination of incremental problem-solving and “big picture” data modeling to reassess resources and attack the problem completely. Since I don’t want you to have to work any harder today, here are the embedded links as text:
If you’re celebrating America today, happy 4th! Thanks for ingesting, -Zach |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! This edition almost became an apology because I’ve been on a tight deadline and pre-baby morning wake up thinking/writing time has become GSD (get sh!t done) hour. Long story short: I got brought in late to a time-sensitive project that required me to speed through a planned pipeline migration. As a recovering news junkie (aka journalist), I used to live and die by deadlines. But, given the unpredictability of data-oriented work and internal deliverables, it’s...
Hi fellow data professional! For years, the opening of The Simpsons, specifically Bart writing lines on the chalkboard, has been incredibly relatable to me. Not because I’m up to mischief (none I’ll admit to here, anyway), but because I spend most days writing the same three lines of SQL over and over again. If you've ever been paranoid about a table's content, you might know what I'm talking about. It’s the aggregate COUNT(*) grouped by a date field, ordered by date DESC. The output of that...
Hi fellow data professional! In a previous newsletter, I mentioned an idea that I wanted to explore deeper. At the risk of double-quoting a la The Office’s Michael Scott quoting Wayne Gretzky (“You Miss 100% Of The Shots You Don’t Take - Waynze Gretzky - Michael Scott”), here is the idea. “To be marketable as a candidate, you don’t just want to show how you can go from A to B (requirements->pipeline). You need to go from A to C (requirements->pipeline->scale/support).” You might be asking...