Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
You Can Build A Data Pipeline In <90 Min.
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourceHi past, present and future data professional! Since today is a U.S. holiday, I won’t take much of your time; the good news is that, when conducted efficiently, building a data pipeline doesn’t have to take days, weeks or months. In fact, you can build a data pipeline in as little as 90 minutes. Accelerating pipeline development depends on a thorough read of the documentation, a familiarity with your scripting language’s requests library and patience dealing with pesky data structures. If you think, during this time, engineers are heads-down, you may have watched The Social Network too many times; personally, I like a little external stimuli while coding, which is how I ended up building a full dashboard during another American pastime–a baseball game. My secret? Distilling data with clean views, which I recommend over bloated source tables for both aesthetic and performance reasons. Even optimizations like views have their limitations, leading to optimization ceilings. The best way to break through, aside from stubbornness, is a combination of incremental problem-solving and “big picture” data modeling to reassess resources and attack the problem completely. Since I don’t want you to have to work any harder today, here are the embedded links as text:
If you’re celebrating America today, happy 4th! Thanks for ingesting, -Zach |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! Once thought to be a purely back office role, data engineering is undergoing a radical transformation and gaining a new responsibility: Front-end deployment. The folks already deploying applications in this capacity are known, incidentally, as forward deployed software engineers or forward deployed engineers (FDEs). Before you worry about needing to learn JavaScript or other web programming paradigms, know that I’m referring to the preparation, deployment and...
Hi past, present or future data professional! As time in 2025 dwindles, I wanted to share what I learned about optimizing design, development and troubleshooting time while working 3 days per week this fall. Quick background: If you’ve been a long-time reader, you’ll know that in March my wife and I had our first child. Consequently, through my employer, I was eligible for several months of parental leave. Anticipating my wife’s return to work (after much needed time off!) I allocated the...
Hi past, present or future data professional! As the winter holidays approach, we’re entering a period of downtime for most orgs. Assuming your employer has hit goals (or accepted losses), allocated coverage for the slew of inevitable vacation requests and maybe even entered a “code freeze”, you’re entering data & tech’s slow season. If you’re working, during this time you may be asked to do any number of “downtime” (actual free time, not data outages) tasks ranging from code refactors to...