Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #31] Master An Underappreciated DE Skill
Extract. Transform. Read.A newsletter from Pipeline. *Today's edition was initially published on Medium on 12/10/24 Hi past, present or future data professional! I’ve recently been honing a data engineering skill that might not occur to you—drawing. When I first started my data engineering job 3+ years ago, any description or information related to my code would be in written form. This meant everything from README documentation to illegible legal pad scribbles would be all I had to inform decisions about design and implementation. Lately, however, my tasks have grown in both complexity and volume. What I need to convey to myself and my team won't fit on one sheet of paper. And if it did, it wouldn’t make a bit of sense. So I’ve turned to diagramming tools. I use tools like Microsoft Visio and Draw IO to create clear depictions of pipelines. This makes it easy to:
And, finally, the act of creating an architecture (arch) diagram subtly communicates something: That significant thought went into what you want to propose or present. Anyone can scribble in a shared doc or reference a notebook of ideas. Taking the time to spend time compiling a visualization demonstrates intent and care. This translates to you communicating your investment in making sure your build is possible and functional. Otherwise, it’s back to the (literal) drawing board. Feel free to scribble down this week’s links.
Questions? zach@pipelinetode.com Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! This edition almost became an apology because I’ve been on a tight deadline and pre-baby morning wake up thinking/writing time has become GSD (get sh!t done) hour. Long story short: I got brought in late to a time-sensitive project that required me to speed through a planned pipeline migration. As a recovering news junkie (aka journalist), I used to live and die by deadlines. But, given the unpredictability of data-oriented work and internal deliverables, it’s...
Hi fellow data professional! For years, the opening of The Simpsons, specifically Bart writing lines on the chalkboard, has been incredibly relatable to me. Not because I’m up to mischief (none I’ll admit to here, anyway), but because I spend most days writing the same three lines of SQL over and over again. If you've ever been paranoid about a table's content, you might know what I'm talking about. It’s the aggregate COUNT(*) grouped by a date field, ordered by date DESC. The output of that...
Hi fellow data professional! In a previous newsletter, I mentioned an idea that I wanted to explore deeper. At the risk of double-quoting a la The Office’s Michael Scott quoting Wayne Gretzky (“You Miss 100% Of The Shots You Don’t Take - Waynze Gretzky - Michael Scott”), here is the idea. “To be marketable as a candidate, you don’t just want to show how you can go from A to B (requirements->pipeline). You need to go from A to C (requirements->pipeline->scale/support).” You might be asking...