Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #31] Master An Underappreciated DE Skill
Extract. Transform. Read.A newsletter from Pipeline. *Today's edition was initially published on Medium on 12/10/24 Hi past, present or future data professional! I’ve recently been honing a data engineering skill that might not occur to you—drawing. When I first started my data engineering job 3+ years ago, any description or information related to my code would be in written form. This meant everything from README documentation to illegible legal pad scribbles would be all I had to inform decisions about design and implementation. Lately, however, my tasks have grown in both complexity and volume. What I need to convey to myself and my team won't fit on one sheet of paper. And if it did, it wouldn’t make a bit of sense. So I’ve turned to diagramming tools. I use tools like Microsoft Visio and Draw IO to create clear depictions of pipelines. This makes it easy to:
And, finally, the act of creating an architecture (arch) diagram subtly communicates something: That significant thought went into what you want to propose or present. Anyone can scribble in a shared doc or reference a notebook of ideas. Taking the time to spend time compiling a visualization demonstrates intent and care. This translates to you communicating your investment in making sure your build is possible and functional. Otherwise, it’s back to the (literal) drawing board. Feel free to scribble down this week’s links.
Questions? zach@pipelinetode.com Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! SQL Lite, the database you most likely learned SQL on, is built atop 100k lines of source code. Sound like a lot? Compare that to Chromium, the engine for Google Chrome, which boasts 30+ million lines of code under the hood. Shortly after acquiring Twitter/X, the world's first trillionaire, Elon Musk, famously asked engineers to tell him how many lines of code they wrote per day, igniting a debate among engineers throughout the software and data domains. When I...
Hi fellow data professional! If you read my note on Tuesday you’ll know I’m coming off of the data engineering week from hell that seeped into my personal life, and delayed the launch of something cool I was planning to share with you; if you want to know more about that, scroll to the end of this message. Last week a flagship data source had a major problem and since it’s within my ownership area, I was the one with the knowledge and responsibility to fix it. I wanted to share the experience...
Hi fellow data professional! Hardly a work day goes by without receiving a request from a data analyst. They range from the mundane “Can you add this column?” to the occasional emergency “The data didn’t load all weekend and the leadership call starts in 15 minutes!” At the end of a jam-packed week I received an unusual request: Help with a Python script. My teammate wanted to know: Best practices How to commit to GitHub What the best way to deploy is They admitted the task was simple,...