Extract. Transform. Read.

A newsletter from Pipeline.

*Today's edition was initially published on Medium on 12/10/24

Hi past, present or future data professional!

I’ve recently been honing a data engineering skill that might not occur to you—drawing.

When I first started my data engineering job 3+ years ago, any description or information related to my code would be in written form. This meant everything from README documentation to illegible legal pad scribbles would be all I had to inform decisions about design and implementation.

Lately, however, my tasks have grown in both complexity and volume. What I need to convey to myself and my team won't fit on one sheet of paper. And if it did, it wouldn’t make a bit of sense.

So I’ve turned to diagramming tools.

I use tools like Microsoft Visio and Draw IO to create clear depictions of pipelines.

This makes it easy to:

• Communicate an idea universally - I work with colleagues around the world so sometimes it’s helpful to communicate in a way that isn’t exclusively English
• Highlight pain points - Applying a different color draws the eye more than an all-caps TO-DO or FIX ME would
• Make quick edits/rearrange components - have you ever tried rearranging paragraphs or pages of text minutes before a meeting? There’s an entire industry dedicated to this called copyediting—and it can’t be done 5 min before a Zoom call
• Visualize independent/dependent tasks - this is critical when revising data pipelines that rely on upstream checks to trigger certain associated actions

And, finally, the act of creating an architecture (arch) diagram subtly communicates something: That significant thought went into what you want to propose or present.

Anyone can scribble in a shared doc or reference a notebook of ideas. Taking the time to spend time compiling a visualization demonstrates intent and care.

This translates to you communicating your investment in making sure your build is possible and functional.

Otherwise, it’s back to the (literal) drawing board.

Feel free to scribble down this week’s links.

• ​Why I Nearly Turned Down A 30k Raise And A Data Engineering Job​
• ​How to Outline Data Engineering Projects​
• ​Writing Your First DAG? Use SQL For More Accurate Data Availability Checks​
• ​Creating The Dashboard That Got Me A Data Analyst Job Offer​

Questions? zach@pipelinetode.com​

Thanks for ingesting,

-Zach Quinn