Extract. Transform. Read.

A newsletter from Pipeline​

Hi past, present or future data professional!

It’s hardly controversial to say debugging is everyone’s least favorite part of programming.

One widely-used debugging method is the rubber duck method, popularized in Pragmatic Programming, which suggests you talk through your code, aloud, to an inanimate object.

Being able to speak intelligently about what prompted a technical decision is one of the most underrated data engineering skills. One of the advantages of entering a discipline considered a growing field is the sheer variety of tools available to automate, optimize and scale processes.

After core competencies like scripting and database querying, analyzing and choosing tools or “tooling” is a skill that makes managers salivate. Because tooling goes much deeper than choosing the “in” tool or whatever application your cloud provider is hawking over cheap bagels (this may or may not be inspired by a real-life incident…).

Tooling is where you get to combine technical & industry knowledge with a bit of creativity–because no decision maker wants to read documentation. They want to be “sold” on the method as hard as you were. And the good news? Practicing tooling isn’t just an on-the-job skill for employed data scientists and engineers.

It begins when you’re conceptualizing a personal project as you’re mulling:

• “Scala vs. Python vs. Javascript?”
• “Airflow vs. Dbt?”
• “Tableau vs. Looker?”
• “Sql Lite vs. literally anything else?”

If you’re a student, be sure to create documentation or, better yet, a LinkedIn post/blog entry/YouTube video with a “Tools used” section. In academic papers I gloss over “methodologies”, but in technical presentations I bolt upright.

And if you’re working in your first job and want to shape your team or org’s tech stack, here’s how I suggest you craft a pitch:

• If starting costs are prohibitive, partner with your sales team to propose a free or reduced cost trial to develop a proof of concept (POC)
• Consider the trade off between initial development time and time saved by implementation
• Mull future implications on scaling with your chosen tool; consider impacts in 1 year, 5 years and even 10 years
• When estimating cost, project a worst-case scenario—or at least be very conservative

Ultimately, be thorough and have some degree of appreciation for your chosen tech, because if your tooling pitch is successful you could be stuck working with it for a long time.

Thanks for ingesting,

-Zach Quinn