Extract. Transform. Read.

A newsletter from Pipeline: Your Data Engineering Resource​

Hi past, present or future data professional!

Few aspects of data engineering are as shame-inducing as saying, after a failed deployment, “But it ran in my environment!”

In my first year as a data engineer I was that guy who made excuses like this and grew frustrated that I would complete a build and then struggle to push it over the finish line.

Here’s what helped me:

• Learning the subtle but important difference between a dependency-related error and a code-oriented issue
• Taking time to actually read documentation rather than skimming it
• Understanding my chosen cloud platform (Google Cloud Platform)
• Distinguishing the important bits of an error string to properly Google a mistake (both in local and cloud dev contexts)
• Not running to my seniors for answers; StackOverflow, Medium, Reddit and platform-specific communities (like Google Community) are hive minds for solving specific errors
• Logging status codes and outputs; you can’t fix what you can’t see
• Creating “clean” dev environments that contain only the dependencies I need

I don’t track my deployment success rate (probably for the best given my initial failures), but I estimate that following the above advice has reduced my failure rate from 20% to between 1-5%.

None of these bullets, however, is a substitute for hands-on experience.
​

To step through your own deployment, enroll in my free 5-day Deploy Your First Cloud Function course.
​
Enroll here: https://pipe_line.ck.page/33a3ad0f36​

As always, please send me any questions: zach@pipelinetode.com.

Thanks for ingesting,

-Zach