Extract. Transform. Read.

A newsletter from Pipeline​

Hi past, present or future data professional!

It’s never good when you wake up to this from a coworker:

💀

The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch.

The thing is, for a first-time violation, I can understand why testing is an afterthought for new engineers. Schools and courses emphasize local output over production so testing feels like an extra step.

To properly test code, you need to configure a clean, production-adjacent environment. If you’re new to this concept, here are 2 of my favorites along with an unusual choice.

The safe choice: Virtual Environment

I use two virtual environments that can be configured interchangeably: Pyenv and Venv.

Pyenv is easy to configure and use within a terminal in a “professional” IDE like VS code. Pyenv is ideal because it allows me to create an environment from a blank slate each time.

Venv is another option. Instead of using Venv in VS Code this is how I set up a virtual Python environment inside of a Virtual Machine (VM).

​Read more to learn how to set up a quick, durable sandbox in a Compute Engine VM.

The portable option: Docker

I’ll confess: I didn’t used to be a fan of Docker. I didn’t really “get” containerization and could set up a virtual environment using the processes described above.

However, I learned that Docker’s true power is its portability. Not only can I create a clean slate (an image), I can push this to a registry to create testing configs before I test script changes in production.

Powerful stuff.

The one issue I had was authenticating with GCP; I describe my solution here.

The unusual pick: Jupyter Notebook

Jupyter Notebook gets a bad rap in the data engineering community. Seen as a tool for data analysts and data scientists, it doesn’t quite make sense for data engineers to develop and test in an environment best known for its nicely rendered outputs.

But buying into that argument would cause you to miss out on some useful features and, frankly, a nice UX.
​I recently discussed my internal struggle between VS Code and Jupyter so you can do your own soul-searching.

And so you don’t have to search for resources, here are this week’s links.

• ​Set Up A Virtual Environment In A Compute Engine VM In 5 Min.​
• ​Authenticate Google Cloud In Docker Without Getting Your Creds Stolen​
• ​Jupyter Notebook vs. VS Code​

What’s your preferred testing ground? Let me know: zach@pipelinetode.com.

Thanks for ingesting,

-Zach Quinn