Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! It’s never good when you wake up to this from a coworker: 💀 The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch. The thing is, for a first-time violation, I can understand why testing is an afterthought for new engineers. Schools and courses emphasize local output over production so testing feels like an extra step. To properly test code, you need to configure a clean, production-adjacent environment. If you’re new to this concept, here are 2 of my favorites along with an unusual choice. The safe choice: Virtual Environment I use two virtual environments that can be configured interchangeably: Pyenv and Venv. Pyenv is easy to configure and use within a terminal in a “professional” IDE like VS code. Pyenv is ideal because it allows me to create an environment from a blank slate each time. Venv is another option. Instead of using Venv in VS Code this is how I set up a virtual Python environment inside of a Virtual Machine (VM). Read more to learn how to set up a quick, durable sandbox in a Compute Engine VM. The portable option: Docker I’ll confess: I didn’t used to be a fan of Docker. I didn’t really “get” containerization and could set up a virtual environment using the processes described above. However, I learned that Docker’s true power is its portability. Not only can I create a clean slate (an image), I can push this to a registry to create testing configs before I test script changes in production. Powerful stuff. The one issue I had was authenticating with GCP; I describe my solution here. The unusual pick: Jupyter Notebook Jupyter Notebook gets a bad rap in the data engineering community. Seen as a tool for data analysts and data scientists, it doesn’t quite make sense for data engineers to develop and test in an environment best known for its nicely rendered outputs. But buying into that argument would cause you to miss out on some useful features and, frankly, a nice UX. And so you don’t have to search for resources, here are this week’s links.
What’s your preferred testing ground? Let me know: zach@pipelinetode.com. Thanks for ingesting, -Zach Quinn |
Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s the most wonderful time of the year… performance reviews. Depending on your outlook on your job/org, that “wonderful” could be sarcastic. However, if you’ve landed on your manager’s nice list, this can be the time to recap your achievements to maintain credibility and work toward the next rung on the corporate ladder. If you’re on a small team serving demanding stakeholders it’s possible...
Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! This holiday season will be a little less bright thanks to my lack of personal GitHub commits. Like you, I began 2024 full of ideas and motivation that, let’s be honest, was depleted by the end of Q1 when I was cranking out enough code at work that would please even notorious code volume stickler Elon Musk. Despite my lacking output, I managed to hunker down to create useful “bespoke” (a.k.a....
Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! When I worked at Disney there was one line (aside from “Have a Magical Day”) that was borderline beaten into us: “We are all custodial employees.” The line meant, of course, to keep areas under your purview neat and presentable (“show ready” in Disney-speak). Using the same logic, I’d like to emphasize that while the various data roles (data analyst, data scientist, data engineer, etc.) have...