Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #23] Testing... Testing
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! It’s never good when you wake up to this from a coworker: 💀 The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch. The thing is, for a first-time violation, I can understand why testing is an afterthought for new engineers. Schools and courses emphasize local output over production so testing feels like an extra step. To properly test code, you need to configure a clean, production-adjacent environment. If you’re new to this concept, here are 2 of my favorites along with an unusual choice. The safe choice: Virtual Environment I use two virtual environments that can be configured interchangeably: Pyenv and Venv. Pyenv is easy to configure and use within a terminal in a “professional” IDE like VS code. Pyenv is ideal because it allows me to create an environment from a blank slate each time. Venv is another option. Instead of using Venv in VS Code this is how I set up a virtual Python environment inside of a Virtual Machine (VM). Read more to learn how to set up a quick, durable sandbox in a Compute Engine VM. The portable option: Docker I’ll confess: I didn’t used to be a fan of Docker. I didn’t really “get” containerization and could set up a virtual environment using the processes described above. However, I learned that Docker’s true power is its portability. Not only can I create a clean slate (an image), I can push this to a registry to create testing configs before I test script changes in production. Powerful stuff. The one issue I had was authenticating with GCP; I describe my solution here. The unusual pick: Jupyter Notebook Jupyter Notebook gets a bad rap in the data engineering community. Seen as a tool for data analysts and data scientists, it doesn’t quite make sense for data engineers to develop and test in an environment best known for its nicely rendered outputs. But buying into that argument would cause you to miss out on some useful features and, frankly, a nice UX. And so you don’t have to search for resources, here are this week’s links.
What’s your preferred testing ground? Let me know: zach@pipelinetode.com. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s hardly controversial to say debugging is everyone’s least favorite part of programming. One widely-used debugging method is the rubber duck method, popularized in Pragmatic Programming, which suggests you talk through your code, aloud, to an inanimate object. Being able to speak intelligently about what prompted a technical decision is one of the most underrated data engineering skills. One...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! If you’re like me, in school you were always envious of your classmates that may not have applied themselves academically but were “good test takers.” Fortunately (for them at least), these folks would likely do well on what is quietly becoming the SAT of programming the GCA, or General Coding Assessment. Now, the General Coding Assessment isn’t any kind of board certifying test like the Bar...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! While many tech-oriented companies have (in one way or another) reneged on remote working arrangements, my employer made an extreme gesture to demonstrate its commitment to the ongoing office-less lifestyle: It removed an entire floor of our two-floor New Jersey office space. Other companies, like Spotify, have unveiled slogans like “Our employees aren’t children. Spotify will continue working...