Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #36] The "I" Word Every DE Needs
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! One of the dirty secrets about my job is how easy it can be to fix broken pipelines. Often I’m retriggering a failed DAG task or, if using a code-less pipeline, literally hitting refresh. In fact, “refresh” is a great example for one of the more abstract data engineering concepts: State. And, specifically the maintenance of state under any condition. This is the definition of an important I-word, “idempotency.” While idempotency sounds like an SAT word, it’s as simple as saying “Every time this process runs the result (end state) will be the same.” An easy-to-grasp example of idempotency is the Google Cloud BigQuery API’s “WRITE_TRUNCATE” property. If you run a pipeline with “WRITE_TRUNCATE”, your data will always be overwritten during the load step. A more precise version of implementing idempotency is something I include in nearly all my pipelines, a DELETE step. This is slightly more precise than overwriting data because I am specifying deletion for a particular window. But this means that when I run a job that deletes and inserts only yesterday’s data, the output will be the same each time, leaving historic data intact and avoiding the very real possibility of data loss. This is a very practical approach to designing data pipelines because you may get spur-of-the-moment requests to reload data or otherwise re-trigger your runs. When executed properly, idempotency is as easy as hitting page refresh. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional and Happy New Year! In the second half of 2025, I made a radical choice: I (largely) stopped blogging. Over the past year, Medium (where I host my content) made a series of changes that de-prioritizes technical content, leading to the departure of several major publications, including Toward Data Science. Pair that platform disillusionment with a bit of burnout, and the result is a feeling that it’s time for a change. For 75+ weeks, I’ve preferred concise,...
Hi fellow data professional - Merry Christmas and Happy Holidays! Since an email is probably one of the least exciting things to open on Christmas morning, I'll keep this brief. As a thank you for subscribing and reading the newsletter this year, I'd like to offer a gift: My FREE guide to web scraping in Python. Centered around 3 "real world" projects, the guide highlights the importance of being able to retrieve, interpret and ingest unstructured data. Get your guide here. Have a restful...
Hi fellow data professional! Once thought to be a purely back office role, data engineering is undergoing a radical transformation and gaining a new responsibility: Front-end deployment. The folks already deploying applications in this capacity are known, incidentally, as forward deployed software engineers or forward deployed engineers (FDEs). Before you worry about needing to learn JavaScript or other web programming paradigms, know that I’m referring to the preparation, deployment and...