Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #36] The "I" Word Every DE Needs
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! One of the dirty secrets about my job is how easy it can be to fix broken pipelines. Often I’m retriggering a failed DAG task or, if using a code-less pipeline, literally hitting refresh. In fact, “refresh” is a great example for one of the more abstract data engineering concepts: State. And, specifically the maintenance of state under any condition. This is the definition of an important I-word, “idempotency.” While idempotency sounds like an SAT word, it’s as simple as saying “Every time this process runs the result (end state) will be the same.” An easy-to-grasp example of idempotency is the Google Cloud BigQuery API’s “WRITE_TRUNCATE” property. If you run a pipeline with “WRITE_TRUNCATE”, your data will always be overwritten during the load step. A more precise version of implementing idempotency is something I include in nearly all my pipelines, a DELETE step. This is slightly more precise than overwriting data because I am specifying deletion for a particular window. But this means that when I run a job that deletes and inserts only yesterday’s data, the output will be the same each time, leaving historic data intact and avoiding the very real possibility of data loss. This is a very practical approach to designing data pipelines because you may get spur-of-the-moment requests to reload data or otherwise re-trigger your runs. When executed properly, idempotency is as easy as hitting page refresh. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! I had a very adult weekend after baby bedtime. I uncorked a bottle of wine, cracked open my laptop and… stayed up late making a dashboard tracking my assets and debts. While I learned some important high-level insights that will help me make financial decisions as I renovate and prepare to move into my home, I realized a basic development truth that could help anyone knee-deep in their own project. If your goal is commit or ship then the simplest implementation...
Hi fellow data professional! Next time you think you’ve tried everything in your job search, remember: I once worked with a guy who got hired at a national broadcast network on the strength of his parody rap. The intern, Jake, didn’t have a network or elite contacts, but he wanted an internship at The Tonight Show, a competitive role with over 10,000 applicants per semester. So, he recreated a shot-for-shot parody of a song previously performed on the show, rewritten with lyrics specifically...
Hi fellow data professional! For the past month I’ve been working on my most ambitious personal project: Purchasing and renovating a house. The first major upgrade? Replacing 70+-year-old windows. And while there’s probably a tech work comparison to gutting a legacy system to build anew, what I want to focus on is the deal I brokered and how you can use similar leverage in your interviews. Because I got $1200 off a house’s worth of windows as a result of market research, due diligence and a...