Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #36] The "I" Word Every DE Needs
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! One of the dirty secrets about my job is how easy it can be to fix broken pipelines. Often I’m retriggering a failed DAG task or, if using a code-less pipeline, literally hitting refresh. In fact, “refresh” is a great example for one of the more abstract data engineering concepts: State. And, specifically the maintenance of state under any condition. This is the definition of an important I-word, “idempotency.” While idempotency sounds like an SAT word, it’s as simple as saying “Every time this process runs the result (end state) will be the same.” An easy-to-grasp example of idempotency is the Google Cloud BigQuery API’s “WRITE_TRUNCATE” property. If you run a pipeline with “WRITE_TRUNCATE”, your data will always be overwritten during the load step. A more precise version of implementing idempotency is something I include in nearly all my pipelines, a DELETE step. This is slightly more precise than overwriting data because I am specifying deletion for a particular window. But this means that when I run a job that deletes and inserts only yesterday’s data, the output will be the same each time, leaving historic data intact and avoiding the very real possibility of data loss. This is a very practical approach to designing data pipelines because you may get spur-of-the-moment requests to reload data or otherwise re-trigger your runs. When executed properly, idempotency is as easy as hitting page refresh. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! It finally happened. I fell for a job scam. Luckily I realized my naivety after responding to the initial email. But let’s back up. We’ll examine Why this particular attempt was so “real” What made me skeptical How to prevent this from happening to you Established professionals in any field have the privileged problem of receiving unsolicited recruiter inquiries. If it’s from a random firm I typically move it to junk; if it’s a big name company, I give a look...
Hi fellow data professional! The best data skills to develop right now might just be cutting and measuring. While that statement might be a bit facetious, the hot media narrative is to push the idea of blue collar work as a viable fallback if you’re having trouble breaking into a conventional tech role. Outlets like CNN have touted the fact that data center engineer is the hottest role in tech. Executives, specifically Nvidia’s Jensen Huang, speculate that data center construction (despite...
Hi fellow data professional! This is the 100th week I’m reaching out into the void of the Internet to connect with you in order to democratize data engineering career knowledge. In the golden age of cable TV, shows would celebrate the 100th episode milestone by airing extended content like a 1-hour special for a sitcom that would typically consume 30 minutes. But I know your time is valuable so I’m going to do the opposite and make this a shorter newsletter than normal. Since I live what I...