Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #36] The "I" Word Every DE Needs
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! One of the dirty secrets about my job is how easy it can be to fix broken pipelines. Often I’m retriggering a failed DAG task or, if using a code-less pipeline, literally hitting refresh. In fact, “refresh” is a great example for one of the more abstract data engineering concepts: State. And, specifically the maintenance of state under any condition. This is the definition of an important I-word, “idempotency.” While idempotency sounds like an SAT word, it’s as simple as saying “Every time this process runs the result (end state) will be the same.” An easy-to-grasp example of idempotency is the Google Cloud BigQuery API’s “WRITE_TRUNCATE” property. If you run a pipeline with “WRITE_TRUNCATE”, your data will always be overwritten during the load step. A more precise version of implementing idempotency is something I include in nearly all my pipelines, a DELETE step. This is slightly more precise than overwriting data because I am specifying deletion for a particular window. But this means that when I run a job that deletes and inserts only yesterday’s data, the output will be the same each time, leaving historic data intact and avoiding the very real possibility of data loss. This is a very practical approach to designing data pipelines because you may get spur-of-the-moment requests to reload data or otherwise re-trigger your runs. When executed properly, idempotency is as easy as hitting page refresh. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Extract. Transform. Read. A newsletter from PipelineToDE Hi past, present or future data professional! I want to share the single most important realization I had back in the summer of 2021. I was burned out, juggling two part-time jobs, trying to plan a wedding, and drowning in full-time job applications. I felt overwhelmed and underprepared as I plunged into a sea of candidates I perceived to be more intelligent and better "fits" than me. My portfolio was full of the usual Titanic, Iris,...
Extract. Transform. Read. A newsletter from PipelineToDE Hi past, present or future data professional! One of the most validating and terrifying professional moments is reaching the final interview round. It is in this context that you meet candidacy’s final boss, who incidentally, usually ends up being your boss' boss. Specifically I’m referring to the department executive responsible for bringing in additional headcount, i.e. you. While this may sound intimidating, the role of the executive...
Extract. Transform. Read. A newsletter from PipelineToDE Hi past, present or future data professional! If you’re a job seeker in the data space, your GitHub portfolio has only one job: To act as a calling card that gets you to the next step of the hiring process. Too often, I review portfolios for potential referrals and see brilliant code buried under structural mistakes that have nothing to do with programming skill. Your GitHub is not just cloud storage for your code; it’s a public display...