Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #21] Data Pipeline Non-Negotiables
Extract. Transform. Read.A newsletter from Pipeline Hi past, present or future data professional! Despite falling into the realm of engineering, data infrastructure construction is a bit like basic art. At times building a data pipeline is as simple as filling in one of those color-by-numbers books. Other times, the process of extracting and ingesting data can be as abstract and disconnected as paint flicked onto a canvas, Jackson Pollack style. No matter the complexity of your build, there are always certain brushes, a.k.a. non-negotiables, you should paint with to create intuitive and robust pipelines. I consider the following recommendations to be non-negotiable because they serve the most basic goal of a data pipeline: Providing reliable, prompt and accurate data to data consumers. A non-negotiable you must include in not only data pipelines, but programmatic scripts at large, is a clear, consistent and accessible form of logging. Good logs will concisely reflect what is going on within a script, revealing insights about each function or step as it is executed. Learn more about the importance of logging and best practices here. Going hand-in-hand with logging is the capturing of and reference to API status codes. While not all APIs will emit similar text messages when a response is triggered, there are universal codes like 200 that can be helpful in indicating the presence of data or other attributes and distinguish an unsuccessful request from a successful effort. Once you have the data, I’d suggest, as a non-negotiable, that you keep it in a consistent format. It might be nice being able to iterate through columns in a data frame, convert it to JSON, and then convert to a final data frame, but the resources required to execute the transformations and redundancy of the operations makes this inefficient. If you have to do significant work to unnest data, for instance, it may be better and more efficient to keep your data in JSON form. Finally, one of the worst things a pipeline can do (after breaking) is generate duplicate data. Nearly every one of my work builds includes what I call a “refresh” query that deletes the current date’s data as the pipe runs. This means that if the pipeline has to run again, it will generate the exact same output. The word for maintaining state like this is “idempotent.” In an org running hundreds of pipelines, you don’t want to create the 1 pipe with an uncontrollable output. To review, non-negotiables include:
This week's links:
What did I miss? Reply to this email and let me know. Thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Extract. Transform. Read. A newsletter from PipelineToDE Hi past, present or future data professional! Despite crushing autocorrect scenarios, most AI code assistants like CoPilot miss a critical step when helping developers of any experience level: Validation. Arguably, leveraging an AI Agent to validate a code’s quality is on the user. But a surprising amount of experienced programmers are taking the worrying approach of believing an AI’s first “thought” when it comes to code that will...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! A data science manager recently gave me some blunt, liberating advice over coffee: “If a team lead really cares what cloud technology you know (AWS, GCP, etc.) and doesn’t consider transferable experience… run.” This critical advice, which informs the conclusion of my soon-to-be-released ebook on data engineering project development, cuts to the core of a major problem in data hiring: The...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present, or future data professional! For all of the latest tech trends that come and go, one idea has always persisted in the tech world: Longevity isn’t cool. I say this because I recently hit a professional milestone: 4 years with my current organization. My career trajectory offers a counter-narrative to the “job-hopping is the only way to succeed” mentality. My goal isn't to convince you to be a “lifer,” but to demonstrate...