Extract. Transform. Read.A Newsletter From Pipeline Hi past, present or future data professional! Since today marks Thanksgiving in the US, I hope this reaches you before your eyes glaze over from the tryptophan-induced turkey coma we all inevitably slip into. While today is a day of gratitude, from a data engineering perspective, I’d like to focus, instead, on the under-the-radar tasks that can make a difference at this time of year—even if they don’t gain you any recognition at work. The reason you should consider doing (or assigning) these tasks in the near future is because, for many data teams that support businesses, Q4 is when things slow down. With many companies closing for the last week of December, it’s important to complete “housekeeping” work to keep interruptions minimal so you don’t have to fix a pipeline after an eggnog or two. Update These Documentation Areas Ahh documentation… Everyone needs it but no one wants to write it. Attempting to overhaul several repos or products’ worth of documentation is daunting. But to simplify an already thankless task, at a minimum ensure your documentation clearly identifies the following.
Delete Unused Resources In many orgs, Q4 represents “pencils down” for the financial divisions responsible for tracking revenue and, more importantly, expenses. I’ve said in the past that if what you’re working on isn’t directly generating revenue, the next best thing is optimizing costs/resource consumption. Deleting unused resources like SQL tables must first happen at the individual level. I can’t tell you how many tables I currently have in production right now with the “_test” suffix. Since the work day inevitably gets busy, it may be a good idea to institute a quarterly deletion. And it’s an even better idea to automate the calculation of your savings. Enable Logs I’d bet that if your production scripts are missing one thing it is a consistent form of logging. Simply writing a “Beginning script” and “data loaded” message isn’t going to cut it. If you’re stuck after logging.info(), read my perspective on how to effectively communicate execution steps using your logs. If you’re working on cloud-based infrastructure, you’ll want to assure your logs are synced with your SaaS/PaaS provider. It’s not very helpful if you can only see error logs locally. You may not get promoted for submitting 100 PRs for adding logging to hastily-written production scripts, but you will save yourself and your team significant time debugging. Ensure Your Privacy Policy Is Updated The holidays usher in a time of year when companies are unusually generous, offering parties, gifts and bonuses. The last thing you want to do is get slapped with a fine for failing to comply with an increasing number of international data privacy policies. This slow time is a good time to volunteer to audit your highly sensitive data sources and, if necessary, leverage your cloud provider’s data protection tools to encrypt and/or flag problematic fields. Knowing your data is sufficiently encrypted and your builds are extensively documented should allow you to sleep much more soundly during your post-feast nap. Here are this week’s links:
Questions? I’ll try to answer between bites: zach@pipelinetode.com Happy Thanksgiving and thanks for ingesting, -Zach Quinn |
Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s never good when you wake up to this from a coworker: 💀 The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch. The thing is, for a first-time violation, I can understand why testing is an...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s been a busy fall; I currently have 14 tasks in various states of development. Right now my JIRA board looks like I just won bingo—twice. Unfortunately when you climb the tech ladder things only get busier which means you’re going to burn out unless you take steps toward proactivity. For me this means learning which tasks I don’t need to (and really shouldn’t) do manually. And before you...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Despite falling into the realm of engineering, data infrastructure construction is a bit like basic art. At times building a data pipeline is as simple as filling in one of those color-by-numbers books. Other times, the process of extracting and ingesting data can be as abstract and disconnected as paint flicked onto a canvas, Jackson Pollack style. No matter the complexity of your build, there...