Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
[ETR #14] Build Your Staging Tables Faster/Safer
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourcePresented by Basejump AI Use natural language prompts to chat with your database in Basejump’s intuitive interface, or embed it directly in your application. Book your demo here.
Hi past, present or future data professional! One thing that makes my work day easier is when I’m Google-ing (as all software developers do) a problem and I come across the holy grail of solutions: A one-line implementation. Like anything, however, a one-liner that is too complex can become a bad thing. Think: Chained Pandas expressions that become unreadable. Or cramming a multi-line query inside of a BigQuery client method. My favorite one line (at least in recent memory) is a clause used with SQL’s ALTER TABLE statement: RENAME TO. You may find renaming a table as compelling as schema creation. But this simple clause can be especially useful in lieu of a more dangerous phrase: CREATE OR REPLACE. The RENAME command allows you to rename a table without having to completely recreate its contents–and risk a SQL statement failing and losing some or all of your data. Specifically, I use RENAME TO when I want to convert a copy table with some change, like an updated schema, to a production table. I do so using these steps:
The best part is that this is a true one-liner. No chains–or headaches–involved. To save you a headache, here are this week’s links:
If you want to read more about this method, I cover the process in more detail here. Questions? You know where to find me: zach@pipelinetode.com. Until next time–thanks for ingesting, -Zach Quinn |
Pipeline To DE
Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! In 2024 I published roughly 75 stories, mostly about data engineering or technology; understandably, with the pace of life and media, you most likely missed something I hope you’ll find valuable and actionable. Keeping with one of my core beliefs, that data-driven tools should result in both professional enrichment and reduce personal problems, my methodology for picking stories out of that...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! If you’ve ever seen the legendary American sitcom Seinfeld, you might be familiar with the fictional holiday the characters create, festivus, “A festival for the rest of us.” As a rejection of conventional winter holidays like Christmas/Haunnukah, a core part of festivus is the “airing of grievances.” While I have yet to attempt this in real-life, I’ve spent the past two years airing my...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s the most wonderful time of the year… performance reviews. Depending on your outlook on your job/org, that “wonderful” could be sarcastic. However, if you’ve landed on your manager’s nice list, this can be the time to recap your achievements to maintain credibility and work toward the next rung on the corporate ladder. If you’re on a small team serving demanding stakeholders it’s possible...