[ETR #14] Build Your Staging Tables Faster/Safer


Extract. Transform. Read.

A newsletter from Pipeline: Your Data Engineering Resource

Presented by Basejump AI

Use natural language prompts to chat with your database in Basejump’s intuitive interface, or embed it directly in your application. Book your demo here.

Hi past, present or future data professional!

One thing that makes my work day easier is when I’m Google-ing (as all software developers do) a problem and I come across the holy grail of solutions: A one-line implementation.

Like anything, however, a one-liner that is too complex can become a bad thing. Think: Chained Pandas expressions that become unreadable. Or cramming a multi-line query inside of a BigQuery client method.

My favorite one line (at least in recent memory) is a clause used with SQL’s ALTER TABLE statement: RENAME TO. You may find renaming a table as compelling as schema creation. But this simple clause can be especially useful in lieu of a more dangerous phrase: CREATE OR REPLACE.

The RENAME command allows you to rename a table without having to completely recreate its contents–and risk a SQL statement failing and losing some or all of your data.

Specifically, I use RENAME TO when I want to convert a copy table with some change, like an updated schema, to a production table. I do so using these steps:

  • Create/backfill a staging table I’ll ultimately convert to prod
  • Use ALTER TABLE `dataset.production_table` RENAME TO `production_table_original`
  • Use ALTER TABLE `dataset.staging_table` RENAME TO `production_table`
  • Double-check all partitions, clustering specifications and metadata descriptions are identical between the tables

The best part is that this is a true one-liner. No chains–or headaches–involved.

To save you a headache, here are this week’s links:

If you want to read more about this method, I cover the process in more detail here.

Questions? You know where to find me: zach@pipelinetode.com.

Until next time–thanks for ingesting,

-Zach Quinn

Pipeline To DE

Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.

Read more from Pipeline To DE

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Despite falling into the realm of engineering, data infrastructure construction is a bit like basic art. At times building a data pipeline is as simple as filling in one of those color-by-numbers books. Other times, the process of extracting and ingesting data can be as abstract and disconnected as paint flicked onto a canvas, Jackson Pollack style. No matter the complexity of your build, there...

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! To those in the U.S.: Happy Halloween! In the spirit of the spooky season, I’d like to scare—I mean warn—you about 3 truly creepy trends that might give you goosebumps during a job search. A Shady Recruiter “Ghost” Writing Your Resume When in the job market, one of the first things you learn, after how to write a resume, is how to format one. I’m sure you know about headings, bullet points, etc....

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I recently participated in a technical design meeting that was derailed by a single, fundamental question. “Why?” Despite the fact that I worked with the particular data source we were discussing for nearly two years, I fell into the common trap of going “on autopilot” and failing to question the initial need for the data. At this point, you would think asking “why” of years’ worth of work would...