Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
[ETR #14] Build Your Staging Tables Faster/Safer
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourcePresented by Basejump AI Use natural language prompts to chat with your database in Basejump’s intuitive interface, or embed it directly in your application. Book your demo here.
Hi past, present or future data professional! One thing that makes my work day easier is when I’m Google-ing (as all software developers do) a problem and I come across the holy grail of solutions: A one-line implementation. Like anything, however, a one-liner that is too complex can become a bad thing. Think: Chained Pandas expressions that become unreadable. Or cramming a multi-line query inside of a BigQuery client method. My favorite one line (at least in recent memory) is a clause used with SQL’s ALTER TABLE statement: RENAME TO. You may find renaming a table as compelling as schema creation. But this simple clause can be especially useful in lieu of a more dangerous phrase: CREATE OR REPLACE. The RENAME command allows you to rename a table without having to completely recreate its contents–and risk a SQL statement failing and losing some or all of your data. Specifically, I use RENAME TO when I want to convert a copy table with some change, like an updated schema, to a production table. I do so using these steps:
The best part is that this is a true one-liner. No chains–or headaches–involved. To save you a headache, here are this week’s links:
If you want to read more about this method, I cover the process in more detail here. Questions? You know where to find me: zach@pipelinetode.com. Until next time–thanks for ingesting, -Zach Quinn |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and nearly 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I dreaded entering the job market after my data science master's. I felt like I knew more than a data analyst but less than a professional data scientist. I've since realized my program was more effective than I thought, but it couldn't prepare me for the key areas like cloud deployments and real-world problem-solving I had to learn on the job as a data engineer. And I’ve noticed these gaps in...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! If you live in the U.S., this week marks the end of back to school season; though, if you’re like my southern relatives, you’ve been back since July. The closest feeling most adults get to back to school (aside from the teachers), is starting a new job. While a new org, title and compensation package represents new opportunities, it’s also easy to feel like the “new kid”, which can lead to being...
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I once participated in a remote job interview in which the interviewer was on the video call while driving... and smoking. While that instance was among the most memorable interview experiences (for the wrong reasons), I’ve had just as many interviews that have blended together and faded into the recesses of my mind. The common denominator, however, was the insistence on asking one question. The...