Extract. Transform. Read.

A newsletter from Pipeline​

Hi past, present or future data professional!

I’ve done some of my best data engineering work while walking my dogs. The good news is you can do the same by leveraging one of the most accessible, intuitive bash tricks I’ve discovered–tmux.

Ever since a teammate introduced me to the tmux framework, I’ve saved several hours of time and compute-consuming “background” tasks ranging from iterative API requests to multi-terabyte backfills. This frees me to concentrate on critical tasks such as architectural design and strategic planning, both of which often require deep thought and, increasingly frequently, face-to-face (or screen) meetings.

What is Tmux?

Tmux, short for terminal multiplexer, is a powerful tool that allows you to run programs in multiple windows and, crucially, detach from those windows while your scripts continue to run in the background. It's like having multiple virtual workspaces within your terminal, enabling you to multitask with unparalleled efficiency.

Imagine being able to launch a long-running data backfill, disconnect from your terminal, and then go focus on other pressing matters - all while the backfill continues uninterrupted as a background task. Within Google Cloud Platform I use tmux within the bash command line or “ssh” terminal associated with my Compute Engine virtual machine (VM) instances.

As long as I keep the VM running, I can minimize or entirely close the SSH window and move onto local applications or other GCP tools.

Why Tmux Is Tmagical

• Maximize Efficiency: Run background processes and free up your terminal for other work
• Preserve Progress: Ensure long-running background tasks continue uninterrupted, even if your connection drops or the operation times out
• Seamless Monitoring: Re-attach to sessions to check progress and verify completion of background operations
• Streamlined Workflow: Manage multiple background tasks within a single terminal window

For details on the tmux commands, I recommend this cheat sheet.

Tmux has become one of my go-to hacks for data engineering. There is a low bar to entry (it’s really only a few bash commands), and it offers a significant return on investment by dramatically improving your productivity and efficiency when managing and executing background processes.

Which means I can walk my dogs a bit slower (in Florida that really matters!).

To kick off your own background processes, dive into my tmux starter:

​https://medium.com/pipeline-a-data-engineering-resource/tmux-is-tmagic-a416cb3788ac​

Thanks for ingesting,

-Zach Quinn