[ETR Vol. 11] Data Engineering, Fast And Slow


Extract. Transform. Read.

A newsletter from Pipeline: Your Data Engineering Resource

Hi past, present or future data professional!

I bet when you think of data engineering (or software engineering, for that matter) you don’t think about development pace. After all, the buzzworthy anecdotes and lofty promises surround performance of a build, not the speed of its architects.

But adaptability and the ability to work very quickly at times is a role expectation you won’t see listed on a job opening. I’ll be very honest–I’ve had days and weeks where things are painfully slow. You almost want a pipeline to break just to say you did something during a work day.

Then there are the rare days where you receive an assignment in the morning and are expected to complete it by day’s end. Being able to calm your nerves, act methodically and gather concrete requirements is the name of the game in either scenario.

And don’t forget, even if you’re not the engineer handling an urgent task, do your part to be the ultimate team player to support classmates, peers and other professionals who seem overwhelmed.

Reduce complexity. Remove burdens–no matter what pace you operate at (or are dictated).

If I can help you by answering a question, shoot me an email: zach@pipelinetode.com.

To leave you plenty of development time, here are this week’s links as plain text:

Thanks for ingesting,

-Zach

Pipeline To DE

Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.

Read more from Pipeline To DE

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Despite falling into the realm of engineering, data infrastructure construction is a bit like basic art. At times building a data pipeline is as simple as filling in one of those color-by-numbers books. Other times, the process of extracting and ingesting data can be as abstract and disconnected as paint flicked onto a canvas, Jackson Pollack style. No matter the complexity of your build, there...

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! To those in the U.S.: Happy Halloween! In the spirit of the spooky season, I’d like to scare—I mean warn—you about 3 truly creepy trends that might give you goosebumps during a job search. A Shady Recruiter “Ghost” Writing Your Resume When in the job market, one of the first things you learn, after how to write a resume, is how to format one. I’m sure you know about headings, bullet points, etc....

Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I recently participated in a technical design meeting that was derailed by a single, fundamental question. “Why?” Despite the fact that I worked with the particular data source we were discussing for nearly two years, I fell into the common trap of going “on autopilot” and failing to question the initial need for the data. At this point, you would think asking “why” of years’ worth of work would...