Extract. Transform. Read.

A newsletter from Pipeline​

Hi past, present or future data professional!

Data science just cracked the top 40… of jobs whose main functions are most likely to be replaced by AI. If you’re up to speed on your AI doomerism news you’ll know that at the end of July, Microsoft released a list of jobs across disciplines and industries that could be majorly disrupted by AI.

On a more positive economic outlook, data engineering is specifically cited as a growing role in the World Economic Forum’s Future of Job’s report.

The contrast in these two lists is striking, especially when data science was infamously one of the “sexiest” jobs of the 2010s.

The other reason the WEF report is interesting is because it explicitly names data engineering as a growing role, when our discipline routinely gets lumped in with data science, computer science or the ever-vague information science categories in federal jobs data.

So for our data science friends, the end is nigh–at least until they’re hired back to clean up AI’s spaghetti code mess. If you’re just dipping your toes into data and analytics engineering roles you’d likely attribute this staying power to an argument I routinely make to my non-tech friends and family: Data is fuel and DEs are the roughneckers piping it to hungry MLs and LLMs.

But data engineering is a sub specialty of software engineering, a field that remains resilient despite every attempt to thwart anyone from physically writing another line of code.

In 10 Years of AI Generative Slop, engineer/blogger Paul Bauer argues that a fundamental flaw of the AI replacing engineer life cycle is that engineers spend 100% of “development time” coding.

As Bauer puts it the “real” breakdown you can expect as a working SWE (or DE) is:

• 25% writing/generating code
• 25% operating/debugging/“fighting” your tech stack
• 25% meetings, admin tasks, mentoring, interviewing
• 16% (oddly specific imo) research/writing
• 9% Slack memes, reading newsletters (ok I threw in that last one)

The advantage DEs have is that we have the 10,000 foot perspective into surface-level and underlying levels of infra which will need to be maintained, optimized and scaled regardless of code output.

I don’t share this just to inject a copium-laced perspective that AI won’t at all touch or reshape the data engineering role before you get a chance to enter the industry.

I am suggesting that more than ever it is critical to understand how your technical work tangibly impacts the business you work for or aspire to work for. SMEs are becoming increasingly more valuable than SWEs.

If you’re veering toward a data science ML Ops path, I wouldn’t put much stock into Microsoft’s list. But I would make every attempt to reduce “research” projects in favor of landing opportunities to work on revenue-driving projects.

Across the board there is one engineer or employee that AI will have a difficult time replacing—one that consistently makes a business money.

And one more thing...

If you want to delve deeper into the argument against long-term AI adoption, you can read Paul Bauer’s article in full here.

Thanks for ingesting,

-Zach Quinn

​
​