Extract. Transform. Read.

A newsletter from Pipeline​

Hi past, present or future data professional!

A data science manager recently gave me some blunt, liberating advice over coffee: “If a team lead really cares what cloud technology you know (AWS, GCP, etc.) and doesn’t consider transferable experience… run.”

This critical advice, which informs the conclusion of my soon-to-be-released ebook on data engineering project development, cuts to the core of a major problem in data hiring: The industry is teaching and hiring cloud talent wrong. Recruiters often fixate on broad, specific tool names, missing the underlying principles of distributed systems and problem-solving. This creates a vicious cycle of anxiety for job seekers and leads to a mismanaged hiring process.

Job postings today list dozens of specific tools: EC2, S3, Python, Kubernetes, etc. For new engineers, this creates a sense of inadequacy. You might grasp the architectural "big picture", but feel unqualified because you only know half the tools listed. This approach is intimidating and drives away excellent candidates. The reality is that most engineers specialize in a subset of technologies; the ability to learn and adapt is far more valuable than knowing every tool.

The trend of emphasizing specific tools harms the hiring process. Recruiters treat their process like a keyword match, which leads to a surge of applicants who know the buzzword but lack practical experience. I've seen peers who could name-drop every new cloud release but struggled with foundational data engineering concepts like schema compatibility when it came time to build. This hiring method rewards "master parrots" instead of engineers who can truly solve business problems.

This focus on buzzwords fuels the boom in professional certifications. While these exams are thorough, many engineers pursue them just to pass resume screening, not to deepen their understanding. A professional certificate validates your ability to choose a use case from a list; it doesn't prove you can implement, debug, and maintain it in production. This creates a false sense of security, leading employers to hire certified candidates who still lack the practical skills they need.

By fixating on a specific vendor, companies overlook top engineering talent.

To avoid falling into one of the dreaded cloud boxes, copy the best engineers and work to become…

• Platform-agnostic: Realize that core concepts like distributed systems and scalability are not unique to a given vendor; pick the best solution even if it requires a “mash up” of cloud providers
• Unafraid to get “under the hood”: Read system design books like Designing Data-Intensive Applications (Martin Klepperman) or read Google’s white paper outlining the product that would become BigQuery​
• Resource fluent: Understand transferable consumption and pricing models to be able to speak intelligently about the business rationale behind decisions to go with one cloud platform over another

And to delve deeper into how company-centric views of cloud have broken the application process, read my blog post on the subject.

Before You Go…

Ready to trade generic projects for the framework that gets you noticed and lands interviews? Join the waitlist for the guide here.

Thanks for ingesting,

-Zach Quinn