Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Can Your Data Pipelines Be Dangerous?
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourceHi past, present or future data professional! Data engineering can be dangerous; ok—not, like, physically, but by building and maintaining data infrastructure, data engineers are given a surprising amount of access and responsibility. Every commit, table alteration and deletion must be made with care. It took 2 years, but I finally learned a shortcut to make developing SQL staging tables less risky and more efficient. Even seemingly minor mistakes like joining on the wrong key can result in losing days or months of valuable data, which can be equal to hundreds of thousands or millions of dollars in revenue visibility. Outside of code mistakes, not paying attention to logistic factors like vendor contracts and API usage can not only result in downtime, in a worst-case scenario it can lead to an all-out blackout. If the stakes sound ominous, I’d suggest examining the root of your hesitation to work more confidently and efficiently—it may even be the code itself. There is a happy medium between freely building data pipelines and using the appropriate guard rails. As long as you take your time and don’t commit code directly to the main branch then you can do data engineering safely and avoid bursting your pipelines. For those who are anti-virus minded, here are this week’s links as plain text:
P.S. Want to learn how to go from code to automated pipeline? Take advantage of my 100% free email course: Deploy Google Cloud Functions In 5 Days. Thanks for ingesting, -Zach |
Extract. Transform. Read.
Reaching 20k+ readers on Medium and over 3k learners by email, I draw on my 4 years of experience as a Senior Data Engineer to demystify data science, cloud and programming concepts while sharing job hunt strategies so you can land and excel in data-driven roles. Subscribe for 500 words of actionable advice every Thursday.
Hi fellow data professional! Big news from my home base of Orlando: Disney hired a new CEO with a pay package of nearly $40 million. If you read beyond the headline you’ll see that his base salary is “only” 2.5 million with the possibility of up to a 250% target incentive and some $26-ish million in stock options. This is why you, the job seeker, need to think beyond base salary and look at TC. Total compensation. Thanks to labor transparency laws passed in hiring hubs like New York and...
Hi fellow data professional! Once, during a virtual interview, I had to nod politely as my interviewer apologized for coughing after their cigarette. Oh, and to make this situation even more cringe—they were driving. Some industries design stressful interview processes to psychologically test a candidate’s poise under pressure. Luckily (for the most part) the software engineering field is not included in this basket of high-stress tests. Sure, we are subjected to moderate stress in the form...
Hi fellow data professional! On a recent holiday, a family member and I were strolling along a beach, talking about AI disruption (relaxing, I know). He, an attorney, assured me his job was AI-proof and jokingly offered to hire me when AI takes my data engineering job. If you ask executives at most companies, they’d find several flaws in that argument. Over 80% of technical executives, including Chief Data Officers and Chief AI Officers, consider data engineering to be an essential role...