Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! The only thing worse than summer temperatures (if you’re in the western hemisphere, that is) is a summer job search. Conventionally, summer isn’t the best time to apply for work; you could probably tell this if you’re currently working and find yourself accepting an overwhelming amount of OOO cal invites. If you are braving the heat of the job market, I want to share a more targeted and...
14 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Well, it finally happened; AI has replaced a build I created and I’ve been made redundant. Thankfully, the person that created the AI integration was also me. And I did this on personal time so this isn’t an apocalyptic scenario. I’ve previously written about a handful of tools I created to optimize the “busy work” of blogging. One of the ways is by adding links to past relevant articles and...
21 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! For the first time since the birth of the Internet, the prevalence of AI summaries has damaged Google’s Search business, possibly irreparably. And while this might simply be a sign the times they are a changin’ (I just watched that new Bob Dylan movie), it points to a harsher reality. These days search universally sucks; I’ve found this is especially true when readers, like yourself, want to dig...
28 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present, or future data professional! A lagging SQL query caused me to nearly miss my flight home. Ok, that’s maybe a bit of an exaggeration; while I would have still gotten on the plane, the query in question did take nearly 2 hours to run… even after working hours! The frustrating thing about SQL and programming in general is that no matter how technically perfect your code is, you will almost always bump up against resource...
about 1 month ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I don’t know about you, but this email won’t be the only thing I scroll through today. Incidentally, one of the most infinitely scroll-able (and doom scroll-able) sites, Reddit, was my source for a data project demonstrating alerting systems. And the best part? You can and should absolutely steal this approach to gain an understanding of how to monitor ingested data and trigger alerts. This...
about 1 month ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I’ve done some of my best data engineering work while walking my dogs. The good news is you can do the same by leveraging one of the most accessible, intuitive bash tricks I’ve discovered–tmux. Ever since a teammate introduced me to the tmux framework, I’ve saved several hours of time and compute-consuming “background” tasks ranging from iterative API requests to multi-terabyte backfills. This...
about 2 months ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Remember when everyone was scrambling to become a "prompt engineer" or "metaverse architect?" Last week The Wall Street Journal published an article (The Hottest AI Job of 2023 Is Already Obsolete) exposing how the "hottest" AI job, the “prompt engineer”, quietly faded. It's a familiar and eye roll-inducing story. The tech industry has been guilty of heralding the “decade’s next job”...
about 2 months ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! While I generally find textbooks a bit dry (especially when it comes to code), I can stomach the occasional white paper. A formative white paper I read in grad school is entitled Dremel: Interactive Analysis of Web-Scale Datasets, which provides a crucial look "under the hood" at the origins of BigQuery. The Story of Dremel Before it became the BigQuery we know (and sometimes love), Google...
2 months ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! For years, a start-up cliche was being the “Uber” of (product, service, etc.). Now, it seems like any content platform wants to be the “Tik Tok” of a given subject area. Case in point for the latter: A fun app I came across called, fittingly, “Gittok.”* Like Tik Tok, Gittok feeds users an endless stream of distraction but instead of dance challenges it serves up a random GitHub repository, like...
2 months ago • 1 min read