Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! If you’re like me, in school you were always envious of your classmates that may not have applied themselves academically but were “good test takers.” Fortunately (for them at least), these folks would likely do well on what is quietly becoming the SAT of programming the GCA, or General Coding Assessment. Now, the General Coding Assessment isn’t any kind of board certifying test like the Bar...
9 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! While many tech-oriented companies have (in one way or another) reneged on remote working arrangements, my employer made an extreme gesture to demonstrate its commitment to the ongoing office-less lifestyle: It removed an entire floor of our two-floor New Jersey office space. Other companies, like Spotify, have unveiled slogans like “Our employees aren’t children. Spotify will continue working...
15 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! The only thing worse than summer temperatures (if you’re in the western hemisphere, that is) is a summer job search. Conventionally, summer isn’t the best time to apply for work; you could probably tell this if you’re currently working and find yourself accepting an overwhelming amount of OOO cal invites. If you are braving the heat of the job market, I want to share a more targeted and...
22 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Well, it finally happened; AI has replaced a build I created and I’ve been made redundant. Thankfully, the person that created the AI integration was also me. And I did this on personal time so this isn’t an apocalyptic scenario. I’ve previously written about a handful of tools I created to optimize the “busy work” of blogging. One of the ways is by adding links to past relevant articles and...
29 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! For the first time since the birth of the Internet, the prevalence of AI summaries has damaged Google’s Search business, possibly irreparably. And while this might simply be a sign the times they are a changin’ (I just watched that new Bob Dylan movie), it points to a harsher reality. These days search universally sucks; I’ve found this is especially true when readers, like yourself, want to dig...
about 1 month ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present, or future data professional! A lagging SQL query caused me to nearly miss my flight home. Ok, that’s maybe a bit of an exaggeration; while I would have still gotten on the plane, the query in question did take nearly 2 hours to run… even after working hours! The frustrating thing about SQL and programming in general is that no matter how technically perfect your code is, you will almost always bump up against resource...
about 1 month ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I don’t know about you, but this email won’t be the only thing I scroll through today. Incidentally, one of the most infinitely scroll-able (and doom scroll-able) sites, Reddit, was my source for a data project demonstrating alerting systems. And the best part? You can and should absolutely steal this approach to gain an understanding of how to monitor ingested data and trigger alerts. This...
about 2 months ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! I’ve done some of my best data engineering work while walking my dogs. The good news is you can do the same by leveraging one of the most accessible, intuitive bash tricks I’ve discovered–tmux. Ever since a teammate introduced me to the tmux framework, I’ve saved several hours of time and compute-consuming “background” tasks ranging from iterative API requests to multi-terabyte backfills. This...
about 2 months ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Remember when everyone was scrambling to become a "prompt engineer" or "metaverse architect?" Last week The Wall Street Journal published an article (The Hottest AI Job of 2023 Is Already Obsolete) exposing how the "hottest" AI job, the “prompt engineer”, quietly faded. It's a familiar and eye roll-inducing story. The tech industry has been guilty of heralding the “decade’s next job”...
2 months ago • 2 min read