Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! In 2024 I published roughly 75 stories, mostly about data engineering or technology; understandably, with the pace of life and media, you most likely missed something I hope you’ll find valuable and actionable. Keeping with one of my core beliefs, that data-driven tools should result in both professional enrichment and reduce personal problems, my methodology for picking stories out of that...
12 days ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! If you’ve ever seen the legendary American sitcom Seinfeld, you might be familiar with the fictional holiday the characters create, festivus, “A festival for the rest of us.” As a rejection of conventional winter holidays like Christmas/Haunnukah, a core part of festivus is the “airing of grievances.” While I have yet to attempt this in real-life, I’ve spent the past two years airing my...
19 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s the most wonderful time of the year… performance reviews. Depending on your outlook on your job/org, that “wonderful” could be sarcastic. However, if you’ve landed on your manager’s nice list, this can be the time to recap your achievements to maintain credibility and work toward the next rung on the corporate ladder. If you’re on a small team serving demanding stakeholders it’s possible...
26 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! This holiday season will be a little less bright thanks to my lack of personal GitHub commits. Like you, I began 2024 full of ideas and motivation that, let’s be honest, was depleted by the end of Q1 when I was cranking out enough code at work that would please even notorious code volume stickler Elon Musk. Despite my lacking output, I managed to hunker down to create useful “bespoke” (a.k.a....
about 1 month ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline. Hi past, present or future data professional! When I worked at Disney there was one line (aside from “Have a Magical Day”) that was borderline beaten into us: “We are all custodial employees.” The line meant, of course, to keep areas under your purview neat and presentable (“show ready” in Disney-speak). Using the same logic, I’d like to emphasize that while the various data roles (data analyst, data scientist, data engineer, etc.) have...
about 1 month ago • 2 min read
Extract. Transform. Read. A Newsletter From Pipeline Hi past, present or future data professional! Since today marks Thanksgiving in the US, I hope this reaches you before your eyes glaze over from the tryptophan-induced turkey coma we all inevitably slip into. While today is a day of gratitude, from a data engineering perspective, I’d like to focus, instead, on the under-the-radar tasks that can make a difference at this time of year—even if they don’t gain you any recognition at work. The...
about 2 months ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s never good when you wake up to this from a coworker: 💀 The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch. The thing is, for a first-time violation, I can understand why testing is an...
about 2 months ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! It’s been a busy fall; I currently have 14 tasks in various states of development. Right now my JIRA board looks like I just won bingo—twice. Unfortunately when you climb the tech ladder things only get busier which means you’re going to burn out unless you take steps toward proactivity. For me this means learning which tasks I don’t need to (and really shouldn’t) do manually. And before you...
2 months ago • 2 min read
Extract. Transform. Read. A newsletter from Pipeline Hi past, present or future data professional! Despite falling into the realm of engineering, data infrastructure construction is a bit like basic art. At times building a data pipeline is as simple as filling in one of those color-by-numbers books. Other times, the process of extracting and ingesting data can be as abstract and disconnected as paint flicked onto a canvas, Jackson Pollack style. No matter the complexity of your build, there...
2 months ago • 2 min read