A glimpse back into 2024

Well, I never thought I’d share my own look back, especially on the social media, but here it feels a bit different. Maybe because this place is rather a place just for me. I don’t feel like I’m spawning everyone’s feed with my own irrelevant thoughts. At the same time, I won’t clutter this place up with a personal, and to the casual reader, boring personal narrative. It’s a technical blog after all, right?

For me (yet definitely not only), the year 2024 was… technologically… hyped! There was so much going on in 2024. Learning new stuff, mastering already known tools, brushing up communication skills in unpleasant situations, and much more.

Originally, I wanted to describe all the different use cases I went through that year. But then I realized, it’s not my CV! I was lucky enough to consult for, participate in, and lead number of various projects, specifically focused on the data engineering, DevOps, cloud engineering and architectural fields.

At the beginning of the year, I got my hands dirty with some automated deployments of Azure resources, and the AKS clusters especially. Then, I ended up designing and implementing a complex data processing part to migrate on-prem system to GCP. And when the year was coming to its end, I found myself in a team of scientists analyzing earth data to reveal how climate change affects us and our technologies in the future.

The tech-stack was mostly made of Azure and GCP cloud platforms, CI/CD tools like GitHub Actions and Azure Devops, kubernetes clusters, Python, PySpark, Dask. Obviously, most of the tools are somehow related to the distributed computing. And that’s what it was a lot about as well. Really trying to stretch and scale those cloud resources.

One of my new amazing colleagues (RS) said:

Well, if I have a job, that runs on a single CPU for 100 minutes, and which can be effectively parallelized, then I want to spin up machines with 100 CPUs and have the job done in 1 minute, and pay the same CPU-time related costs!

Which, imho, is exactly how we want to leverage the power of the cloud resources. Why’d we pay the same expenses for the longer variant of the same job, if we can pay the same but have the task done in a fraction of the original time?

I’m aware that the few last sentences cover a lot of challenges. It’d take a single post or more to elaborate on each of the implicit topics hidden there. Maybe, I’ll find some of them so interesting (or exhausting) to make a post about it.

Among all the tools and topics, I chose some I’d love to learn in my free time this year. Hopefully, I’m lucky enough to spend some time on some of them while working on the real projects.

Here are the chosen ones, with the very rigorous reasoning why these:

• Rust , because it feels like a teenager learning how to skateboard (tried that once, haven’t really succeeded).
• Dask , because it’s so exciting, like cheating on the loyal Spark (never tried that).
• Algorithms , because it’s like eating a habanero, painful at the beginning but amazing in the end (tried that many times, works more than once for a single instance).

Even though, I didn’t really succeed to post more the last year, it was mainly because I spent all the time actually doing the things, which was cool as well! At least most of the time .

Anyway, let’s see what we got in the new year, 2025!