Flux
Toutes les sources

SSP Data Engineering Blog

28 articles Flux RSS
Programmation IA
Data Engineering Is the Cache Invalidation Problem Ft. Josh Wills
Programmation IA

Data Engineering Is the Cache Invalidation Problem Ft. Josh Wills

In this episode, we dive into how much models actually matter: more like a preference, like whether you use an iPhone or Android, than their capabilities. We talk about bare-metal disks when running Kubernetes, then move to agents and how data engineering becomes like being an emergency pilot in a plane, and how we automate away the boring work with no more database migrations or unit tests. We go into spiritual territory, where he feels in his gut whether a pipeline will work. Then we finish…

SSP Data Engineering Blog
Why We Need Yet Another (OSS) Semantic Layer
Programmation IA

Why We Need Yet Another (OSS) Semantic Layer

Most semantic layers come with a great declarative modeling language to define metrics and measures well1, but very soon after, you start building wrappers around the semantic layer that enable agentic work and context engineering, that work well with harnesses and interfaces, and that allow flexible queries at runtime, without the need to rebuild or recalculate the measures in YAML. The other problem is that most data warehouse or analytics solutions basically lock you in with their…

SSP Data Engineering Blog
Omarchy Quattro: One Year In — Part 3
Programmation IA

Omarchy Quattro: One Year In — Part 3

This is my Part 3 about Omarchy, after using it for more than a year full time, as my main daily driver first on Lenovo ThinkBook 14 AMD, and now TUXEDO InfinityBook Pro 14 Gen10, for my business and personal use, replacing my MacBook Pro M1 after using macOS for 15 years. See more in Part 1 about my transition, what tools I needed, and in Part 2, the good, the bad, and the fixable. I’m also the original author of the Windows VM work (PR #1333, merged via #2462), making Excel and Word available…

SSP Data Engineering Blog
Obsidian Vault to Enterprise Company Brain: Where Agents Write, Branch and Merge
Programmation IA

Obsidian Vault to Enterprise Company Brain: Where Agents Write, Branch and Merge

I’ve been using a personal second brain for many years. I started with OneNote and pivoted to Obsidian due to the linking feature. Since then, I have built a full net of knowledge for my personal life, but not only that: I also manage my solo business with Obsidian, such as my CRM or everything knowledge-related to my company. But what if you want the same approach for a bigger company, enterprise scale with lots of information internally, but also available externally? How do you manage…

SSP Data Engineering Blog
The Grammar of Data: From Definition to Execution
Programmation IA

The Grammar of Data: From Definition to Execution

In Part 1, we discovered the grammar for data: a way to define a data project with its complex requirements and how we define it declaratively as a grammar in one sentence with nouns (sources), transformations (verbs), templates, and modifiers, essentially being able to define it once and run it anywhere with different execution engines. This Part 2 will demonstrate how this looks in a data engineering digest project where we process data from RSS feeds, a live Bluesky firehose, and GitHub…

SSP Data Engineering Blog
If You Always Enjoy It, You're Not Pushing It Hard Enough—Benn Stancil (Show Your Workflow #1)
Programmation IA

If You Always Enjoy It, You're Not Pushing It Hard Enough—Benn Stancil (Show Your Workflow #1)

I had the pleasure to discuss and analyze the workflow of none other than Benn Stancil, legendary writer and storyteller in the data space and beyond, who started writing back in 2013 (with a seven-or-eight-year startup break in between). This is the first interview in the series «Show your Workflow», and I couldn’t be happier to start with Benn. As this series is all about how writers write, how to come up with stories, how to take notes (or not), and each one’s workflow, I asked Benn about…

SSP Data Engineering Blog
Writing really is an Emotional Rollercoaster
Programmation IA

Writing really is an Emotional Rollercoaster

[[Writing]] has become my passion over time; I have built a whole company based on technical writing services (even though most say that’s impossible). After doing it for 11 years publicly (much longer privately), I still enjoy it every day. If I could have one dream, I’d probably imagine the book-author lifestyle, where I go to a faraway place or island and just write in my journal, articles from my vast notes system (~3 million words, or 2-3 books.

SSP Data Engineering Blog
Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin
Programmation IA

Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin

It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still need to coordinate and understand what the agents produced, potentially test it, which makes it even…

SSP Data Engineering Blog
The Act and the Outcome of Creation
Programmation IA

The Act and the Outcome of Creation

Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious. It gives us joy, and to every artist, it is the ultimate (flow) state of happiness. The Act of Creation The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom. Creating should be done like:

SSP Data Engineering Blog
The Grammar of Data: Define Once, Run Anywhere with Cross-Engine Expressions
Programmation IA

The Grammar of Data: Define Once, Run Anywhere with Cross-Engine Expressions

Grammars for languages or any other field are a beautiful thing. They compress complex systems into a language with a couple of rules. For the spoken language example, we know when to capitalize a letter or how to start a sentence. There are clear rules. Grammars also help us remember, as we do not need to recall every little rule, but apply them in a structured way. For text editing, we have Vim motions that help us navigate a text document with 1000s of shortcuts, but because there is a…

SSP Data Engineering Blog
Where AI Agents Belong in Data Engineering: The Correctness Layer
Programmation IA

Where AI Agents Belong in Data Engineering: The Correctness Layer

With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That’s also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you’re running underneath. This is especially true during maintenance or extending the project, where verification is the real job.

SSP Data Engineering Blog
The Process of Smart Note-Taking
Programmation IA

The Process of Smart Note-Taking

I’ve taken notes over my whole life. As a Swiss, I’m very particular about organization and structured in filing my knowledge. I’ve tried and taken notes in many different ways, from forgetting everything, to noting down every detail, to only taking listings of the most important things. Today, I can say, there’s no right way, but I can see it’s working for me, as my most interesting ideas grow inside my note taking system, and they power what I share online, and earning a living from it.

SSP Data Engineering Blog
Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations
Programmation IA

Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations

Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines. When orchestration gets at the heart of triggering and observing most of the schedules or assets, there is even more importance involved. In Part 1 I argued that the open data platform makes orchestration data and its flow simpler, providing integration into data assets and transformation, ETL, BI and even Kubernetes all…

SSP Data Engineering Blog
Vibe Coding Is Dangerous, Agentic Engineering Isn't—Wes McKinney
Programmation IA

Vibe Coding Is Dangerous, Agentic Engineering Isn't—Wes McKinney

This series interviews real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Wes McKinney. Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He also created Ibis to address these issues with a different approach to Python dataframe libraries, by decoupling…

SSP Data Engineering Blog