Flux
Toutes les sources

SSP Data Engineering Blog

21 articles Flux RSS
Programmation IA
Writing really is an Emotional Rollercoaster
Programmation IA

Writing really is an Emotional Rollercoaster

[[Writing]] has become my passion over time; I have built a whole company based on technical writing services (even though most say that’s impossible). After doing it for 11 years publicly (much longer privately), I still enjoy it every day. If I could have one dream, I’d probably imagine the book-author lifestyle, where I go to a faraway place or island and just write in my journal, articles from my vast notes system (~3 million words, or 2-3 books.

SSP Data Engineering Blog
Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin
Programmation IA

Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin

It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still need to coordinate and understand what the agents produced, potentially test it, which makes it even…

SSP Data Engineering Blog
The Act and the Outcome of Creation
Programmation IA

The Act and the Outcome of Creation

Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious. It gives us joy, and to every artist, it is the ultimate (flow) state of happiness. The Act of Creation The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom. Creating should be done like:

SSP Data Engineering Blog
The Grammar of Data: Define Once, Run Anywhere with Cross-Engine Expressions
Programmation IA

The Grammar of Data: Define Once, Run Anywhere with Cross-Engine Expressions

Grammars for languages or any other field are a beautiful thing. They compress complex systems into a language with a couple of rules. For the spoken language example, we know when to capitalize a letter or how to start a sentence. There are clear rules. Grammars also help us remember, as we do not need to recall every little rule, but apply them in a structured way. For text editing, we have Vim motions that help us navigate a text document with 1000s of shortcuts, but because there is a…

SSP Data Engineering Blog
Where AI Agents Belong in Data Engineering: The Correctness Layer
Programmation IA

Where AI Agents Belong in Data Engineering: The Correctness Layer

With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That’s also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you’re running underneath. This is especially true during maintenance or extending the project, where verification is the real job.

SSP Data Engineering Blog
The Process of Smart Note-Taking
Programmation IA

The Process of Smart Note-Taking

I’ve taken notes over my whole life. As a Swiss, I’m very particular about organization and structured in filing my knowledge. I’ve tried and taken notes in many different ways, from forgetting everything, to noting down every detail, to only taking listings of the most important things. Today, I can say, there’s no right way, but I can see it’s working for me, as my most interesting ideas grow inside my note taking system, and they power what I share online, and earning a living from it.

SSP Data Engineering Blog
Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations
Programmation IA

Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations

Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines. When orchestration gets at the heart of triggering and observing most of the schedules or assets, there is even more importance involved. In Part 1 I argued that the open data platform makes orchestration data and its flow simpler, providing integration into data assets and transformation, ETL, BI and even Kubernetes all…

SSP Data Engineering Blog
Vibe Coding Is Dangerous, Agentic Engineering Isn't—Wes McKinney
Programmation IA

Vibe Coding Is Dangerous, Agentic Engineering Isn't—Wes McKinney

This series interviews real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Wes McKinney. Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He also created Ibis to address these issues with a different approach to Python dataframe libraries, by decoupling…

SSP Data Engineering Blog
Beyond the Semantic Layer: Building a Context Layer for the Agentic Era
Programmation IA

Beyond the Semantic Layer: Building a Context Layer for the Agentic Era

Writing SQL was never the hard part. Making it accurate and trustworthy against your warehouse always was. Point an AI agent like Claude or Codex at your data stack and ask a real analytics question, and the answer is usually mediocre: the agent can scrape some context from your git repos or whatever metadata it can find, but it doesn’t know your joins, your metric definitions, or the business rules that give a number its actual meaning.

SSP Data Engineering Blog
Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft. Chris Riccomini
Programmation IA

Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft. Chris Riccomini

This series interviews (see #1 with Mark Freeman) real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Chris Riccomini. Chris has seen the data stack evolve over the years. He thinks AI will soon handle the majority of data engineering work, provided with the right tooling and access to CLIs and APIs. He also thinks LLMs might not speak SQL, but a format…

SSP Data Engineering Blog
The Dagster Almanack: From Complexity to Composability
Programmation IA

The Dagster Almanack: From Complexity to Composability

I have read the “Poor Charlie’s Almanack” by Charlie Munger and thought about what it would take to write one for Dagster. A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life. My goal is to offer a collection of wisdom, insights, and principles gathered over the years. Giving you an outside view from someone who has used Dagster since back in 2019, used it at enterprise scale…

SSP Data Engineering Blog
Internal vs. External Storage? What's the Limit of External Tables
Programmation IA

Internal vs. External Storage? What's the Limit of External Tables

When I started my career as a data warehouse engineer and business intelligence engineer in 2003, external tables with materialized views were the standard. We used external tables to integrate CSV files and other data not already in Oracle databases. Oracle External Tables have existed since 2001, and that’s where I first used them. If the Lindy Effect continues to hold, we’ll use external tables even longer. But why have they survived for so long?

SSP Data Engineering Blog
AI Reveals Why BI Still Matters
Programmation IA

AI Reveals Why BI Still Matters

Ask a BI engineer what they actually spend their time on: it’s not building dashboards. More often: fixing the join that broke in the overnight pipeline, untangling the metric definition that means three different things to three different teams, or getting last week’s numbers into an Excel by Monday morning. The dashboard was always the easy part. This article looks at how BI evolved, how dashboards are actually used today, and what survives when AI enters the picture — starting with the…

SSP Data Engineering Blog
Specs Over Vibes: Consistent AI Results ft. Mark Freeman
Programmation IA

Specs Over Vibes: Consistent AI Results ft. Mark Freeman

There’s so much going on in the AI space, and how to work with AI agents is changing every day. Everyone is overwhelmed and almost numb from so many possibilities, yet you need to find a way to work with AI, not to get left behind, right? You might use AI agents all day long, parallelizing them with AI orchestrators like Agent Teams, Gastown, tmux, git worktree, and AI-based IDEs, but in the end, you just coordinated an AI. You still have to learn what it created, understand it, check for…

SSP Data Engineering Blog
Building an Agent-Friendly, Local-First Analytics Stack with MotherDuck and Rill
Programmation IA

Building an Agent-Friendly, Local-First Analytics Stack with MotherDuck and Rill

Imagine going from a 100-million-row dataset to an interactive analytics app with just a few prompts. What used to take hours or days can now be done in minutes by combining local-first databases and BI tools with an agentic coding workflow. When Rill bet on YAML dashboards and CLI-first workflows in 2022, they weren’t thinking about AI agents. Neither was MotherDuck when they built serverless DuckDB around the thesis that most data fits on a laptop. But it turns out, what is developer-friendly…

SSP Data Engineering Blog