Flux
Toutes les sources

SSP Data Engineering Blog

28 articles Flux RSS
Programmation IA
Beyond the Semantic Layer: Building a Context Layer for the Agentic Era
Programmation IA

Beyond the Semantic Layer: Building a Context Layer for the Agentic Era

Writing SQL was never the hard part. Making it accurate and trustworthy against your warehouse always was. Point an AI agent like Claude or Codex at your data stack and ask a real analytics question, and the answer is usually mediocre: the agent can scrape some context from your git repos or whatever metadata it can find, but it doesn’t know your joins, your metric definitions, or the business rules that give a number its actual meaning.

SSP Data Engineering Blog
Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft. Chris Riccomini
Programmation IA

Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft. Chris Riccomini

This series interviews (see #1 with Mark Freeman) real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Chris Riccomini. Chris has seen the data stack evolve over the years. He thinks AI will soon handle the majority of data engineering work, provided with the right tooling and access to CLIs and APIs. He also thinks LLMs might not speak SQL, but a format…

SSP Data Engineering Blog
The Dagster Almanack: From Complexity to Composability
Programmation IA

The Dagster Almanack: From Complexity to Composability

I have read the “Poor Charlie’s Almanack” by Charlie Munger and thought about what it would take to write one for Dagster. A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life. My goal is to offer a collection of wisdom, insights, and principles gathered over the years. Giving you an outside view from someone who has used Dagster since back in 2019, used it at enterprise scale…

SSP Data Engineering Blog
Internal vs. External Storage? What's the Limit of External Tables
Programmation IA

Internal vs. External Storage? What's the Limit of External Tables

When I started my career as a data warehouse engineer and business intelligence engineer in 2003, external tables with materialized views were the standard. We used external tables to integrate CSV files and other data not already in Oracle databases. Oracle External Tables have existed since 2001, and that’s where I first used them. If the Lindy Effect continues to hold, we’ll use external tables even longer. But why have they survived for so long?

SSP Data Engineering Blog
AI Reveals Why BI Still Matters
Programmation IA

AI Reveals Why BI Still Matters

Ask a BI engineer what they actually spend their time on: it’s not building dashboards. More often: fixing the join that broke in the overnight pipeline, untangling the metric definition that means three different things to three different teams, or getting last week’s numbers into an Excel by Monday morning. The dashboard was always the easy part. This article looks at how BI evolved, how dashboards are actually used today, and what survives when AI enters the picture — starting with the…

SSP Data Engineering Blog
Specs Over Vibes: Consistent AI Results ft. Mark Freeman
Programmation IA

Specs Over Vibes: Consistent AI Results ft. Mark Freeman

There’s so much going on in the AI space, and how to work with AI agents is changing every day. Everyone is overwhelmed and almost numb from so many possibilities, yet you need to find a way to work with AI, not to get left behind, right? You might use AI agents all day long, parallelizing them with AI orchestrators like Agent Teams, Gastown, tmux, git worktree, and AI-based IDEs, but in the end, you just coordinated an AI. You still have to learn what it created, understand it, check for…

SSP Data Engineering Blog
Building an Agent-Friendly, Local-First Analytics Stack with MotherDuck and Rill
Programmation IA

Building an Agent-Friendly, Local-First Analytics Stack with MotherDuck and Rill

Imagine going from a 100-million-row dataset to an interactive analytics app with just a few prompts. What used to take hours or days can now be done in minutes by combining local-first databases and BI tools with an agentic coding workflow. When Rill bet on YAML dashboards and CLI-first workflows in 2022, they weren’t thinking about AI agents. Neither was MotherDuck when they built serverless DuckDB around the thesis that most data fits on a laptop. But it turns out, what is developer-friendly…

SSP Data Engineering Blog
Why I Still Blog — and Why the Future of Blogging Is Connected
Programmation IA

Why I Still Blog — and Why the Future of Blogging Is Connected

I’ve been online twenty years, and blogging for ten of them. This is the story and lessons learned of blogging online for a decade. It goes beyond blogging topics and includes note-taking (workflow), how to write well as well as the medium in which writing works best, and also the format in which writing works long-term such as writing in open formats and methods such as vim motions to navigate and edit like a surgeon.

SSP Data Engineering Blog
Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data
Programmation IA

Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data

Continuing from Part 1, where we learned what git for data is, how the architecture and use cases work, how you can achieve git-like functionality with different approaches, and how the key is to avoid moving data as much as possible to keep state that can be referenced and rolled back to, but at the same time saving cost by not duplicating all data every time you create a new branch.

SSP Data Engineering Blog
Building an Obsidian RAG with DuckDB and MotherDuck
Programmation IA

Building an Obsidian RAG with DuckDB and MotherDuck

I always wanted a personal knowledge assistant based on my notes. One that uses Obsidian’s backlinks and connections to surface ideas I’ve forgotten or never thought to link together. So I built one. A RAG system that runs locally with DuckDB as a vector database, then syncs to MotherDuck for a serverless web app running entirely in the browser via WASM. Think of it like J.A.R.V.I.S1 for your markdown files: search about a topic, and it shows connected notes up to two hops away, semantically…

SSP Data Engineering Blog
Arch Linux (Omarchy) — 8 Months Later: The Good, the Bad, and the Fixable
Programmation IA

Arch Linux (Omarchy) — 8 Months Later: The Good, the Bad, and the Fixable

This is a follow-up to my part 1 of Switching macOS to Arch Linux with Omarchy, where I documented my first months with Arch Linux and [[Omarchy]], after switching from 15 years of using macOS and Windows on and off at work since 2003. Back then, I had a checklist of basics I needed before I could commit to Linux as a daily driver: Obsidian, a Raycast-like launcher for fuzzy finding files and folders, screenshots (Snagit), daylight adjustment (f.lux), calendar events in the top bar. Those were…

SSP Data Engineering Blog
Why Coinbase and Pinterest Chose StarRocks: Lakehouse-Native Design and Fast Joins at Terabyte Scale
Programmation IA

Why Coinbase and Pinterest Chose StarRocks: Lakehouse-Native Design and Fast Joins at Terabyte Scale

Why is StarRocks gaining popularity among data engineers who need fast analytics on large-scale data? To find out, I did a deep dive on the companies actually using StarRocks in production, interviewing engineers and studying technical case studies from Coinbase, Pinterest, Fresha, Grab, TRM Labs, and Shopee. They all share a similar pattern: customer-facing analytics on Snowflake got too slow, and they needed sub-second query responses without heavy pre-denormalization in Flink or Spark.

SSP Data Engineering Blog
A Diary of a Data Engineer
Programmation IA

A Diary of a Data Engineer

You ingest data. You model it. You transform it. You serve it. Someone asks for a change. Everything breaks. You rebuild. This is the loop. It was the loop in 2005 with SSIS and star schemas. It’s the loop in 2025 with dbt and Iceberg, or 2026 with prompting AI agents. The tools change. The loop doesn’t. The Invisible Plumbers When I started my career in 2003, there was no “data engineering”. There was no big data, no data science. We called it Business Intelligence. Data Warehouse Developer.…

SSP Data Engineering Blog