Explainer · 6 min read
What is a data pipeline?
A data pipeline is a set of steps that runs on a schedule and moves numbers from the places they are created to the place they are read, without a person in the middle. That is the whole idea. Everything technical about it is in service of that one sentence.
Most businesses already have a pipeline. It is just made of people. Someone logs into the billing system on Monday, exports a report, pastes it into a spreadsheet, fixes the branch names by hand, adds last week’s figures, and emails the file to four people. That routine is a pipeline: collect, clean, store, serve. It works, until the person is on leave, or the export format changes, or two of the four recipients start keeping their own version of the sheet.
A built pipeline does the same four things, in the same order, at four in the morning, and tells someone when it fails.
The stage that decides whether it works
Collecting data is largely a solved problem. Cleaning it is where projects succeed or quietly fail. The same customer will exist in your systems as four different spellings. Dates will arrive in three formats. A branch that closed in 2023 will still be producing rows. Someone will have typed a quantity into a price field.
Cleaning means writing those corrections down once, as rules, instead of making them from memory every month. It also means deciding what happens to a row that cannot be fixed. Our answer is that it goes into a quarantine table with a reason attached, where someone can look at it. Rows should never be silently dropped: a total that is quietly missing 3% is worse than no total at all, because nobody knows to distrust it.
Why one warehouse matters more than any dashboard
The store stage puts everything in one place with one definition of each term. This sounds like housekeeping. It is actually the thing that changes meetings. When sales and finance arrive with different revenue figures, the argument is almost never about arithmetic. It is that one of them counts an order when it ships and the other when it is invoiced, and nobody ever wrote that down. A warehouse forces the definition to be written once, and then every report inherits it.
Serving is the last stage, and the least interesting one technically. Reports, dashboards and other systems read from the warehouse rather than from source systems or from someone’s laptop. Two departments reading the same table cannot produce two different totals.
How you know it is still working
A pipeline that runs unattended needs to be watched, which is why monitoring loops back to collection rather than sitting at the end. Two things are checked: did the job finish, and does the result look plausible. If Tuesday’s load brings in eleven rows when it normally brings four thousand, that is a failure even though nothing crashed. You should hear about it from us before you hear about it from a number that looks wrong.
What a pipeline is not
It is not artificial intelligence, and it does not produce insight on its own. It produces reliable, current, agreed numbers. Everything else — reporting, forecasting, automation — is built on top of that, and is unreliable without it. A forecast trained on inconsistent history is a confident guess.
You probably need one if a monthly figure takes more than a day to assemble, if two departments produce two versions of the same number, or if a specific person leaving would stop your reporting. You probably do not need one yet if all your data already lives in one system that reports well, and nobody is copying anything into a spreadsheet.
This is the kind of work our big data engineering line does.