Services

Data automation consulting

The goal is not fewer hours. It is that the number is right on Monday morning without anyone having stayed late on Friday to make it so — and that you find out when it is not.

The real cost of manual reporting

The hours are the obvious cost and the least important one. What manual reporting actually costs you is latency and trust.

Latency, because a process that takes three days describes a week that has already ended. By the time the numbers are agreed, every decision they were meant to inform has either been made without them or postponed.

Trust, because a human doing the same join by hand forty times will do it slightly differently on some of them. Nobody can point at which, so the whole report inherits the doubt — and meetings start with an argument about the data rather than the result.

What is worth automating

Automating the wrong thing is worse than leaving it manual, because it puts a disputed process beyond easy reach.

Anything on a schedule that produces the same shape every timeautomate: Exports, syncs, reconciliations, refreshes. The work is identical each week and the only variable is whether someone remembered.
Anything with a clock attachedautomate: Routing, escalation, SLA breaches, renewal stages. Humans are bad at noticing that nothing has happened yet.
Validationautomate: Checks nobody enjoys running are the checks that stop being run. This is the highest-value automation and the one most often skipped.
Anything with judgement in the middlecareful: Automate the gathering and the writing-back; leave the decision with a person. Summarise and tag, then let the rep decide what happens next.
A process nobody has agreed on yetleave it: Automating a disputed process makes the dispute permanent and much harder to change. Settle the definitions first.
Something that runs twice a yearleave it: The automation will have rotted by the second run and nobody will remember how it worked. Write it down instead.

Five principles that decide whether it survives

01Idempotent by default: Running the same job twice must produce the same result as running it once. Upsert on a stable key rather than insert. Retries are inevitable, so make them boring.
02Fail loudly, never silently: The worst pipeline is one that stops delivering without telling anyone, because the dashboard keeps showing last week's number and nobody notices until a decision has been made on it. Alert when a source stops arriving, not only when a job errors.
03Validate at the boundary: Check row counts, null rates and value ranges as the data arrives, and reject the batch rather than let it through. A pipeline that only checks at the end has already contaminated the warehouse.
04Backfill and incremental are different jobs: A backfill that ignores rate limits gets throttled halfway and leaves the data in a state nobody can reason about. Write them separately, and make the backfill resumable.
05Version the logic, not just the data: When a number changes, the first question is always whether the data changed or the definition did. If the transformation lives in a UI with no history, you cannot answer that.

Choosing where the logic lives

Workflow tools — n8n, Make, Zapier — are right for event-driven work between applications: something happened in one system, do something in another. They are quick to build and easy for a non-engineer to read.

Scheduled code — Python on a runner — is right when the transformation is complex, needs testing, or has to be reviewable. The moment a workflow canvas has thirty nodes and a branch you cannot follow, it has become code with a worse editor.

The warehouse — BigQuery or equivalent — is where joins and business definitions belong, not scattered across either of the above. If “revenue” is calculated in three workflows, you have three definitions of revenue and no way to reconcile them.

The failure mode we see most often is business logic distributed across a dozen automations because each was the fastest way to answer one request. It works until someone asks why two dashboards disagree.

A worked example

A multi-channel training provider was running management reporting off manual exports from five platforms, with spreadsheet updates, data checks and repeated reconciliation. By the time the numbers were agreed, the week they described was over.

We rebuilt it as scheduled pipelines into BigQuery, a modelled reporting layer with KPI definitions agreed in writing first, automated validation rules, and alerting when a source stops delivering. The alerting mattered as much as the pipelines: a silent gap in a daily refresh is how a wrong number reaches a board pack.

Reporting preparation fell from roughly 30 hours a week to about 2 hours of review and validation — 93% less manual preparation, six sources consolidated, daily refresh instead of weekly. Numbers now arrive before the meeting instead of after it. Full case study.

Note what did not get automated: the 2 hours that remain are review and validation by a person. That is deliberate.

Start with the report that takes longest

It is usually the one crossing the most systems, which makes it both the most painful and the most instructive thing to fix first. See also automated reporting.

Discuss a project

More on core services

All services →

Or start with the revenue operations overview that the rest of these hang off.

Have a messy system?

That is usually where we can help.

Tell us what is not working, what is still manual, or what you cannot currently see clearly. If it is not something we should take on, we will tell you that too.

Discuss a project

hello@core-x.solutions

Start with whichever is true

Taking on new work.