Eighteen months stalled, two vendors, three rebuilds - in production in 2.5 months.
Situation
The previous attempts weren't incompetent - they were conforming to an architectural constraint they had no standing to challenge. The solution had been built inside a software-development framework, containerised with model-serving nested inside, and could never run beyond a laptop-scale proof of concept. The problem wasn't the vendor. It was a combination of architecture, decision rights and lost stakeholder confidence.Assessment of situation and what I did
What the assessment found I gathered input from every involved party and ran a code-level and architecture-level investigation of the latest iteration. Three problems, stacked:
- A decision-rights problem — the architecture had been fixed by a function that was not a delivery organisation
- A technical problem — the toolset served software delivery, where the workload demanded a data-development framework; no amount of vendor effort inside that constraint could produce a production-grade system.
- A confidence problem — after repeated failed deliveries, stakeholders no longer believed the initiative could land at all.
What I did
- Described the real state of the solution to senior management in plain, non-technical terms — separating genuine showstoppers from ordinary technical debt.
- Was explicit that the existing build would never sustain production load or any growth in scope, and managed the understandable reluctance to change course a fourth time.
- Stopped the current approach & proposed, architected and led a fourth rebuild on a battle-tested data architecture sized for the actual workload and its future scope.
Outcome
A functioning data product in governed production in 2.5 months, delivered with two half-time people — myself and a data scientist. The product was handed over to the local team, which took full ownership of further architectural and development decisions.
What I would do differently
I initially underestimated how badly repeated failure had damaged executive trust. The engineering team already understood the architectural problem; the real constraint was confidence at senior-management level. I now test stakeholder trust explicitly in the first week of any recovery.