Most retail AI partner evaluations look remarkably similar to each other. A request for proposal goes out, several vendors and consultancies respond with polished decks, each one shows a handful of impressive logos and a similar set of capability slides, and the procurement team ends up choosing largely on the strength of the presentation and the reference call rather than on any structural difference between the candidates. This is not because procurement teams are careless. It is because the questions that actually predict whether a partner will deliver are rarely the questions a standard RFP process asks.
The partners who perform well in a pitch and then underdeliver in execution almost always fail in the same handful of places: they have genuine enterprise AI experience but limited retail-specific depth, they have built impressive demos but have not actually operated a system through a peak trading season, or they propose an architecture that looks elegant in a slide but creates a dependency that becomes expensive to unwind two years later. None of these failure modes are visible in a typical RFP response. All of them are visible if you ask the right ten questions.
This checklist is built from the patterns we see across retail AI engagements, both the ones that work and the ones that get re-scoped or re-staffed midway through. Use it as a working document during your evaluation, not as a box-ticking exercise.
Why Generic Enterprise AI Experience Is Not Enough
The first filter that genuinely matters, and the one most evaluation processes skip entirely, is whether a partner has solved retail-specific problems before, not whether they have broad enterprise AI experience. These are not the same capability, and the gap between them shows up at the worst possible time.
A partner with strong general enterprise AI credentials but no retail-specific delivery history will, with good intentions, build a personalisation or forecasting system the way a generic enterprise system gets built: solid architecture, reasonable model choices, clean code. What that partner will not anticipate, because they have not lived through it before, are the retail-specific operational realities that determine whether the system actually performs: how dramatically demand volatility spikes around key trading events, how fragile real-time inventory accuracy is across a large store network, how a personalisation model needs to behave differently during a flash promotion than during steady-state browsing. These are not edge cases in retail. They are the normal operating conditions, and a partner encountering them for the first time on your production system is a very different proposition from a partner who has already made and corrected these mistakes on someone else’s engagement.
The 10 Questions That Actually Predict Delivery
1. Can they show specific retail outcomes, not just retail logos?
A logo slide showing well-known retailer names proves a relationship existed. It does not prove the partner delivered a measurable outcome. Ask specifically what was built, what the measured result was, and ask to speak to the client team that lived with the system after launch, not only the executive sponsor who approved the budget.
2. Have they operated a system through a peak trading event?
Building a model is one thing. Watching it perform correctly when traffic and transaction volume spike five to ten times above baseline during a peak event is a different test entirely, and it is the test that most cleanly separates partners with genuine production experience from partners who have only built in controlled pilot conditions. Ask directly whether they have run a system they built through a Black Friday, a major promotional event, or an equivalent demand spike, and ask what broke and how they responded.
3. What does their data integration approach actually look like?
Every partner will claim to integrate with your existing systems. The meaningful question is whether they build toward a unified, governed data architecture that other AI capabilities can also draw from, or whether they build a point integration specific to this one project that will need to be redone for the next initiative. The former compounds in value. The latter creates integration debt that accumulates with every new AI use case you add.
4. Who specifically will be staffed on the engagement, not just who pitched it?
The team in the sales pitch is frequently not the team that does the delivery work, particularly at larger consultancies where senior staff lead the pitch and a different delivery team is staffed afterward. Ask for named individuals who will be doing the actual engineering and data work, ask about their specific retail experience, and be cautious of any partner unwilling to commit to specific staffing before contract signature.
5. What is their model for ongoing operations after go-live?
A strong partner has a clear answer for what happens after launch: who monitors model performance, how retraining is triggered, what the response time is for an incident during a peak trading period. A weak answer here, vague language about ongoing support without specifics, is one of the most reliable predictors of a system that performs well at launch and degrades quietly over the following year.
6. How do they handle the gap between their reference architecture and your actual legacy systems?
Every partner has a reference architecture that looks clean in a sales deck. Almost no retailer’s actual technology stack looks like the reference architecture, because legacy systems, custom integrations, and historical technical debt are the norm rather than the exception. Ask specifically how they approach the discovery process for understanding your actual systems before committing to a timeline and cost, and be sceptical of a fixed-price proposal delivered before that discovery work has happened.
7. What happens if you want to bring the work in-house later?
A partner confident in the value they deliver should be comfortable building in a way that allows for a structured handover to an internal team later, including clear documentation, knowledge transfer, and an architecture that is not deliberately difficult to operate without them. A partner who is vague or resistant on this question is signalling a business model built around indefinite dependency rather than genuine value delivery, which is worth knowing before you sign rather than after.
8. How do they price the engagement, and what is excluded?
Fixed-fee, time and materials, and outcome-based pricing models each carry different incentive structures, and none is universally correct. The more important diagnostic is what is explicitly excluded from the quoted price: integration work beyond a defined scope, ongoing operations after a defined period, retraining cycles, additional use cases. A proposal with a low headline number and a long exclusions list is not actually the lower-cost option once the full scope is delivered.
9. Can they demonstrate genuine governance and compliance maturity?
Retail data, particularly customer and payment data, carries real regulatory and reputational risk. A partner should be able to speak specifically about how their architecture handles data lineage, quality controls, and access governance, not in abstract terms but with reference to how they have actually structured this on prior retail engagements. Vague reassurance here is a red flag, not a green light.
10. Do they sequence the work, or pitch everything at once?
Partners who genuinely understand retail AI delivery tend to recommend a sequenced approach, building the data foundation and an early use case before expanding scope, because they have seen what happens when retailers try to deploy multiple ambitious AI capabilities simultaneously on infrastructure that is not ready for any of them. A partner who proposes a large, ambitious, all-at-once programme without first assessing your data maturity is either inexperienced in retail delivery or more interested in the size of the initial contract than in the outcome.
The Reference Call Questions Most Teams Forget to Ask
A reference call is only as useful as the questions asked during it, and most reference calls default to a narrow set of questions that the partner’s reference is well prepared to answer positively. A more useful reference call goes further.
- Ask what went wrong during the engagement, not whether anything went wrong. Every real engagement has friction somewhere. A reference who cannot name a single challenge is either not being candid or was not close enough to the work to know.
- Ask whether the team that was staffed at the start of the engagement was the team still there at the end, and if not, how the transition was handled.
- Ask how the partner performed during the first real operational stress test, a peak event, an unexpected data quality issue, a production incident, rather than during the comfortable early weeks of the engagement.
- Ask whether the reference would re-engage the same partner for a second, different use case, which is a more honest signal than whether they were satisfied with the first engagement.
Weighing the Answers: What Matters Most for Your Situation
Not every criterion on this list carries equal weight for every retailer, and a useful evaluation process weights the ten questions against your specific situation rather than treating them as a flat checklist.
If you are early in your AI maturity journey and building your first meaningful capability, the data integration approach and the ongoing operations model matter more than almost anything else, because the foundation you establish now will determine how easily every subsequent capability gets built. If you already have a mature data foundation and are evaluating a partner for a specific, well-scoped use case, the retail-specific delivery experience and the peak trading event track record become the more decisive factors, because the foundational risk is already lower.
If your organisation has a strong internal engineering team and is looking for a partner specifically to accelerate delivery rather than to own the work indefinitely, the handover and knowledge transfer question deserves disproportionate weight, because the value of the engagement is measured by how well your internal team can operate and extend the system after the partner’s engagement ends.
Warning Signs Worth Taking Seriously
Beyond the ten questions, a small number of patterns during the evaluation process itself are worth treating as genuine warning signs rather than minor friction to work through.
- A proposal delivered before any meaningful discovery of your actual systems, particularly a fixed price and fixed timeline based on assumptions rather than your real technical environment.
- Reluctance to name specific delivery staff, or repeated substitution of generic case studies for direct answers about retail-specific experience.
- An architecture that is difficult to explain in plain language. If the partner’s own team struggles to describe how the system works without retreating into jargon, that is a signal about how maintainable the system will be once they are gone.
- A pricing structure where the attractive headline number depends on a long list of exclusions that, once added back, make the total cost comparable to or higher than a more transparent competitor.
- Defensiveness, rather than directness, when asked what has gone wrong on previous engagements. Every real partner has a story here. The ones without one have not done enough real work to have collected the lessons.
Making the Final Decision
By the time you have worked through these ten questions with each candidate partner, the differentiation between them is usually much clearer than it appeared during the initial pitch stage, where most proposals look similarly polished. The partner who answers the ongoing operations question with specifics rather than reassurance, who can name the team that will actually do the work, who has a credible story about a peak trading event they navigated, and who is transparent about handover and pricing exclusions, is consistently the partner who delivers what was promised rather than a system that needs significant rework six months after go-live.
Closing Thoughts
The questions in this checklist exist because we at TechBlocks have seen what happens on both sides of them. We have been the partner brought in after another vendor’s architecture turned out to be too rigid to extend. We have been on calls where a retailer asked us what went wrong on a previous engagement, and we have learned that the honest answer is always more useful to them than the polished one.
What that experience has taught us is that the retailers who get this decision right are not the ones who find a partner with no flaws. They are the ones who ask precise enough questions to understand exactly where a partner’s strengths and limits actually are, before the contract is signed rather than after.
Our AI-Native Retail Studio exists because retail technology decisions, data, AI, commerce platforms, fulfilment, store operations, are rarely solved well in isolation, and most retailers end up stitching together point solutions from several vendors who were never designed to work together. If your evaluation is leading you toward a partner who can take ownership of that whole picture rather than one piece of it, that is the conversation worth having with us next.
Get in touch with our team at TechBlocks today.
FAQs on Retail AI Technology
Three to four serious candidates is generally the right range. Fewer than that risks not having enough comparative signal to identify genuine differentiation. More than that tends to slow the process down without meaningfully improving decision quality, because the differentiating factors usually become clear well before a fifth or sixth evaluation. The quality of the evaluation process, asking the right questions of each candidate, matters considerably more than the number of candidates evaluated.
A small, well-scoped paid pilot can be valuable when the use case is genuinely novel or when the gap between candidates remains unclear after the standard evaluation process. It is less valuable, and sometimes misleading, when used as a substitute for proper discovery, because a pilot built quickly to win a contract often looks different from the production system that gets built afterward. If you do run a pilot, structure it around the same data and integration challenges the production system will face, not a simplified demo environment that avoids the hard parts.
For the model and intelligence layer, general AI expertise combined with genuine retail domain knowledge tends to outperform either one alone. For the data integration and operational layers, retail-specific experience matters disproportionately more, because the failure modes in those layers are driven by retail-specific conditions, peak trading volatility, multi-channel data complexity, that a partner without prior retail exposure is unlikely to have anticipated, regardless of how strong their general AI capability is.
For a meaningful retail AI engagement, four to eight weeks from initial outreach to signed contract is typical for a careful evaluation, including discovery calls, proposal review, reference checks, and any pilot work. Evaluations that compress significantly faster than this usually skip steps, most often the discovery work needed to produce an accurate scope and the reference calls needed to validate delivery claims. Evaluations that extend well beyond this often reflect unclear internal decision-making criteria rather than genuine diligence.
It depends heavily on which criteria are weak. A partner who is strong on retail delivery experience and data architecture but newer to your specific commerce platform is a manageable gap, closeable through proper discovery and a slightly extended timeline. A partner who is strong on presentation and general AI capability but cannot speak credibly to peak trading experience or ongoing operations is a structural gap that tends to surface as a real problem after go-live, not before. Weigh the gaps against which layer of the decision they affect, the data foundation and operations gaps are far more consequential than gaps in presentation polish or breadth of case studies.
