The recommendations that lift basket size, a Databricks engine for an online retailer

The recommendations an online retailer can’t tune

An online retailer already sits on the data that drives good recommendations. Every product a shopper views, every cart they build and abandon, every order they place. The store collects it on every visit, and in most cases it goes straight into Shopify and stops there, doing nothing for the next person who lands on the same product page.

The usual fix is a recommendation app from the store’s plug-in marketplace. It installs in an afternoon and puts a “you might also like” strip on the page, which feels like progress. Then the limits show. The store cannot see why it suggests what it suggests, cannot push higher-margin lines or clear ageing stock, and cannot stop it doing the one thing a lazy recommender always does, which is recommend the best-sellers people were going to buy anyway. The shopper data feeding it also leaves the store’s control, which matters under the Privacy Act 1988 and the rising expectation that personalisation is transparent.

The deeper problem is ownership. A black-box app rents the retailer a result without the model, the features or the ability to tune any of it. For a store whose catalogue, margins and customers are its own, that is the wrong trade. The signal is theirs. The model that learns from it should be theirs too.

Why Databricks, and owning your own model

The aim is a recommendation and segmentation engine the retailer owns and tunes, trained on its own browse and purchase data rather than a generic plug-in. We headline these builds on Databricks for three practical reasons that a one-click app cannot match.

First, the lakehouse holds the raw signal and the engineered features in one place, so the same browse and purchase history feeds recommendations today and forecasting or segmentation tomorrow. Second, feature engineering and model training sit where the data lives, so the store builds features that matter to it, such as margin, brand affinity and category mix, rather than whatever a vendor decided to expose. Third, model serving runs from the same platform, so a trained model returns suggestions to the storefront under the store’s own control.

The supporting stack feeds that core. Shopify is the source of truth for the catalogue, the orders and the on-site behaviour. We model that data through Snowflake into a clean layer of customers, products and events, then land it in the lakehouse for feature engineering and training on Databricks. Recommendations and segments flow back to the storefront, so the shopper sees personalised merchandising and the retention team sees who sits in which segment.

We separate the engine from the storefront on purpose. The store keeps shipping on Shopify while the model improves behind it, and the same modelled data can drive email, merchandising and planning, not just one strip on a product page.

Building it, and where it got hard

The model maths is rarely the hard part in retail recommendations. The trap we see is a recommender that scores well in testing and changes nothing at the till. Two related problems cause it, and the first build hit both.

The first was cold-start and popularity bias together. A new product has no purchase history, so a model trained only on past orders never learns to recommend it, and a brand-new visitor has no behaviour to personalise from. Left alone, the model retreats to the safe bet and pushes the same best-sellers to everyone. That looks busy and adds almost nothing, because those are the lines people would have found anyway, while the long tail that could lift basket size never gets surfaced. The fix was content-based features so a new product is matched on category, brand, price band and description rather than history, segment-level popularity as a sensible fallback for a new shopper, and deliberate exploration so fresh and long-tail items stay in rotation long enough to earn their own data.

The second was measurement, and it was the more uncomfortable one. Early offline metrics looked strong. Click-through on the recommendation strip rose. Revenue did not move. The model had learnt to predict what shoppers already intended, not to influence what they bought. We stopped trusting offline click metrics and moved to a holdout. A slice of traffic saw the personalised recommendations and a comparable slice did not, and we measured the difference in basket size, conversion on recommended items and revenue per session. That changed how we tuned the model, because now it had to beat a control rather than a benchmark. Segmentation helped here too. Grouping shoppers by behaviour meant recommendations could fit the segment rather than treat every visitor the same, which is where the measurable lift came from.

Throughout, shopper data stayed governed. Personal data is handled under the Privacy Act 1988, recommendations are explainable because the features are the store’s own, and the store can tell shoppers plainly how personalisation works rather than pointing at a vendor’s black box.

What changed

In a representative build the personalised “you might also like” placements lifted average basket size by a high single-digit percentage, measured against a held-out control rather than against offline clicks. Adding exploration and content-based features pulled products from outside the top sellers into view, so a wider slice of the catalogue earned attention and sales instead of the same dozen lines. Segmenting shoppers by behaviour let the store tune recommendations and merchandising per group, rather than serving every visitor the same homepage.

These figures are illustrative. They describe the pattern we see rather than a published result for a named retailer. The shape is the point. The store’s own browse and purchase data, which was sitting idle in Shopify, starts working for the next shopper, and the retailer owns the model that makes it happen, sees why it recommends what it does, and tunes it for margin and the long tail rather than renting a result it cannot see into.

Where this fits

A recommendation and segmentation engine is one application of our Artificial Intelligence service, built on Databricks, for retail and ecommerce. It is a contained, high-return first build, because the data already exists in your store and the value comes from modelling it properly and measuring real uplift. It is a sensible step up from a generic recommendation app, and the same data layer extends to forecasting, email and merchandising later. If your store data is sitting idle and a plug-in has stopped paying off, the place to start is to map your Shopify browse and purchase data and decide the recommendations worth owning.