Kantar Worldpanel Plus uses Azure Databricks to produce real-time shopper insights

“We have the end-to-end solution on Microsoft Azure delivering data. Now, it’s about innovation and scale – and we have a world of possibilities,” says André Gabriel Garrido, Head of Software Development at Kantar Worldpanel. 
 

Data and insights are at the heart of Kantar’s operations

Kantar is one of the world’s leading data, insights and consulting companies. Its validated insights, evidence and advice shape its clients’ decisions and are at the heart of their growth. Within the Kantar group, Kantar Worldpanel focuses on shopper behaviour using retail data and insights, helping its clients to understand consumer behaviour in today’s omnichannel world. It operates in 69 countries, working with more than 150,000 brands across 250 categories.  
 

To provide the insights its clients need, Kantar Worldpanel operates a number of specialist panels which are representative of different markets. In the UK, this includes a ShopandScan panel on which 30,000 shoppers scan their shop receipts with barcode scanners to capture data that is representative of the UK shopper.  
 

In 2018, Kantar decided to develop this area of the business with the creation of a new UK shopper panel: Worldpanel Plus. The vision was to enable the creation of client-ready reports on purchasing behaviour from multiple sources, including panellist data, scanned receipts and survey data. 
 

Daniel Kinneally, Business Operations Director at Kantar Worldpanel, explains, “Requirements are changing. Brands want more detailed information across a broader panel that is representative of their market.” 
 

Modernising the experience for shoppers 

Instead of shipping hardware to panellists, Kantar wanted to run the new panel with an easy-to-use solution that was more cost efficient and available to all. It devised a system based on a mobile app. Using the app, panellists would simply take a photo of their receipts using their existing smartphones. 
 

This would be more cost-efficient, lower the barrier to entry and enable the widening of the panel sample size. However, it posed a number of technical challenges. Daniel Kinneally describes the problem: “How do we take those unstructured images, with the limited amounts of information we can get from them, and turn that into an actionable dataset?”  
 

A new data platform was required that would support the larger sample size, store images of the photographed receipts, deploy optical character recognition (OCR) to discover and collate information from the receipts, query that data against attribution and dictionaries to add meaning, and store it in a state that makes it easy to query and interrogate.  
 

“We would need everything to get the data into a state so that our clients could work with it,” explains Daniel Kinneally. “And it should be an efficient technical process. And it should be fast.” 
 

Choosing Microsoft Azure 

Kantar was already partnering with Microsoft to move to the Azure cloud as part of a wider group initiative to modernise its technical infrastructure, move to a more predictable op-ex spend model, and achieve greater stability, scalability and performance. This made Microsoft Azure an obvious choice for the new Worldpanel Plus data platform. 

“It’s about the prioritisation of our investment dollars,” states Daniel Kinneally. “This is about cost avoidance. We want to grow in a particular area and, rather than replicating the old, we’ve taken the opportunity to build a new piece of tech from scratch.” 
 

What was less certain, were the technical choices around that. Initially, Kantar Worldpanel opted for a SQL-based data platform. However, it soon became clear this wouldn’t offer the speed or performance needed. 
 

André Gabriel Garrido explains, “Because our needs are complex, with unstructured data and all the machine learning on top of that, we needed a state-of-the-art platform and state-of-the-art technologies for big data. We needed to deliver and process that data as quickly as possible – almost real time. Databricks on Microsoft Azure was the only platform we found that could meet our requirements.”  

Developing the Shoppix mobile app

The front-end of the solution would be the new Shoppix app, which approved panellists could download and use on either iOS or Android. App development began in 2016, in parallel with work to build the new data processing platform.  
 

The app was developed in .NET and based on containerised Azure microservices running in Azure Kubernetes and Azure Docker containers, hosted in Microsoft Azure and leveraging Azure App Services. Live data has been added by panellists since 2017 and the number of demographically representative panellists now scans 100,000+ receipts per day.  
 

Microsoft Azure + Databricks = a winning combination

At the same time, the Kantar Worldpanel technology team led by André Gabriel Garrido began work on the data processing platform. Raw data ingested via Azure Event Hubs landed in Delta Lake sitting on Azure Data Lake where Azure Databricks was used to process it. There it could be curated, filtered, cleaned, and augmented and consolidated to produce business-level aggregated data. 
 

“The support we received from Microsoft and the Databricks teams helped us to get the platform live without any problems. They supported us to use the best technical standards that exist to create a state-of-the-art solution,” André Gabriel Garrido recalls.
 

“Our relationship is way more than client-provider – it is truly a partnership. They want us to have a state-of-the-art platform and we know that if we have any difficulty or question, we have access to a huge team that can help us.” 
 

Leveraging the new data processing platform 

The data processing platform went live in 2020 with data dating back to 2017. The Kantar Worldpanel Plus team is over the moon with the results.  

“Now we can deliver data to our commercial services and our clients in a matter of minutes,” reports André Gabriel Garrido. 
 

As an example, the legacy processes for scheduled shop linking data would take 2.5 hours to run. On the new platform, this has been reduced to 12 minutes. Similarly, the reprocessing of 40 million receipts of back data would take four weeks on the old system. Today, such a job would take 12 hours – a tiny fraction of the time.
 

“Moreover, if there is a need for a new kind of offer or a different kind of analysis on this data all we need to do is configure it,” adds André Gabriel Garrido. This stands in stark contrast to previous ways of working, when it would take a month to develop a query on the old platform followed by long hours of processing.  
 

“This is very new to Worldpanel,” continues André Gabriel Garrido. “It is new technologies and new ways of working. We have been an agile team from the start.”  

A huge boost to talent and retention 

Kantar Group has Unified Support with Microsoft, giving the Kantar Worldpanel team access to a wealth of learning resources and certifications as well as prioritised access to a broad team of specialists.   
 

Access to the accreditation and learning resources is an important benefit, helping the team to attract and retain talent. “It’s important to staff,” says Daniel Kinneally. “They want that accreditation and learning.” 
 

“Working with the latest, state-of-the-art technologies – like machine learning and all that sexy stuff – and state-of-the-art ways of working, and all the complexities of our data, is like the holy grail for good developers and good data engineers,” agrees André Gabriel Garrido. “Output is very simple but under the hood it is very complex. It makes us an exciting place to work.”  
 

The benefits of the Microsoft Azure platform

“Security is another box that it’s easy for us to tick,” continues André Gabriel Garrido. “Microsoft helps a lot, having solutions provided almost out of the box.” 
 

“We are a data-driven company working with members of the public, of course the security of PII data is a concern,” emphasises Daniel Kinneally. “Security is the utmost priority.” 
 

The Kantar Worldpanel team makes use of Azure Key Vault, Azure Monitor, and Microsoft Sentinel to protect and monitor its Azure environments.  
 

“It’s another key advantage of using Microsoft Azure,” states André Gabriel Garrido. “All these things are ready to be used. They are very simple to integrate and very simple to use. That’s key because, at the end of the day, that’s development time and resources we save for other, better things.” 
 

Integrations enable end-to-end services on Microsoft Azure

At the other end of the equation to the Shoppix app, the data must be presented to clients in an easy and convenient way. For the most part, this is delivered via a proprietary application called PRISM running on Microsoft Azure. 
 

However, the integrations offered with the rest of the Microsoft stack offer the Worldpanel team greater and much-needed flexibility and agility. The team has used Power BI to generate specialist reports when speed and agility are required – first for a COVID tracker but with the potential to meet other ad-hoc or tailored client requests.
 

“We are using Power BI for more premium, ad-hoc, and bespoke reports that are shared with our clients,” explains André Gabriel Garrido. “Because of the integration between Azure and Power BI, this is very easy to use and fast to implement so we can exploit opportunities and generate revenue quickly and more easily.” 
 

A future of innovation 

Although Kantar’s Cloud Centre of Excellence specifies and disseminates best practice at a group-wide level, the Kantar Worldpanel Plus team are keen to share their learnings and best practice throughout the group. 
 

“People recognise us as a model that can be replicated to other teams,” states André Gabriel Garrido. “Not only our ways of working, but also our platform, the technologies, and how to process this data. What we’re doing is very innovative in our sector. There are not many platforms that could deliver our data – daily data, fast data, the kind of schedules we want – given the complexity of the datasets we’re working with. There is little that is comparable in our space and among our competitors. We have 700TB of processed data that can be processed in minutes. Processing this and doing all the activities we do with such a large dataset is possible because of this combination of technologies with Microsoft Azure and Databricks.” 
 

This commitment to innovation means the future roadmap is fuzzy; it can change from month to month in line with business need. Whatever the future holds, however, Microsoft and Databricks will continue to be key to the team’s success.  
 

“Having end-to-end Microsoft is better for my team,” says André Gabriel Garrido. “It’s one set of training, one set of support. We can focus on one technology and not have our focus spread. End to end, we are delivering data. Now, it’s about innovation: more countries, more advanced machine learning models, more data sets. We’re working purely on innovation and our core platform is built on Microsoft technology.”