Centrica creates data platform with Azure HDInsight and Power BI

Building a new data platform with Microsoft Azure

The data modernization journey commenced with the goal of migrating from the existing Hadoop platform to Platform as a Service (PaaS) powered by Azure HDInsight. This vision encompassed the eventual decommissioning of the current data lake and its associated user cluster environment. “We wanted to modernize the current data platform with a holistic and strategic organizational goal of moving towards a cloud centric enterprise," said Eddie Edwards, Head of Data Solutions at Centrica.

This transformation had several key objectives in mind: 

• Reduce maintenance and run cost 
• Optimize infrastructure cost and support with a secure, high-performing, highly-scalable platform with minimal manual operations
• Reduce revenue loss due to data which couldn’t be processed on the existing Hadoop platform due to environment and platform constraints
• Drive user simplicity and standardization
• Provide enhanced data analytics and machine learning capabilities, while separating compute and storage
• Remove data silos and drive use and democratization with increased self-service

As part of this strategic technology transformation and data modernization initiative, Centrica's data team, along with the solution partner Cognizant, were tasked to deliver the migration based on its prior success in similar deliveries within Centrica and proven capabilities.

Once Azure HDInsight was confirmed as a target data platform, Centrica gradually started the as-is migration with no (or minimal) code change to existing production jobs due to HDInsight OSS and different versions support, as well as minimal impact to business users, ensuring data availability and SLAs. The data platform is supported by tools such as Azure Data Factory, Azure Data Lake Gen2, Azure HDInsight, and Power BI.

Raw data from various sources is ingested, transformed, and curated, supporting multiple types of workloads such as Spark and Hive in Azure HDInsight. Processed data is then stored in Azure Data Lake Gen 2 and served using different tables in the catalog. At the data lake level, many governance principles and practices are implemented. This allows the team to track who is ingesting data and how it's being used, and to uphold robust data access and governance policies. The curated data is then made available to users in multiple forms, including interactive dashboards, reporting, data sharing with other consumers, and real-time insights.

"The goal of Centrica's data platform is to consolidate data, foster scalability, and drive cost-efficiency," said Darren Miles, Centrica's Chief Information Officer. "This platform not only promotes collaboration and streamlined data access for the development of data products but also significantly contributes to lowering operational costs, reducing the overall TCO, improving performance and scalability. As a result, data processing and analysis accelerated, transforming time-intensive tasks into immediately available assets."

Centralized data lake and reporting

With data now centralized and easily accessible, it serves as the backbone for a wide range of solutions, with a strong emphasis on Power BI reports. Eddie Edwards, Centrica's Head of Data Solutions, highlights the significance of this centralized data accessibility, stating, “the ease of using Power BI has greatly improved with our centralized data platform and curated datasets in place. It's now much simpler for everyone to leverage Power BI.”

Centrica also relies on self-service Power BI usage, empowering users across the organization to quickly create reports and gain valuable insights. To further promote the adoption of Power BI and governance, Centrica has established a Center of Excellence focused on best practices. Their goal is to ensure that users have the knowledge and tools they need to make the most of Power BI. Centrica's data teams actively collaborate with Microsoft to enhance their enterprise skills initiative, offering comprehensive training and certification programs.

"Centrica's successful transition to a cloud-based data platform with Azure HDInsight and Power BI is a prime example of how companies can leverage modern technology to drive cost-efficiency, scalability, and collaboration," said Miles. "Our data modernization strategy serves as a model for other organizations looking to optimize their data infrastructure."

Extending the data lake

With the new data platform, Centrica has been able to break down the data silos, reduce costs, and improve time to value by making data accessible on Azure. Now, not only different teams and users can get a better understanding of the data, but also could connect by using their tool of choice such as Azure Databricks or HDInsight. 

“Overall cost of ownership reduced significantly by more than 70%, reducing significant capital expenditure on hardware and infrastructure," said Eddie.

Moving forward, Centrica would like to expand the usage and accessibility of their data platform, as well as evolve data persona capabilities. As data standards continue to evolve, the team remains committed to aligning with these standards and ensure data quality. The data team is looking into Microsoft Fabric and Microsoft Purview as part of their platform to help gain more insights and control over data. The team also wants to offer the best persona experience possible to continually improve ease of use and help individuals get value from data quickly.

Find out more about Centrica on Twitter, Facebook, and LinkedIn.