San Diego Association of Governments improves regional planning outlook with advanced data processing and analytics using Microsoft Azure

A trusted source of quality information for California’s San Diego County for more than 50 years, the San Diego Association of Governments (SANDAG) is a Metropolitan Planning Organization and Council of Local Governments. To accurately plan for the area’s transportation needs, SANDAG uses large data sets, complex computer modeling, and simulation to project population, employment, and housing growth. SANDAG processes millions of data points from numerous public and private sources to inform its planning, including census, economic, and traffic information.

SANDAG’s existing on-premises IT infrastructure often used lengthy manual processes to feed and analyze the data it used in long-range growth and transportation forecasts for the San Diego area, a sprawling metropolis of more than 3 million people. Employees often spent days or weeks processing new data sets, which created a backlog. Extensive socioeconomic and transportation modeling using this data shapes SANDAG’s long-term planning, and it realized that its systems could no longer keep pace. It began to consider a cloud-based data analytics solution to reduce manual processes, ingest data more quickly, and harness advanced analytic capabilities to inform forecasting.

Migrating to Azure in less than 90 days, increasing processing capabilities by 10 times

The team at SANDAG had an existing relationship with Microsoft, and it reached out to the company to discuss a proof of concept for migrating its data estate to Microsoft Azure. From the beginning, the agency was impressed with the support it received from Microsoft. “Microsoft was very responsive and willing to work with us every day,” says Daniel Flyte, Cloud Solutions Architect at SANDAG. “The initial technical support we received from Microsoft was a big reason we decided to continue working with them.” The SANDAG team decided to go beyond the proof of concept and received funding for the Microsoft Data Platform in 30 Days (DPi30) Solution Assessment Program, which helps customers deploy and use Azure in 30 days.

The organization’s priority was migrating its critical database infrastructure. Within about three months of its outreach to Microsoft, SANDAG had established an initial cloud-based environment that was ingesting data with Azure Data Factory. As advanced data capabilities such as Azure Synapse Analytics came online, the SANDAG team used the Azure-based solution to increase its processing capabilities by almost 10 times. The pilot convinced SANDAG to pursue a broader cloud-based data lakehouse architecture on Azure.

Reducing data processing from months to days with support from Microsoft partner Quisitive

During the next nine months, SANDAG worked with Microsoft partner Quisitive, a cloud solutions company that has extensive experience with Microsoft offerings, to build an integrated and scalable data management system. SANDAG found that Quisitive understood the agency’s data processing challenges, underlying business problems, and resource limitations and provided direction for a successful migration. SANDAG established a formal data lake with virtually limitless storage using Azure Data Lake Storage, which helped the agency eliminate its information backlog.

“In the past, when we acquired a new data set, we had to evaluate its usefulness and design schemas and prepare ingestion scripts—a process that could take two or three months because some of the data sets are very complex,” says Flyte. “We’re doing that now in days or weeks, an overall 30–40 percent gain in speed and efficiency.”

SANDAG also reduced the time needed to process modeling outputs from hours to minutes. To inform infrastructure planning, the agency runs hundreds of scenarios to optimize different criteria, such as social equity, air quality improvement, and congestion relief. Each set of models generates terabytes of data that the agency needs to store, and by using Azure Databricks, SANDAG can process it within minutes of a model completing its run.

Fostering an innovative work culture across SANDAG

SANDAG’s digital transformation boosted internal interest in innovating through technology to advance the organization’s planning mission. As the cloud migration strategy evolved, SANDAG employees in various departments wanted to explore the potential benefits for their own work. “We’re probably about 18 months in, and our cloud-based strategy has grown across the entire agency,” says Flyte. “Groups that had never really been involved with IT saw the success of the solution and how quickly it could be deployed, and they wanted to see how they could use it too.”

Employees also found that the impressive reduction in manual processes gave them more time to support other organizational goals. People who had been dedicated to receiving, cleaning, and processing data now have additional capacity to further mine and enrich it, supporting better analysis and decision-making. The IT staff also benefited from the rollout of Microsoft Sentinel, which boosted its cybersecurity posture with a range of plug-ins. “Before Microsoft Sentinel, we were very out of date,” says Bill Mount, Information Security Officer at SANDAG. “With this service, we were able to develop new capabilities with our existing staff. Everything was out of the box for Microsoft Sentinel, and I just had to enable and connect it.”

Using comprehensive data analytics to serve the San Diego region now and in the future

The decision to migrate to an Azure-based solution is helping SANDAG provide more comprehensive data and realistically forecast future infrastructure needs. As a valued source of information for regional developers and city planners, SANDAG has transformed its ability to quickly ingest and analyze large data sets, providing new insights for itself and the San Diego region. Looking forward, the agency is excited to explore how AI models and Azure Cognitive Services can further advance its mission.

SANDAG’s collaborative relationship with Microsoft is a key part of its success. For state and local governments aiming to manage vast quantities of data in the modern era, SANDAG’s transition to a cloud-based solution is a blueprint for a successful modernization. “I wish I’d known sooner that we had these great resources available from Microsoft and that the company would support us through the entire cloud transition,” says Flyte.