A new machine learning model and Internet of Things (IoT) sensors and automation enables Microsoft smart buildings to keep company employees as comfortable as possible. Microsoft’s real estate operations team relies on energy smart buildings, structures with interconnected automation and sensors, to responsibly maintain a base level of comfort.
Microsoft has deployed more than 50,000 sensors in roughly 100 buildings throughout Microsoft’s Puget Sound region in Washington state. The company is using data captured from these sensors to identify issues and inefficiencies as they happen, allowing them to be fixed before employees even notice them.
“Hot and cold calls are the biggest part of our facilities management requests,” says Mark Obermayer, a senior program manager on the Real Estate & Facilities (RE&F) team, the group responsible for managing the buildings across Microsoft. “A lot of our work is making sure our employees are comfortable and productive. It makes a big difference.”
Fortunately for those responsible for responding when one of these sensors goes off, the vast majority of all the signals emitted from Microsoft smart buildings don’t necessitate a response. Puget Sound could see hundreds of thousands of signals in a single week, with fewer than 1 percent being actionable.
“A portion of a building being off by a couple of degrees might not be a big deal,” Obermayer says. “It might be that the wind is blowing from the north that day.”
What Microsoft Digital came up with was a way to not only generate work orders in a quick manner—a few clicks—but also to predict which faults are a priority.
– Mark Obermayer, senior program manager, Real Estate & Security
To wrangle and maximize this data, RE&F tapped Microsoft Digital, the organization that powers, protects, and transforms the company, to figure out when a response is needed.
This meant finding a better way to parse the plethora of IoT data that the sensors were producing. In short, artificial intelligence and machine learning were needed.
“In the past, someone would manually enter tickets to check out a group of faults,” Obermayer says. “What Microsoft Digital came up with was a way to not only generate work orders in a quick manner—a few clicks—but also to predict which faults are a priority.”
[Discover how Microsoft’s smart buildings showcase Azure Digital Twins. Learn about Microsoft’s new era of smart building in Singapore. Find out how Microsoft promotes environmental sustainability from the inside out.]
Making sure work orders work
As sensors from Microsoft smart buildings feed this IoT data to Iconics (a third-party solution), faults, or specific violations of established rules, are identified. When a fault is recognized, a technician creates a ticket in Facility Link, the building management system Microsoft Digital built on Microsoft Dynamics 365 to manage work orders.
“Iconics and Facility Link weren’t communicating,” says Garima Gaurav, a senior program manager with Microsoft Digital, who identified several opportunities to introduce improvements across Microsoft’s heating, ventilation, and air conditioning (HVAC) systems. “Some technicians were spending the same amount of time writing tickets as working on the fix.”
In addition to being inefficient, manual processes were generating errors due to incomplete or inaccurate tickets. Incorrect work orders left jobs unfinished, leaving equipment running suboptimally and requiring additional technician visits.
To fix this, Obermayer and Gaurav reached out to Kundan Karma, a senior software engineer with Microsoft Digital.
“Technicians had to go to two places,” Karma says. “They went to Iconics, to perform the analysis, and they used Facility Link to submit the ticket. The new IoT Connector that we built brings them together.”
Built on Microsoft Azure, the IoT Connector immediately removed manual steps, reducing errors, and improving communication. Creating a ticket became a one-click process, with greater accuracy and faster processing time for technicians.
“In the IoT Connector, we take care of all the data,” Karma says. “It’s a bridge between two systems.”
Designed with auto-healing and telemetry fail-safes, the IoT Connector gives RE&F confidence that faults will be captured and reported as tickets with greater accuracy.
“If messages between the two systems fail, the IoT Connector will resubmit,” Karma says. “After a certain number of retries or if there’s a major problem, it will create a ticket for an engineer to look at.”
Improved communication introduced a handful of ancillary benefits—specifically, visibility.
Where a technician might previously circumvent inputting information into a work order, automated copying facilitated by the IoT Connector made tickets in Facility Link a single click away.
“In cases where someone just does the fix without a work order, we don’t know what’s been done,” Obermayer says. “This left us with an incomplete history. We couldn’t see the demand for certain things.”
Now capable of tracking work orders, RE&F has a better understanding of what’s going on within specific buildings and assets. These insights are improving decision-making, especially as it relates to energy efficiency.
A firehose of IoT data
The IoT Connector shines a light on some challenges that come with scaling energy smart buildings.
“The target was 100 buildings,” Karma says. “We were so focused on integrating Iconics with Facility Link that we didn’t consider the volume of data. When we first rolled out the IoT Connector, we had to stop at 13 buildings. One building was generating approximately 2,000 faults per day.”
Extrapolated across Puget Sound’s 100 buildings, that amounted to roughly 200,000 faults in a single day. The scale of data being generated by IoT sensors could overload Microsoft’s entire Dynamics 365 system, bringing things to a standstill.
“The issue was conversions,” Gaurav says. “Only meaningful faults require an actionable response. We only want to check on real issues.”
Getting useful information out of IoT sensors is a challenge.
“There are different tolerances and different polling schedules for different pieces of equipment,” Obermayer says. “It changes from building to building.”
Microsoft Digital needed to separate the wheat from the chaff.
“If you have data generated in the thousands, it’s easy to miss important alerts,” Gaurav says.
Reducing the number of faults meant rethinking the way alerts from energy smart buildings were generated.
“What we realized is that 75 percent of the total faults were coming from one source, terminal units, and most of them were never converted to any work orders,” Gaurav says. “It was taking up most of the UI and creating too much noise. The way this data is now processed has adjusted how we’re digesting and prioritizing alerts.”
Terminal units, for example, were reordered and reprioritized to reduce the amount of noise being generated.
“We tried to group faults together,” Gaurav says. “One fault can trigger other alerts, but you don’t need multiple work orders.”
We want the model to mimic the behavior of a technician. It can go through the same decisions a human being can and reach the same conclusion.
– Kundan Karma, senior software engineer, Microsoft Digital
Instead of treating all alerts as individual issues, alerts could be grouped so several related faults resulted in a single ticket.
“Would a technician investigate that?” Karma says. “We want the model to mimic the behavior of a technician. It can go through the same decisions a human being can and reach the same conclusion.”
Teaching a machine to think like a technician
To get things started, Microsoft Digital looked at the history of faults and determined how they were converted to work orders.
Brendan Bryant, a mechanical engineer with DB Engineering, one of Microsoft’s partners, helped translate the technician’s process to the team. These inputs allowed the Microsoft Digital team to build a machine learning model that could mimic the behavior of a technician.
“We had key performance metrics from six to eight months’ worth of IoT Connector data,” Bryant says. “I helped Kundan look at HVAC telemetry and all the IoT metrics to get his team the information they needed to train the algorithm the right way.”
But before they could get there, naming conventions for assets and structures had to be standardized.
“This is one of the reasons we put in our own system,” Obermayer says. “How things would work was that a vendor would decide on an asset name when the building was constructed, then we’d change vendors or use a different vendor for a different building.”
The result was a variety of similar, yet varied, naming conventions. Facility Link meant RE&F could standardize and align all data points for energy smart buildings across campus.
“We can now look at a data point and tell you the number of air valves in Puget Sound,” Obermayer says. “Data and problem types are now the same on every system, making energy smart buildings more precise and efficient.”
Alignment of nomenclature also meant Bryant could better convey priority issues.
“There’s a lot of engineering intuition involved, especially when checking what’s false and what’s true,” Bryant says. “It’s a large amount of data provided by all of the equipment, so you have to make a judgement based on what you’re seeing.”
To help train the model to identify real issues over false alarms, Bryant and Karma moved away from real-time response and started viewing faults in aggregate.
“Something might show up on a Tuesday and be gone by Wednesday,” Bryant says. “There’s no value in creating a work order for that. But if it’s an issue for most of a week, that’s something we want to flag.”
Once aggregated, certain key performance metrics became strong predictors of a fault.
“In order to maintain high confidence that a fault needs to be addressed, we need a longer period of data,” Bryant says.
As the team continued their efforts, items that would result in a work order were flagged while all others were archived. From this, the model began to predict the faults that would result in work orders, flagging them for attention and archiving the rest.
“The technician can view anything flagged as ‘false’ and review it,” Karma says. “If needed, the technician can pull the fault from the archive and review it on the fly. The model learns from the mistake when it’s time to retrain.”
Thanks to machine learning and new practices, the number of faults was reduced by 80 percent to 90 percent.
“When we were onboarding, we couldn’t do all of Puget Sound’s smart buildings because the number of faults was huge,” Gaurav says. “Once we were confident that the faults generated were manageable and convertible to work orders, we were able to quickly onboard the rest of campus.”
Predicting the future for smart buildings
With the IoT Connector, Microsoft’s technicians are more efficient, disparate systems are better integrated, and modern infrastructure is in place to further sustain energy smart buildings.
“Right now, we’re only looking at HVAC, but there are so many other IoT assets throughout Microsoft,” Karma says. “A/V, security cameras—you name it. The next phase is to integrate all of these items into the IoT Connector.”
Flexibility within the IoT Connector allows it to be utilized with any asset across any region in the world.
“It becomes a scalable implementation,” Gaurav says. “We can even use it in areas that will eventually become energy smart buildings to help support those efforts.”
Karma also sees the IoT Connector, which is built on Microsoft Dynamics 365, as being available to other companies looking to improve the efficiencies of energy smart buildings.
“What we’re planning is to create the IoT Connector in a generic way so that other people can benefit from it outside of Microsoft,” Karma says. “Any other team should be able to use our learnings.”
The standardization of assets in Facility Link has helped spur other RE&F initiatives.
“Having this data is super important,” Obermayer says. “This will impact everything from procurement decisions to the management of movable assets.”
As Karma continues to refine the model, retraining hones prediction accuracy.
With each iteration, the model gets stronger.
“The big thing looking forward is helping to teach the algorithm so that we understand when it makes a decision and why,” Karma says. “Eventually the model will be able to assign work orders automatically.”
Gaurav agrees.
“The model is robust and converts some fixed number of alerts to tickets automatically. However, we also allow technicians to review through the list of alerts and allow them to manually create tickets as and when needed,” Gaurav says.
For Obermayer, all of this is a dramatic improvement.
“We started with thousands of faults but could only address about one percent of the issues,” Obermayer says. “We got the number of faults down so that we’re actioning 10 to 20 percent, which means we’re hitting meaningful faults. Artificial intelligence and machine learning are improving the business of energy smart buildings.”
