Posted by
Many businesses and individuals talk a mean game about how they’re changing the world, catering to customer demands, delivering bold new user experiences.
Spotlight: On-demand video
AI Explainer: Foundation models and the next era of AI
Explore how the transformer architecture, larger models and more data, and in-context learning have helped advance AI from perception to creation.
And why not? It’s easy. Such claims rarely are held accountable. They’re easy to twist into whatever direction is most convenient.
Talk is cheap—sometimes. But not always.
I just had a most fascinating discussion with Dimitrios Lymberopoulos (opens in new tab), a researcher in the Sensing and Energy Research Group (opens in new tab) at Microsoft Research Redmond (opens in new tab). He’s at TechFest 2013 (opens in new tab) this week, discussing a project called Enabling Real-Time Business-Metadata Extraction, his focus for the last six months. I asked him a few questions about the project, and he responded in terms at once sincere, ingenious, and passionate.
A lightly edited version of our conversation:
Q: What’s this work all about?
Lymberopoulos: When people search for businesses today, they get a list of businesses with mostly generic information, like an address or a phone number. This is not really helpful when you’re deciding where to go next. We think what users would like to know at the time of a query is: “What is the state of the business? What does the business feel like right now?” For instance, what if we let users know how crowded the business is, or what kind of music it plays, or how loud the music is, or what were the last two songs played, or how loud the human chatter is, or how noisy the business is. That could be really useful information for the user in deciding where to go, given his current context or preferences.
If we have this data, we can put it into a search engine, which then can start understanding the physical world. Today, users can’t come to a search engine and say, “Get me crowded bars playing loud rock music right now.” Search engines don’t know what a crowded bar is, they don’t know what a bar playing loud music is, and even worse, they don’t know if a bar is doing this right now. This is what we’re trying to enable.
Q: How do you do that?
Lymberopoulos: We’re proposing to crowdsource that through real-user business check-ins. We have users carrying devices going to businesses. A subset of those users will try to check in to a business. At the time of a check-in, the user will tell us which business he is visiting, and usually the sensors on the phone are not obstructed, so we can use them to monitor the business environment and make sense of it. We’re leveraging the microphone on the device to record eight to 10 seconds of data, and then we carefully analyze this data to figure out the level of human chatter and music. Then we train decision-tree models that can take this raw audio and map it to data such as the occupancy level, the music level, how loud the human chatter is, how noisy it is.
As people check into the business throughout the day, in essence, we’re creating a new, big data stream that is real-time and focused on physical locations—in this case, businesses. By enabling this data collection, we can enable services such as search engines to provide a totally different experience, like “Get me a quiet Italian restaurant with outdoor seating right now.”
Q: How did you come up with the idea for this work?
Lymberopoulos: I’ve been working on local search over the last two years. Contextual search meant that when people were going to a specific location and searched for something, they would always get the same search results. That was a poor experience. What we were doing was trying to get contextual information to create more dynamic results.
For instance, if you search for “restaurants” in the morning, most likely you’re looking for breakfast places, versus lunchtime, when you look for pizza or burger places, or versus dinnertime, when you might be looking for something better. Then there’s weather information. If the weather is sunny, you might want to go to someplace that has outdoor seating.
This is how it all started: How can we customize the search results to that given user, to that given time, to that given context.
Now, we’re taking this to the next level by generating real-time, rich business data that we don’t have today. By doing that, we create a whole new local-search experience. Our motivation is that the metadata we have for businesses is so poor—and it’s not just us. The metadata we have in search engines today for businesses are very shallow, so shallow that even personalization based on metadata is shallow, as well. Our point of view was: How can we create richer data, more real-time data, that users could find useful?
Q: What part of this work makes you the most proud?
Lymberopoulos: The most interesting thing here, and the thing that I’m really passionate about, is that we’re generating a big data stream that does not exist today.
The second thing is: We’re generating a big data stream that has unique characteristics. It can be real-time and help us understand the physical world. This is information we don’t have today. It opens the door to how we can opportunistically sense the environment and use the sensing data to offer a whole new experience in our services. Search engines are a good vertical example of that: how we can go from a data collection to understanding the data to offering a new local-search experience. How can we understand our physical world in real time and expose that to our services to offer a much better experience to our end users?
Creating this data stream—creating the technology that enables us to collect data we didn’t have before in real time about the physical world—is what makes me the most proud.