Posted by
Earlier today, during Microsoft Research’s Silicon Valley TechFair (opens in new tab), we learned about visualization of big-data collections using Holograph. Another aspect of the big-data movement, though, is enabling data analysts to develop an application and then deploy it seamlessly to the cloud.
Spotlight: Event Series
Microsoft Research Forum
Join us for a continuous exchange of ideas about research in the era of general AI. Watch the first three episodes on demand.
Such processing was the focus of the TechFair project called Naiad on Azure: Rich, Interactive Cloud Analytics, the subject of plenty of interest on the show floor. Derek Murray (opens in new tab), a researcher at Microsoft Research Silicon Valley (opens in new tab), host of the event, found a few moments to pull himself away to discuss the latest extension of the Naiad effort.
Naiad (opens in new tab), a .NET-based platform for high-throughput, low-latency data analysis, includes tools built atop Microsoft Azure (opens in new tab) to deliver interactive analyses of huge data sets.
From Murray’s perspective, it’s all about the users.
“We have made it easy for people to get their data into Azure," he says, "feed it into Naiad, start processing, and serve it using Azure websites.”
You can get more detail at the Big Data for Developers (opens in new tab) website, but for programmers, the next step for interested programmers is to take Naiad—and its cousin project, DryadLINQ (opens in new tab)—out for a spin.
“We’re targeting the people who want to build,” Murray confirms. “What we want to do is to show that if you’re an experienced programmer, and you just want to take your application or your data-processing pipeline and scale it out across a cluster, we’ve got the tools to make it easy to take what you’ve already got and put it in a distributed setting.”
Naiad was born from dissatisfaction with an existing solution. Murray and his colleagues Michael Isard (opens in new tab) and Frank McSherry (opens in new tab) knew how to write programs that ran across a lot of machines, but the tools for distributing them needed to deliver better performance. The team looked for areas for exploration, new programming models, and finally selected what Murray calls “a really solid set of low-level system principles that we could talk about in research publications.”
Now, though, they want users. That need, coincidentally, plays to a position of strength—within Microsoft Research and among Microsoft customers and partners.
“You can say, ‘What is the next feature we’d like to add?’ write a paper about that, and follow up a year later,” Murray says. “Or you can try to get people to use it. That is a wonderful thing about working for Microsoft: There’s a ready-built audience for it internally and also a huge ecosystem that might be potential users for this. We’re trying to engage with them, get them interested, find out once they’ve used it what their pain points are, and use that to motivate our research going forward. We want to take advantage of the amplifying effect of working for Microsoft Research to take in the bigger picture and do better, more relevant research.”
It’s an ongoing concern, but the researchers continue to push forward, and the prospect of moving to a user-generated research phase provides plenty of motivation.
“Success for a project like this is when there are people who are relying on it for revenue-generating business,” Murray states. “For me, the personal satisfaction I would get from a system is when I would see people actually using it and it’s actually benefiting them.
“This kind of systems research is supposed to be productivity enhancing. You can measure how quickly it runs and how much improved it is, but the actual objective of this work is to make people’s lives easier. When we discover that it is making people’s lives easier, that it’s making people more productive, that’s when we’ve know that we’ve succeeded.”