PROSE

AI-Assisted Programming

The PROSE research and engineering team, led by Sumit Gulwani, develops program synthesis APIs over a wide variety of modalities — programming by examples (PBE), natural language, temporal context — with applications in diverse areas such as assisting sloppy programmers through automatic program repair and conversational AI.

Programming by examples (PBE) is a new frontier in AI that enables users to create scripts from input-output examples. PBE can provide a 10-100x productivity increase for developers in some task domains. About 99% of computer users are non-programmers and PBE can enable them to create small scripts to automate repetitive tasks. Moreover, PBE is revolutionizing data wrangling as well, as data scientists spend up to 80% time transforming data into a form suitable for machine learning (ML). We own part of the AI surface for mass-market products over four customer segments: developers (Visual Studio (opens in new tab), VS Code (opens in new tab)), data scientists (Notebooks (opens in new tab)), low-code/no-code users (Excel (opens in new tab), Power Apps (opens in new tab), Power Automate (opens in new tab), Power Query (opens in new tab)), and students (Github Education (opens in new tab)).

We advance the state of the art in program synthesis through innovations and deliver these innovations in mass-market products, as indicated by active customer usage and testimonials. We also heavily publish in top-tier conferences, which gained significant recognition and delivered impact over the broader research community. We empower our team members through the culture of strong collaboration within impactful projects, trusted relationships with product partners, and accelerate impact by giving back through mentorship to long-term interns and research fellows (learn more about our research fellowship program). Our research culture is based on: customer connection that enables us to pick the relevant problems, synergy between research and engineering, an approach towards building foundational frameworks, and cross-disciplinary work.

Read about Flash Fill in Excel

Flash Fill (Excel)
- Perhaps the closest thing to magic Excel users will ever see in a workbook is Flash Fill.
  — Susan Harkins, TechRepublic, 10 best new features in Excel 2013 (opens in new tab)
- This is one of the main reason I purchased Office rather than sticking with free programs, only to find out that it is not supported on Mac versions of Excel. Absolutely unbelievable and disappointing. (What customers say when PROSE technology is taken away from them on missing Flash Fill feature in Excel for MAC)
- Excel without flash fill will be in shambles.
- I thought Excel for Mac has the same functionalities as the Windows version. Apparently not. Disappointed in Microsoft!!! We want flash fill!!
- You NEED to Add Flash Fill for Mac ASAP or I will be forced to stop using Microsoft Services completely. This is using to much of my time.
- Please add this for Mac! the company just gave us Macs & now I’m missing one of the best features in Excel 🙁
- PLEASE add the flash fill feature for MAC! It is an amazing tool, I can’t believe you don’t already feature it on your Mac versions
- Flash Fill is an extremely powerful and useful feature. Porting this to the Mac would greatly expand it’s usefulness, and lock in Excel as a key tool for anyone analyzing data, regardless of what OS they are running.
Table Extraction from Text Files

Custom Fields (Azure Log Analytics) (opens in new tab)
- I’ve been blown away by what customers are able to do with it. Once customer had a requirement where they needed to use alerts in OMS via custom fields over SCOM data to extract AD information. The information was complicated and laid out in way where extraction via Regex would have been difficult and the customer did not want to learn Regex. Custom fields enabled them to extract the data easily so they could build alerts and dashboards based on these new fields.
- Custom fields is very useful and is used in all my demos. Once I dug into it I found so many use cases! All of our verbose alert descriptions are great now because now I can pull the data out easily and put it into visualizations. The accuracy of the field extractions has improved so much since the preview I haven’t had any issues in a long time. — Shawn Tierney, MVP, Infront Consulting
ConvertFrom-String (PowerShell) (opens in new tab)
- PowerShell v5 ConvertFrom-String query by example templates are brilliant
  — Doug Finke, PowerShell MVP
- Just… amazing. The number of transformations it can run through for this is incredible. Can’t wait to do a video on this new feature.
  — Reid Havens, Power BI MVP
- Not certain how you managed to get me excited about TEXT files but you did it!
  — Charles Sterling
Blog posts
Table Extraction from WebPages (PowerQuery)

🏆 Voted top 5 out of 150 new features in Power Query in 2018.
- For anyone that has futilely struggled expanding their way through the DOM of a web page in Query Editor, “Web by Example” is the best thing since sliced virtual network architectures over 5G.
  — Greg Deckler
- Indeed is a great feature. I’ve been doing it with Python and beautifulsoup. having that feature in PowerBI is great! 🙂
- I got so excited about this the day of release! I’ve managed to get so many obscure things working!
- That is absolutely fantastic. I use the web connector a lot, it is one of my favorite connectors
- A very exciting improvement to getting data from the web
- This update is really cool.
Youtube videos created by users
Table Extraction from PDF (PowerQuery)

🏆 Top requested feature (opens in new tab) on Power BI Ideas (opens in new tab) (3000+ votes)
- PowerBIs built in data from PDF function. It works very well is has been reliable throughout my testing.
  — dantm on Reddit
YouTube videos
- Extract Data from PDF Files with Power Query in Power BI (opens in new tab)
- Importing PDF Data Into Power BI (opens in new tab)
BluePencil (IntelliCode Suggestions in Visual Studio)
- Saw IntelliCode for Visual Studio pop in for the first time today. Saved me several hours of work. Very nice.
  — Evil Blond Dad (@sonofalink) May 14, 2021 (opens in new tab)
- As for what I love is the encouragement it gives to actually start a refactor due to the amount of work it can save me.
  — Mordechai Zuber (@mordzuber) May 13, 2021 (opens in new tab)
- I’m in the middle of changing a load of Memory and Span to their ReadOnly variant, and the new @VisualStudio (opens in new tab) intellicode feature knows what I’m doing and starts suggesting the change to ReadOnly in the right places, which is pretty wild.
  — Alistair Evans (@evocationist) April 13, 2021 (opens in new tab)
See more at Blue-Pencil: customer quotes

Blog posts
- DSLs: Given a domain-specific language (DSL) and input-output examples for the desired program’s behavior, PROSE synthesizes a ranked set of DSL programs that are consistent with the examples.
  - Data.Diff: Find differences in distribution of numeric, string, and categorical data.
  - Detection: Libraries for detecting data, encoding and file types.
  - Extraction.Json: Extraction.Json automatically extracts tabular data from Json files. It supports extracting Newline Delimited Json and truncated Json.
  - Extraction.Pdf: Extract tabular data from PDFs.
  - Extraction.Spreadsheets: Extract tabular data from Excel files.
  - Extraction.Text: Extraction.Text extracts data from semi-structured text files using examples. The Usage page and the Sample project illustrate the API usage.
  - Extraction.Web: Extract tabular data from webpages.
  - Matching.Text: Matching.Text automatically identifies different formats and patterns in string data.
  - Read.FlatFile: Automatically infer properties of CSV and fixed-width files (skipped lines, delimiters, etc.) to parse the files.
  - Split.Text: Split.Text is a system for splitting data in plain text format, where there may be multiple fields that need to be separated into different columns.
  - Transformation.Json: Transformation.Json transforms the structure of Json using input/output examples.
  - Transformation.Text: Transformation.Text is a system that performs string transformations using examples allowing for many tasks involving strings to be performed automatically.
  - Transformation.Tree: Transform ASTs using examples.
- NuGet package (opens in new tab): NuGet package for automatically synthesizing programs from examples.
- Tutorial (opens in new tab): A step-by-step walk-through of the process of building a DSL in PROSE and enabling program synthesis for it.
- API and samples (opens in new tab): If you want to apply an existing PROSE DSL, check out its documentation and our samples (opens in new tab).
- Release notes (opens in new tab): Read the release notes for the SDK.
- GitHub (opens in new tab): PROSE is not open source, but we do publish on GitHub sample code using the SDK. For questions, email us (opens in new tab) or open an issue on GitHub (opens in new tab).
- FAQ: Get answers to frequently asked questions.

PROSE

Flash Fill (Excel)

Table Extraction from Text Files

Table Extraction from WebPages (PowerQuery)

Table Extraction from PDF (PowerQuery)

BluePencil (IntelliCode Suggestions in Visual Studio)