Automated Discovery of Search Interfaces on the Web

Web search engines work well for finding crawlable pages, but not for finding datasets hidden behind Web search forms. We describe a novel technique for detecting search forms, which could be the basis for a next-generation distributed search application. We use automatic feature generation to describe candidate forms and C4.5 decision trees to classify them. In two testbeds, we get an accuracy of more than 85% and a precision of more than 87%. One of our decision trees is effective on both testbeds, suggesting that it is a useful general-purpose tree.