Like most organisations, yours needs to extract information from the Internet in order to make timely and accurate business decisions. Your job is choosing the best way to get that information and make it usable for a wide range of business processes. Basically, you have two ways you can go – web scraping or web information integration.
Whether you are acquiring a million records a day from a few hundred sites or just a handful of records from thousands of sites daily, focusing on just one aspect of data handling will leave you short. Here’s a start to defining the most important capabilities when it comes to extracting web data and transforming it be leveraged within any of your applications or processes.