Automating web intelligence generates 6-figure cost savings for this Global Auto Manufacturer.
Process Automation Solution:
Automated Data Collection
- Home
- Solutions
- Process Automation
- Data Collection
Competitive advantages of real-time, automated data collection & syndication
You compete on the quality, completeness, and timeliness of the data behind the services you provide. With the explosion of data on the web, you must be able to efficiently collect, transform, integrate, and migrate the right data to enhance or create new data services — or risk ceding advantage to a competitor who has better data than you. But extracting web data efficiently and completely is impossible using fragile web-scraping scripts or manual cut-and-paste processes.
Kapow changes the game. With the Kapow Katalyst™ platform you can automatically extract and transform data from virtually any web-based data source, including HTML, XML, AJAX, JSON, Flash, FTP, Excel and PDF. With Kapow's powerful and easy-to-use ETL capabilities, maximize process efficiency and ensure the quality and compatibility of extracted data with your existing information systems. Gone are the days of having to rewrite applications to access their data. And the teams of employees who have been manually cutting-and-pasting web data can now be rededicated from drudgery to driving innovation.
Featured Customer Success Stories in Automated Data Collection:
Challenge
- Marketing team receives periodic reports with information on sales, leads, ad revenue and other team performance metrics.
- The information in these reports was collected manually from disparate sources.
- Manual web intelligence affected the timeliness and accuracy of information, making the information less valuable to decision makers.
Solution
- They now use Kapow to automatically collect data from third party marketing sites and feed it into a data warehouse for reporting and analysis.
- Kapow is also a central component of their interactive media ROI tool. Without it they would be unable to collect a critical piece of the data they need for performance metrics relating to online advertising and search placements.
Results
- Data is now accurate and up-to-date.
- 6-figure cost savings relating to vendor reporting and pricing functions.
AGCO, the largest pure play, full-line agricultural equipment manufacturer, uses Kapow to automate web data extraction on hundreds of thousands of parts.
Challenge
- AGCO's parts division needed to pull data from 10 websites, which sell AGCO parts as well as competitors' products.
- Their existing data feed was inflexible. They got infrequent data dumps, and AGCO had to do a lot of manual work on the product description lists.
- Overall not a very good or agile solution, given that they were extracting hundreds of thousands of parts in a very competitive industry.
Solution
- Kapow provides a fully automated data feed solution, including robots to pull from the 10 key websites and a hosted installation of Kapow running on Amazon EC2.
- Kapow also provides ongoing robot maintenance and support.
- An in-house deployment is planned for the next phase.
Results
- Kapow eliminated manual work associated with web data acquisition.
- Competitive information is up-to-date.
This innovative travel service company uses Kapow to comb 500 sites for the best deals.
Challenge
- Aggregate complete and unfiltered travel information from hundreds of travel sites in real time.
- Provide service at no cost to users; rely on sponsored links and ads for revenue.
Solution
- Kapow automates web intelligence with 600+ robots.
- Batch collection runs every few hours. Robot runs are based on frequency of requested routes.
- Provides real-time updates on-demand to confirm current prices.
- User requests are routed to airline sites to book travel.
Results
- Allows users to search almost 500 travel sites and compare results in one legible display. Unlike many aggregator sites, they include major booking engines, national carriers and most low-cost airlines.
- Separates price search from booking process, enabling much more rapid search results.
- Automated web harvesting makes the company's innovative business model economically viable.
Automating web data extraction was key to expanding business for PPG Industries, the world's leading provider of coatings and specialty products and services.
Challenge
- An insurance company has outsourced its claims department to PPG. PPG handles claims requests from car owners through a call center application.
- PPG needs access to each insurance policy data, which typically resides on a mainframe.
- They are now encountering clients whose data resides in web applications. They need to automate web data extraction and integrate the data with internal systems to replace time consuming manual data processing by call center staff.
Solution
- Kapow robots perform web data extraction automatically.
- Robots are wrapped in web services, which make them easy to integrate into the existing solution.
Results
- Automating web data extraction and integration will result in an estimated 50,000 additional cases from the insurance company.
With Kapow, analysts and traders get real-time information on factors that affect trading price.
Challenge
- In 2007, the EU introduced requirements for greater transparency in the utilities market.
- Organizations affected by the new regulations had to deliver hourly, daily and weekly reports online.
- Deutsche Börse wanted to capture this information in real time to inform trading decisions.
Solution
- With Kapow, they were able to extract and transform data in real time, providing analysts and traders with critical information affecting trading price.
- The solution retrieves information from 150 websites.
Results
- Kapow was the only vendor who could fulfill the requirements in their timeframe.
- “With Kapow, we built powerful fundamental data models. Analysts and traders get critical factors that affect the price of a trade in real-time.” - Mario Schultz, Director, Market Data & Analytics, Deutsche Börse
Neckermann, one of the leading multi-channel retailers in Europe, keeps popular products in stock and priced to beat the online competition with automated web intelligence from Kapow.
Challenge
- Neckermann lacked fundamental metrics on their own web business and needed a more effective way to check competitor pricing to compete effectively with online retailers.
- An audit quantified the problem, showing SKUs priced too high relative to the competition and out-of-stocks on popular items.
Solution
- With Kapow, the company has automated web intelligence to support competitive pricing, category management and inventory applications.
Results
- Increased service levels and revenue by reducing out-of-stocks.
- Increased revenue resulting from dynamic pricing strategies informed by real-time view of competitor pricing.
P&G uses Kapow to monitor social media sites for consumer reaction to their company and products.
Challenge
- P&G wanted a more automated way of tracking and responding to social networking discussions that mention P&G products.
- They were performing manual search of websites, but this was too time consuming and they were missing out on vital information needed to make the data sampling relevant.
- The Text Analytics team was responding to requests on an ad hoc basis by doing a search with custom-built robots and providing their analysis in PowerPoint.
Solution
- P&G is now using Kapow robots for social media monitoring.
- They build custom robots to search social media sites for key words related to the brand, circumstance (for example, new product launch) and location.
Results
- LOB managers get real-time insight on consumer reaction to new product releases, product recalls and other issues that affect sales and margins.
- With fresh data, P&G can react and respond quickly to opportunities and threats. This is particularly important for new product releases and product recalls, when problems with products are in the news and under discussion on social media sites.
NewsBank, a content aggregator for small newspapers, reduced their development time for web intelligence scripts from 2.5 hours to 15 minutes.
Challenge
- Replace Perl scripts, which were costly to develop.
- Deliver newsfeeds to customers as RSS feeds.
- Harvest articles covering multiple pages.
- Eliminate “black box” tools. Give developers control of the harvesting process.
- Integrate the solution into their batch scheduler.
Solution
- With Kapow, collected news articles are written to a database and delivered to NewsBank customers as RSS feeds.
- A batch scheduler manages robot runs, which are executed several times a day.
Results
- Reduced development time from 2.5 hours for Perl scripts to 15 minutes for robot “scripting.”
- Increased sourcing. They are now harvesting articles from 300 sites.
- Reduced development and maintenance costs, enabling them to accomplish more with existing staff.
With “directory assistance” from Kapow, this top 5 U.S. bank increases conference call security and engagement.
Challenge
- The bank had no visibility into who was joining and dropping out of conference calls.
- They wanted to maximize conference call participation and productivity while preventing unauthorized access.
Solution
- With Kapow, they were able to aggregate internal and external caller information from internal HR phone registries and public white pages.
- A custom web interface enables real-time call monitoring. They can now monitor all joins, drops and time spent on the call.
Results
- Eliminated unauthorized call access and security breaches.
- Demonstrated viable use of aggregated data for “long-tail” solutions.
- Required development of fewer than ten robots and a simple web interface to integrate data feeds.
With Kapow, InfoGroup collects and refines data to the highest compilation standards from the widest range of web sources.
Challenge
- Top internet search engines, in-car navigation systems and operator-assisted directory services rely on InfoGroup as the source of truth.
- The challenge for InfoGroup was to maintain the highest data compilation standards and seek out additional sources of validated data.
Solution
- Kapow enables them to extract web data from a wide range of sources, including FCC filings from county court houses and basic company data from public websites to supplement the business and consumer data products they sell.
- Much of the information they receive is from subscription-based data feeds. They use Kapow for all custom data requests, which are not available from data providers.
- By automating web data extraction, the Business Content Group can support the diverse needs of all InfoGroup departments and divisions.
Results
- Improved quality of data they provide to their customers.
- Dramatically reduced or eliminated hours of manual web intelligence.
- They can now get to data sources they couldn't reach in the past.
With Kapow, the energy division of IHS can monitor thousands of sites for factors affecting supply and demand, without relying on offshore resources for web intelligence.
Challenge
- The energy division of IHS needed to monitor thousands of government regulatory product vendor websites for factors that influence oil and gas supply and demand.
- They were relying on off-shore resources to access web data, reducing their control of what is a strategic aspect of their business.
Solution
- With Kapow's automated web data acquisition, they can gather and aggregate critical web information in real time without outsourcing.
- No coding or manual data extraction is required.
Results
- Increased efficiency with automated data extraction.
- Enhanced and improved quality of master data.
- Enhanced product offerings based on better web data.
Innovative online lender automates web data extraction from partner sites with Kapow.
Challenge
- The lender's website allows customers to search through a database of thousands of entries, but each required manual web intelligence of photos, descriptions, prices and locations.
- Manual data entry was error prone and costly.
- When source data changed, there was no way to automatically update the information on the lender site.
Solution
- Kapow automates web data extraction from partner sites and integration to lender site.
- Data is automatically updated as changes occur, based on ongoing monitoring of partner sites.
Results
- Significant savings by eliminating manual data entry.
- Data on the lender site is up-to-date and accurate.
With Kapow, Live Matrix can extract data from any site to feed their innovative real-time events portal – even content from sites using JavaScript and Ajax.
Challenge
- The Live Matrix site is a real-time events portal, collecting and displaying data on all types of live events, but they had problems extracting content from sites using JavaScript and Ajax.
- They needed to scale quickly.
Solution
- Kapow enables them to extract data from sites using JavaScript and Ajax.
- Kapow's visual scripting environment enabled the Live Matrix developers to scale quickly.
Results
- Based on a successful POC involving 20 challenging sites, Live Matrix decided to move to Kapow for web intelligence.
- With Kapow's state of the art software, Live Matrix met their launch date and became the first TV Guide for the web.
Automating integration of new account information was key to business growth for XING, a leading social networking site for business professionals in Europe.
Challenge
- XING competes with LinkedIn and other social networking sites aimed at business professionals.
- To encourage adoption of their service, they wanted to automate integration of customer account and contact information from LinkedIn, FaceBook, Hotmail and Yahoo Mail to XING's profile and contact databases.
Solution
- With Kapow, a new XING user can enter his third party network credentials into XING's registration form and robots will log into the third party sites and collect all additional profile and contact information to enrich the new XING account.
- Collected contacts that are not currently registered XING users receive an invitation to join XING.
Results
- Customers save time by having all their contact and profile information entered automatically, making the new service instantly useful.
- XING gains valuable marketing information at no cost.