eWorks | Referenz | Automatic web crawler

Automatic web crawler

Automated analysis of websites

eWorks planned and developed a web crawler for the automated analysis of websites on behalf of an information service provider. The objective was to increase data quality and processing speed while at the same time reducing the manual workload.

The task was to carry out automated Internet searches based on data records from an input database and to store the results found in a results database. The web crawler developed solves the task by searching for increasingly precise information in several phases. In each phase, the results deemed to be less promising are eliminated and the remaining results are used in the next phase. The results are interpreted and summarized using powerful, heuristic algorithms that have been developed in-house and integrated into the web crawler.

eWorks designed the web crawler, developed it with C# and Microsoft.NET as a multi-threaded WinForms application and handed it over to the client. Since then, the client has been using the web crawler on a regular basis, significantly reducing the workload for previously manual searches and increasing throughput at the same time.