Design and Implementation of a Language Specific Crawler to Improve Crawling of Persian Web Documents

The huge content and dynamic nature of the web creates new challenges in the area of web crawling. Crawling methods are categorized into general and focused crawling. In the general crawling method, all web pages are gathered but in focused crawling only those web pages are gathered that belong to a specific subject. Language specific crawling is a type of focused crawling that collects web pages and documents which are written in a target language. Since the web contains a wide spread area of unstructured and written data in different languages, web crawling based on a specific language is a big challenge in the area of web information retrieval. In this paper, a Persian language specific crawler has been presented and described. The results of our experiments show that the proposed Persian crawler improved the performance of Persian web pages crawling.