Distributed Indexing of world wide web using migrating crawler

The explosive growth of the web-based resources coupled with high-rate of change has made an up-to-date index needed for search (engines) a challenge. Crawlers are no longer capable of keeping up pace with the rapid growth of web using traditional approaches. The present paper offers an alternative method called distributed crawling using mobile agents. The aim is to offer a scalable crawling methodology that is less network intensive, would keep pace with changes in resource profiles, employs time perception and could be easily upgraded.