Indexing Web Crawlers

Willow and Blackjack

The history of the Twotrees Indexing Web Crawlers dates back to the inception of the public Internet. Twotrees began operating two different indexing web crawlers in the 1990's that began traversing and indexing the web. The two crawlers had indexed over 800 million domains and 6 billion websites by the mid 2000 decade for the purpose of filtering Internet access for School Districts.

They were housed on 2 SGI supercomputers wia total of 64 processors and two fibre channel storage arrays and were indexing over one million websites per hour from 2001-2009. In 2010 the bots were redesigned and adapted to deal with the changing nature of the Internet. The bots were moved to run on a distributed grid of over 200 bot nodes around the world with a revolutionary indexing scheme. The Twotrees Willow Crawler was ranked in the top 30 indexing bots on the Internet.

The two web crawling bots are still active though they have been downsized to few hundred nodes and redesigned to use the agents for security research purposes rather then general Internet indexing. If you have experienced an issue with one of the bots do not hesitate to report the issue to us thought the contact form on our website.