The history of the Twotrees Indexing Web Crawlers dates back to the inception of the public Internet. In the early 1990s, Twotrees began operating two different indexing web crawlers that traversed and indexed the web.They were housed on two SGI supercomputers with a total of 64 processors and two fibre channel storage arrays. From 2001-2009, the two crawlers were jointly indexing over one million websites per hour. By the mid-2000s, the two crawlers had indexed over 800 million domains and 6 billion websites. The bots were used in combination with proprietary ASA hardware for the purpose of filtering Internet access for school districts. In 2010, to adapt to the changing nature of the Internet, the bots were redesigned to use a revolutionary indexing method and distributed over a grid of 200 bot nodes around the world.  In 2008, the Twotrees Willow Crawler was ranked in the top 30 indexing bots on the Internet. The two web crawling bots are still active though they have been downsized to a fewer nodes and the agents have been redesigned.

 

If you have experienced an issue with one of the bots, do not hesitate to report the issue to us through our Contact page.