The history of the Twotrees Indexing Web Crawlers dates back to the early days of the public Internet. In the 1990s, Twotrees launched two distinct web crawlers designed to traverse and index the growing web. By the mid-2000s, these crawlers had indexed over 800 million domains and 6 billion websites, primarily for the purpose of filtering Internet access for school districts.

The crawlers were powered by two SGI supercomputers, each equipped with 64 processors and two fiber channel storage arrays, allowing them to index over one million websites per hour from 2001 to 2009. In 2010, the bots underwent a redesign to keep up with the evolving landscape of the Internet. They were transitioned to a distributed grid of over 200 bot nodes located worldwide, utilizing a cutting-edge indexing system. The Twotrees Willow Crawler became one of the top 30 indexing bots on the Internet.

Today, both crawlers remain active, albeit on a smaller scale, with only a few hundred nodes in operation. Their primary focus has shifted from general web indexing to supporting security research. If you encounter any issues with one of the bots, please feel free to report them via the contact form on our website.