In June 1993 Matthew Gray at MIT developed the web crawler, World Wide Web Wanderer, to measure the size of the web. Later in the year the World Wide Web Wanderer was used to generate an index called the "Wandex", providing what was probably the first web search engine.
"Matthew Gray is the author of the World Wide Web Wanderer, the first autonomous agent on the web, used to track web growth since 1993. He is the worldwide authority on historical growth of the web. He set up one of the first 100 web servers in the world, www.mit.edu, in June 1993. In 1994, he founded a company, net.Genesis, to develop tools for web developers. With net.Genesis, he co-authored a book "Build A Web Site", published by Prima Publishing. Additionally, he was a member of the Apache Group, a non-profit group of volunteers who wrote Apache, the worlds most popular web server. In January, 1996 he left net.Genesis, which still continues to operate, to return to MIT to complete his degree in physics. He received is undergraduate degree in physics in 1997 and his masters degree from the MIT Media Lab in 1999. From 2001 to 2006, he was CTO of Newbury Networks, Inc., a provider of wireless location technology. He is currently a software engineer at Google" (http://www.mit.edu/~mkgray/bio.html, accessed 9-2020).