What Are Spiders?

[AjaxPress]

MSNBot Spider
Yahoo! Slurp Spider
Google AdSense Spider
AskJeeves Spider

I see these things while stumbling through the list of currently active users. It seems like some sort of marketing thing. Also that MSNBot Spider is particularly active.

[drgoodtrips]

The way that search engines give you results so quickly is that they separate looking up the pages from providing them to you. That is, when you do a search, the google server doesn't say "let me go out and scour the internet for 'good+chinese+food'". Google has a whole boatload of sites indexed and saved on their servers that they can quickly provide to you (this is what it means when you click the 'cached' link during a google search - google shows you its copy instead of redirecting you to the link).

In order to do this, the search engines are constantly revisiting the sites that have registered with them and updating the content. Those are what the 'spiders' are. Any discussion site that registers with search engines must provide the search engine a means of indexing and storing its content. This isn't advertising, per se (though it indirectly translates into advertising as that is how the search engines pay for all of those icy cool rooms full of serve farms).

[Speakeasy]

Before a search engine can tell you where a file or document is, it must be found. To find information on the hundreds of millions of Web pages that exist, a search engine employs special software robots, called spiders, to build lists of the words found on Web sites. When a spider is building its lists, the process is called Web crawling. (There are some disadvantages to calling part of the Internet the World Wide Web -- a large set of arachnid-centric names for tools is one of them.) In order to build and maintain a useful list of words, a search engine's spiders have to look at a lot of pages.

How does any spider start its travels over the Web? The usual starting points are lists of heavily used servers and very popular pages. The spider will begin with a popular site, indexing the words on its pages and following every link found within the site. In this way, the spidering system quickly begins to travel, spreading out across the most widely used portions of the Web.


"Spiders" take a Web page's content and create key search words that enable online users to find pages they're looking for.


Google began as an academic search engine. In the paper that describes how the system was built, Sergey Brin and Lawrence Page give an example of how quickly their spiders can work. They built their initial system to use multiple spiders, usually three at one time. Each spider could keep about 300 connections to Web pages open at a time. At its peak performance, using four spiders, their system could crawl over 100 pages per second, generating around 600 kilobytes of data each second.

Keeping everything running quickly meant building a system to feed necessary information to the spiders. The early Google system had a server dedicated to providing URLs to the spiders. Rather than depending on an Internet service provider for the domain name server (DNS) that translates a server's name into an address, Google had its own DNS, in order to keep delays to a minimum.

When the Google spider looked at an HTML page, it took note of two things:

* The words within the page
* Where the words were found

Words occurring in the title, subtitles, meta tags and other positions of relative importance were noted for special consideration during a subsequent user search. The Google spider was built to index every significant word on a page, leaving out the articles "a," "an" and "the." Other spiders take different approaches.

These different approaches usually attempt to make the spider operate faster, allow users to search more efficiently, or both. For example, some spiders will keep track of the words in the title, sub-headings and links, along with the 100 most frequently used words on the page and each word in the first 20 lines of text. Lycos is said to use this approach to spidering the Web.

More at:

HowStuffWorks "How Internet Search Engines Work"

[Tanngrisnir3]

Originally Posted by drgoodtrips

The way that search engines give you results so quickly is that they separate looking up the pages from providing them to you. That is, when you do a search, the google server doesn't say "let me go out and scour the internet for 'good+chinese+food'". Google has a whole boatload of sites indexed and saved on their servers that they can quickly provide to you (this is what it means when you click the 'cached' link during a google search - google shows you its copy instead of redirecting you to the link).

In order to do this, the search engines are constantly revisiting the sites that have registered with them and updating the content. Those are what the 'spiders' are. Any discussion site that registers with search engines must provide the search engine a means of indexing and storing its content. This isn't advertising, per se (though it indirectly translates into advertising as that is how the search engines pay for all of those icy cool rooms full of serve farms).

These are spiders as well.....

[drgoodtrips]

Originally Posted by Tanngrisnir3

These are spiders as well.....

That kind is a different line of work than mine (though I do find spiders to be interesting).

[Wallaroo]

Originally Posted by Tanngrisnir3

These are spiders as well.....

Is this picture taken in america? Glad I dont live there!

[Donkey_Left]

Spiders are arachnids. Where did you go to school?

[Wallaroo]

Are all the baby spiders not about to eat their mother on this picture? Can you find spiders like that in america, because thats fucking disgusting?

[Ernie S.]

The large, handsome Trochosa robusta has until recently been living unobserved, despite its considerable size, on the island of Bornholm. The body itself is about 2cm across, while the legs extend to a diameter of 5-6cm, which is big by Danish standards. If you haven't yet stumbled over this species it is probably because it prefers to live on chalk sea-cliffs, where it hides during the day-time in cracks and fissures. It only emerges at night to hunt insects on the vertical cliff faces.

[Wallaroo]

Originally Posted by Ernie S.

The large, handsome Trochosa robusta has until recently been living unobserved, despite its considerable size, on the island of Bornholm. The body itself is about 2cm across, while the legs extend to a diameter of 5-6cm, which is big by Danish standards. If you haven't yet stumbled over this species it is probably because it prefers to live on chalk sea-cliffs, where it hides during the day-time in cracks and fissures. It only emerges at night to hunt insects on the vertical cliff faces.

Thats pretty interesting. Ive never seen spiders larger than a diameter of 6 cms legs included. The summer only last for around 4 months here, so fortunately they die befor getting too big.

Did you btw know that the island of Bornholm had a mountain taller than Mount Everest before the last ice age? It might have been the tallest mountain ever I think.

[Slon]

It's the search engine actively searching for web pages all the time and recording information about links (as well as linking to them) so that you can get a nice title+description after you search for something. It would be like if I found a domain name, say, microsoft.com. Then, I'd create a link to it that says "Microsoft Corporation" linking to microsoft.com and a small description based on words found on that page. Well, search engines do this automatically using computers all the time. Google also sometimes saves a copy of the page. That way, you can not only click on the link to visit the side, but also click on "cached" to view the page as it appeared at the time Google visited it. Good way to get information from pages that have been deleted. Google also converts some pdf files into an HTML version so that people without a pdf reader (or those who don't want to bother running it) can read it. I think they also perform character recognition on image-only pdf files so that you can search for them using text even if they have no true digital "text" in them.