Results 1 to 2 of 2

Thread: How the search engines plumbs every website?

  1. #1
    Join Date
    Jan 2013

    Arrow How the search engines plumbs every website?

    If i wanted to create a script for indicize various website, what technology i should use? How search engines like google plumbs every website and memorize them in a database?
    I want to create a script that indicize specified sites (call them A, B and C). How a script can open an url and examine their contents?

  2. #2
    Join Date
    Aug 2004
    Many programming languages include an implementation of cURL. Then they may use something like libXML to parse the text returned by the URL that was followed, to find more links to try and to analyze the page contents. For instance, in PHP

    PHP cURL
    PHP DOM (libXML)
    "Well done....Consciousness to sarcasm in five seconds!" ~ Terry Pratchett, Night Watch

    How to Ask Questions the Smart Way (not affiliated with this site, but well worth reading)

    My Blog
    cwrBlog: simple, no-database PHP blogging framework

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
HTML5 Development Center