Searching multiple domains with Alltheweb

[martinbsp]

For some time now I've been looking for a way to search from a specified group of websites (added to on a continuous basis by myself) using the 'live' database of one of the search engines.

I've now found a way of doing this a post on Erik Benson's website - he pointe out that Alltheweb.com allows the searching of an unlimited number of multiple domains at the same time:

http://alltheweb.com/search?q=[vti]&advanced=1&dincl=imeche.org.uk+cetr.com+vti-europe.org


What I want to be able to do is incorporate a version of the above code into a standard "Alltheweb.com" search box. I will manually add domains to the above line as I find them, all I want the end user to be able to do is search within them via a standard search box.


Here's the code for the "Alltheweb.com" search box:

<form action="http://www.alltheweb.com/search" method="get"> <p><img src="http://www.alltheweb.com/g/atwlogo_small.gif"
width="94" height="16" alt="AlltheWeb :: find it all" /><input name="q" size="30" /> <input name="cat" type="hidden" value="web" /> <input name="charset" type="hidden" value="selected" /><input type="submit" value="Web Search" /></p> </form>


More versions can be found at: http://alltheweb.com/help/webmaster/add_to_site



I'm not a programmer so this is beyond me. Is the above a fairly simple process and, if so, would someone be able to post some code up for me?

Thanks

[DaveSW]

I don't know if you can do that with all the web but google definitely does.

http://www.google.com/services/free.html

If you register for more free services then you can add multiple domain names, co-branded.

[martinbsp]

Dave

I think you might have missed my point...

I want to search more than one domain of my own choosing AT THE SAME TIME through a search engine.

Google will not allow me to do that as it's limited to 10 words.

Alltheweb and Yahoo do allow this, so for example if you cut and paste

http://alltheweb.com/search?q=[vti]&advanced=1&dincl=imeche.org.uk+cetr.com+vti-europe.org

into Explorer, you will get all the results for the keyword "vti" from the domains imeche.org.uk, cetr.com and vti-europe.org


See my previous post.

Thanks

[DaveSW]

Originally posted by martinbsp
I want to search more than one domain of my own choosing AT THE SAME TIME through a search engine.

Google will not allow me to do that as it's limited to 10 words.


Google free site search is not limited to 10 words and will search any number of domains. However, you must register to access this service

read the bottom of this page:

http://www.google.com/searchcode.html

[martinbsp]

BUT it won't let me search only the domains specified by me at the same time.

It will let me search as many domains as I choose one by one, but not all together at the same time.

So it doesn't do what Alltheweb.com can

[DaveSW]

Viewing the page you link to it only shows results on vti-europe.org for me.

[martinbsp]

First of all, make sure you copy the whole of the link.

Now, replace the word vti in the [ ] with the word "Tribology" and you'll find you get results from all the pages.

[martinbsp]

Dave

now you know how it works do you know how to answer my original question?

Thanks

Martin

[DaveSW]

I'll experiment with it.

Sec

[DaveSW]

Do you have an example with one domain in it?

[martinbsp]

If you understand that bit of code you just need to cut it to give you that.

[DaveSW]

I fully understand the piece of code. However, that is for the simple web search. To change that to give the output you want I really need the code for their advanced search that searches one site.

The other alternative might be to write a piece of javascript that simply takes the search string and adds in the extra end part of the url.

[martinbsp]

Hi Dave

I actually only need their simplist search functionality, i.e. searching for one keyword at a time only.

Once the search has been done I can use the "Search within results" option at the bottom of the results page to progressively narrow down the search.

Hope this helps.

[DaveSW]

do you have php support on your server?

[martinbsp]

Dave

I have absolutely no idea. Would I need it? Could I not just stick up a normal webpage with the box on? Sorry, I'm not technical so these questions might seem a bit daft.

Any chance I could give you a call about this?

Thanks

Martin

[DaveSW]

OK what it is is this:

Using the simple form tag with no other scripting, all my 'unusual characters (like [ ] + etc) are being converted into their codes. I don't know whether it's screwing it up or not.

I was thinking of using php to submit the form to, so that the php would just drop that search term into a a new url.

However, if your server doesn't support php I'll just have to try javascript.

[martinbsp]

Ok, perhaps the Java script route might be simpler.

Thanks very much for your help.

Martin

[martinbsp]

Dave

I'm having to leave the office now for a meeting but hope to catch up with you about this on Monday.

Have a great weekend.

Martin

[DaveSW]

Sure thing. I'll see what I can do.

[DaveSW]

I've asked Pyro to help me on this one, and I think he's done it the javascript way.

;)

[pyro]

This should be close to what you were looking for. If you want, I can show you the PHP version, as well.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Example</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<script type="text/javascript">
function searchatw() {
query = document.searchfrm.searchfield.value;
window.location.href = 'http://alltheweb.com/search?q=['+query+']&advanced=1&dincl=ryanbrill.com+alistapart.com+mezzoblue.com';
return false;
}
</script>
</head>
<body>
<form name="searchfrm" action="" onsubmit="return searchatw();">
<p><input type="text" name="searchfield">
<input type="submit" name="submit" value="search"></p>
</form>
</body>
</html>

[pyro]

Since it was so simple, here's the PHP version, too:

<?PHP
if (isset($_POST['searchfield'])) {
$search = $_POST['searchfield'];
header('Location:http://alltheweb.com/search?q=['.$search.']&advanced=1&dincl=ryanbrill.com+alistapart.com+mezzoblue.com');
}
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Example</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body>
<form name="searchfrm" action="<?PHP echo $_SERVER['PHP_SELF']; ?>" method="post">
<p><input type="text" name="searchfield">
<input type="submit" name="submit" value="search"></p>
</form>
</body>
</html>

The header should be all on one line, though...

;)

[martinbsp]

Morning Pyro / DaveSW

what a great suprise to be greeted with on Monday morning. Thanks guys, very much appreciated.



"Since it was so simple, here's the PHP version, too:"

Er, I have a Masters in Philosophy which was easier than understanding this little lot !!

You guys are genius.

Martin

[martinbsp]

Just had a few more thoughts regarding search engines on a shoestring.

This is probably a bit naughty...

Once a search has been carried using the code below, is it possible to then apply Vivisimo's clustering engine (free too!) to the results and get Vivisimo type results for the search? My guess is no, because Vivisimo rides on the back of other search engines.

Ideas guys?

I guess it MAY be possible if the results page(s) could automatically be inserted into the Vivisimo box (as the link would be different every time).

http://vivisimo.com/link2us/Use_Vivisimo_to_search_your_Site.html


Martin

[DaveSW]

Naughty?

That word doesn't appear in my dictionary. What does it mean? ;)

You could change the code to search Vivisimo but I don't know the code for searching multiple urls in Vivisimo. I can get the code for one domain but not multiple. Unless you just happen to have it?

:D

Incidentally, (I don't know if you already are) but using frames would enable you to 'brand' your search results.

[martinbsp]

Dave

I don't have the Vivisimo code at that level so I'm giving up on that one. Their engine is quite tasty but it relies on metadata searching, an approach I'm not keen on (I know from our own organisation that metatagging is just not something people are bothered with doing properly).

I had however thought of another way of presenting results in a clustered format to make for easier reading.

After having done the search is it possible to amend the code to get the results to be clustered/displayed by their respective domain whilst still retaining their FAST relevancy ranking within the respective clusters?

So for example, you have:

CLUSTER: sbac.co.uk
result
result
result
result

CLUSTER: aviation-industryconferences.com/
result
result
result
result

etc
etc


Martin

[DaveSW]

By using the alltheweb search, you are simply passing the search terms to alltheweb and sending your visitor to the results page. You don't have any control over how the results look.

Unless there is a clustering option available for alltheweb, which can be turned on by a keyword in the url, it would be nearly impossible. Checking their advanced search page I can't find any such option.

It would be possible using iframes, but I guess it would start to get a bit messy that way.

[martinbsp]

Hi

I wanted to see if the functionality of the code posted here for me can be developed further.

Using the alltheweb index I can search as many pre-selected domains as I wish, unlike Google which has a limit of 10 (excluding wildcards, which defeats the object of the exercise).

I recently downloaded the Google Desktop searchbar. What I want to know is, can I type a query into the deskbar engine and get it to automatically query the alltheweb.com index via the pre-programmed domains I have selected? This means I'd be using the FAST index and not Google's which is fine.

I have read
(http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&group=google.public.labs.google-deskbar&selm=a39c93c1.0402011448.94c5194%40posting.google.com)

that it is possible to build a permanent custom search into the deskbar using the Windows Registry. I'm aware that this is of little use in this scenario, but it led me to think that maybe there is a way I get the deskbar to point at the alltheweb.com customised search.

What I want people to be able to do is allow download and use the deskbar to carry out searches, but use alltheweb.com index to do the custom searches in the domain names I have specified.

This would mean that I'd have to do is update the domains I want indexing in FAST remotely from my PC - the users would have to do nothing from their end.

Phew!

Any ideas/code most welcome.