trouble with cURL, java and cookies

[TecBrat]

I have a client who already paid $3k or more to a software company to set up a classifieds application. Obviously he doesn't want me to re-develop that whole app. We are now taking over the site. We will build a brochure site as a front end that we can control SEO and whatever else we want to do. I found that I can access most of the features of the site by scraping it with curl. I am doing a multitude of str_replace to tweak it out to fit our site, then including the result just like I do our content pages. This even works for logging in because it uses a session key that I can add to my links or form inputs. The one thing I am having trouble with is a "save search" feature and a "remember this ad" feature. If I read it correctly in the viewsource, both are using java to set cookies. Am I out of luck on this or can I set cookies myself, on my visitors system, then read them back in and put them in the next cURL request? My skill with PHP is between "okay" and "pretty good". My skill with java is like a monkey with a hammer. I have never used cookies at all.
Thanks,
TecBrat.

[TJ111]

#1. Paragraphs man, paragraphs :)

#2. You can accomplish this pretty easily with PHP using cookies (http://www.php.net/setcookie) or sessions.


This even works for logging in because it uses a session key that I can add to my links or form inputs
#3. Sending the session key in a query is dangerous business. It can cause security problems like session stealing or session fixation. If I read that right, I believe thats what your doing.

[TecBrat]

#1. Paragraphs man, paragraphs :)
Sorry 'bout that. I usually talks much more gooder grammer. :p


#2. You can accomplish this pretty easily with PHP using cookies (http://www.php.net/setcookie) or sessions.

I'll try reading them in, because I think the javascript may already be writing them, now that I think about it. (Remember the monkey and the hammer?)


#3. Sending the session key in a query is dangerous business. It can cause security problems like session stealing or session fixation. If I read that right, I believe thats what your doing.
The commercial (I think PERL) script that he (my client) purchased includes a session_key or sessionkey as one of the query arguments in many cases. It does time out after a while, and is not recognized after a logout.

[TJ111]

I'll try reading them in, because I think the javascript may already be writing them, now that I think about it. (Remember the monkey and the hammer?)

Wait, so are the cookies being created by java, or javascript?


The commercial (I think PERL) script that he (my client) purchased includes a session_key or sessionkey as one of the query arguments in many cases. It does time out after a while, and is not recognized after a logout.
Yes, but you'd be better off having the session key stored only as a cookie and not as a query at all. Not doing so opens up a risk of session fixation, which can be pretty dangerous if this site has to do with money transactions or personal info.

I'm a little confused as to exactly what you are trying to do with cookies. Here's what I think you want.
1. Cookies are set by java or javascript.
2. Read the cookies, then use stored values in future requests.

You'd be better off using sessions for this, since sessions are already being used. You can save anything you want as a session "variable", then call it in the future when you need it, eliminating the need for passing the value in requests, it will travel along with the user.

[TecBrat]

Wait, so are the cookies being created by java, or javascript?

I believe that would be javascript. Please forgive my unfamiliarity with the differences between the two.

I suppose one would say I have chosen a dirty, or messy, way of dealing with an issue. I am in completely unfamiliar territory here.

Maybe if I explain it a little better:
There is a site built either in Perl or something proprietary that acts a lot like Perl. classifieds.myclient.com. I do not have FTP access to this site. I want to develop at www.myclient.com. We use tables (I know, everyone says I should use CSS instead) and put an include() in the main area with a $_GET['mainpage']. One of the mainpage files will do a cURL of classifieds.myclient.com. I have been able to do everything I need on that site except these javascript functions. I even re-wrote the actionline in the forms to call the cURL document and post the data back to classifieds... The site, classifieds.myclient.com, uses a session, but I don't think it is a PHP session.

There are a couple of items on classifieds.myclient.com that use javascript. there is a savesearch who's link is like this: javascript:savesearch(link), with a relative path and querystring in place of "link" Here is the code from viewsource on the page in question. (This is from accessing classifieds.myclient.com directly, not by cURL)<script type="text/javascript">
//<![CDATA[

function savesearch(link) {

var link = link.replace(/;/g, "%3B");

//var link = escape(link);

var name = prompt("Please name this search", "Enter a name for this search here");

if (name==null) {
}

else {
function FixCookieDate (date) {
var base = new Date(0);
var skew = base.getTime(); // dawn of (Unix) time - should be 0
if (skew > 0) // Except on the Mac - ahead of its time
date.setTime (date.getTime() - skew);
}

var expdate = new Date ();
FixCookieDate (expdate); // Correct for Mac date bug - call only once for given Date object!
expdate.setTime (expdate.getTime() + (24 * 60 * 60 * 1000 * 30)); // 30 days from now

var old_ss = "";

//var old_ss = escape("");

var pointer = "savedsearches" + "=";
var stuff = old_ss + "{" + name + "|" + link + "}";

document.cookie = pointer + stuff + ";expires=" + expdate.toGMTString() + ";";

alert("You have saved this search as '" + name + "'. You can view all of your saved searches by clicking on the 'My Searches' link in the navigation bar.");
}
}
//]]>
</script>

(if you notice there is a string replace at the top of the script that replaces ";" with "^%3b" this is dealing with the query string in which the classified program uses semicolon instead of ampersand as a separator.)

I don't know all of what it is doing, but I get the idea of it. I can see that it is setting document.cookie. I need to make sure the next time cURL is called, the classifieds.myclient.com gets the contents of that cookie. I need to make sure that if various users from all over the world are on the site, their info is kept separate. Preferably, if they leave and come back, the system will recognize them and know what searches they previously saved.

I am not trying to be the type that expects others to do his work. I get frustrated with that type myself. I am just so out of my element here that I need a big kick in the posterior to get headed in the right direction. Thanks for trying already, and thanks for not giving up on me.

[TJ111]

Can you show me the part of the script that reads the cookies and creates the saved-searches list? (Not all of the part that creates the list, just the relevant parts). Or is that created by the perl/cgi?

[TecBrat]

Can you show me the part of the script that reads the cookies and creates the saved-searches list? (Not all of the part that creates the list, just the relevant parts). Or is that created by the perl/cgi?
That's done by the other script that may be perl/cgi. I don't have access to that source.

If you'd like (if you think it will help) I'll IM you the actual urls, I just don't like to post clients' urls in public forums. We wouldn't want my posts to show up in Google for their name.

[TJ111]

Send it to me in a private message on here and I'll check it out.

Do you know whether the problem is with the cookies being set, or with the server reading the cookies back and creating the saved searches?

[Webnerd]

cURL needs to track and store the cookies in it's cookie-jar

[TecBrat]

cURL needs to track and store the cookies in it's cookie-jar
I understand this, but the problem is that the cookies are set by javascript, after the page is loaded. I need to find a way to deal with that. I have sent a PM to TJ to have a look at the actual site.

[TJ111]

It seems that cURL doesn't have the ability to parse javascript-cookies, as they are created directly on the client without the use of http headers. If you have libcurl installed in php you can make a script that set's the cURL cookies, then use Ajax to send the cookie info to the sscript. It may not be the best method, but it'll hopefully do the trick.

Something like:

$ch = curl_init();
$cookie = urlencode(serialize($_GET['cookie_info']));
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_close($ch);
Then instead of setting the cookie with javascript, send the full contents of the cookie-to-be as a string in the $_GET part of an ajax call. Hope that makes sense.

I don't have much experience with cURL, so that may not be 100% correct, but I think its about right. Someone with more cURL experience can hopefulyl chime in about that idea.

[TecBrat]

I added these two snips to the the file that actually does the cURL request. Cookiejar and or Cookiefile wouldn't work because the javascript was writing the cookie to the end-user's computer.

$sscookie=$_COOKIE['savedsearches'];
and
if($sscookie){
// assumes $ch is already defined as a cURL handle
curl_setopt($ch, CURLOPT_COOKIE,'savedsearches='.$sscookie);
}
One might also try something like this in-case there are mulitple cookies

<?
// not yet tested
$somecookie='somecookie='.$_COOKIE['somecookie'];
$anothercookie='anothercookie'.$_COOKIE['anothercookie'];
$cookies=$somecookie.';'.$anothercookie;
// ...


// assumes $ch is already defined as a cURL handle
curl_setopt($ch, CURLOPT_COOKIE,$cookies);
?>

It worked for saved searches, now I need to work on it for a checkbox :)