Scrap Dynamic Content

[Nakul Sargur]

Hi all,

I am a newbie. I want to know how to extract/scrap dynamic content from web pages. I am able to get the static contents using Web harvest API. Thanks in advance for helping.

Regards,
Nakul Sargur

[JMRKER]

Why?

[Nakul Sargur]

I am doing a trial assignment on web scraping. I m a fresher. Using Web harvest (Java) API i am able to extract static content. But some data are enclosed inside javascript functions and html element. Need some guidance. Thanks in advance for helping.

[criterion9]

If the javascript is completing the content you'll need to parse through it to find out where the data is actually coming from (i.e. javascript variable, ajax call, etc).

[Nakul Sargur]

@criterion : the contents are present within html elements
For example :
<div id='tttt'> Hai </div>
<div id='zzzz'> Hello </div>
<li> <a ... > yyyyy </a> </li>
<li> <a ... onclick=return viewdetails('12345','yyyyyy','3435534','','rtefdg')> View </a> </li>

I am able to extract the contents "Hai" and "Hello". But i am unable to extract the contents "yyyyyy" ,"3435534". Because the contents are present within html tag. Currently i am using Web harvesting API for extracting the contents from website. This API gives the result after filtering the html elements. So that i am unable to extract html attributes value.

[criterion9]

@criterion : the contents are present within html elements
For example :
<div id='tttt'> Hai </div>
<div id='zzzz'> Hello </div>
<li> <a ... > yyyyy </a> </li>
<li> <a ... onclick=return viewdetails('12345','yyyyyy','3435534','','rtefdg')> View </a> </li>

I am able to extract the contents "Hai" and "Hello". But i am unable to extract the contents "yyyyyy" ,"3435534". Because the contents are present within html tag. Currently i am using Web harvesting API for extracting the contents from website. This API gives the result after filtering the html elements. So that i am unable to extract html attributes value.

What does the viewdetails function look like? Is it using AJAX?

[Nakul Sargur]

I think, there is no ajax call. Viewdetails function performs to display the parameter value (data) into new small window in that same page when you click view link in that page. The Viewdetails functions gets parameter value when the site loaded initially. I want to scrape the parameter value from site.

[criterion9]

I think, there is no ajax call. Viewdetails function performs to display the parameter value (data) into new small window in that same page when you click view link in that page. The Viewdetails functions gets parameter value when the site loaded initially. I want to scrape the parameter value from site.

Instead of guessing what might happen can you just post the function? I can guess at an answer or blindly suggest all sorts of things that may not help at all.

[Nakul Sargur]

<li id='tt2#10383004'><a onclick="new ContactDetailDiv('10383004','Alpine Housing','Pc7mCSeZzz4=','Builder','Syed Baseeruddin','3327','N','55.0 Lac(s)','Sale','/property-builder-details/developer-Alpine-Housing-in-Bangalore&operating-in=Bangalore&id=Pc7mCSeZzz4=?stdStatus=Y','/property-for-sale-rent/real-estate-builder-in-Bangalore-developer-Alpine-Housing&id=Pc7mCSeZzz4=');viewContactsDivDWR('10383004','9972595345','','91-80-32975001','search','property','false','N');_gaq.push(['_trackPageview', '/view-contact-details.html']);" href='javascript:void(0);' >View Contact Details</a></li>

[criterion9]

That is alot more detail...just need one more bit. The part that defines what "ContactDetailDiv()" does/is. Looks like it is an object.