Web Scraping: selecting correct XPath
I'm learning web scraping and I'm having a bit of trouble understanding the purpose of the XPath full length path. I'm using this python library: http://docs.webscraping.com/examples.html for the web scraping. It's a standard library that uses xpaths to select the individual html tags.
I'm confident my problem lies with my understanding of xpath not with the library it's self. I have no trouble selecting unique html tags by their id's like so:
This line fills the form input field of the id'fo-to' with the "BOG" value.
However, it becomes a problem when I attempt to select a submit button without a id, it does have a class but without a unique class so as I understand, I must reference the parents of the input field. Here's the input field:
At this point I thought the most logical thing to do was use a browser tool (in this case a extension for chrome called xpath helper) to select the absolute path of the button to quickly identify the button desired. The buttons absolute Xpath path is the following:
<div class="cb-span-5 cb-advanced">
<input type="submit" class="btn_alt" value="Search Now" alt="Search for Flight + Hotel" role="button"/>
This how ever doesn't work. The button will not be clicked by the web scraper and I cannot get it to work in inputting values for input fields. Is this not the common way of going about this? There must be a faster way to get the correct XPath than looking at source code and painstakingly writing it?
/html[@class='tmp_b_001 homepage ']/body/div[@id='ContentArea']/div[@id='HP']/div[@id='page-content']/div[@id='center-column']/div[@class='parsys par2']/div[@class='cmp_cnb_002 section']/div[@class='tvly_component']/div[@class='round cmp_cnb_002_container']/div[@class='cb520 cannonball cb-default-fo']/div[@class='cb-forms']/form[@id='cb-form-fo']/div/ul[@class='cb-rows last']/li[@class='cb-row']/div/div[@class='last cb-form-column']/input[@id='fo-from']
I'd greatly appreciate a tap in the correct direction.
Thank you for your time.