I tried to scrape some data from this url:https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?zip=92612
I first debugged and tested in Scrapy shell. Since I just wanted the model name of the car, I copied the Xpath from Chrome. Here is how I did this:
I right clicked the name of model and clicked inspect
I found the text of model name and copied the Xpath
After I got the Xpath, I typed the command below:
response.xpath('//*[#id="cargurus-listing-search"]/div[1]/div/div[2]/div[2]/div[4]/div[1]/div/a/div[3]/div/div[1]/h4/text()').extract()
But the shell returned an empty list.
You could use the following to extract all the car names
response.xpath(//div[#class="titleWrap"]/h4/text()).extract()
Related
I'm trying to get the propierty of an HTML value using Bash. It's a personal project so I can use cURL, wget, everything, but needs to be bash.
Suppose I want to scrape Google to get the value of the jscontroller property that appears in the first div inside the body
Basically, how can I do it?
JHow do I grab this text here?
I am trying to grab the text here based on that the href contains "#faq-default".
I tried this first of all but it doesn't grab the text, only the actual href name, which is pointless:
//a/#href[contains(., '#faq-default-2')]
There will be many of these hrefs, such as default-2, default-3 so I need to do some kind of contains query, I'd guess?
You are selecting the #href node value instead of the a element value. So try this instead:
//a[contains(#href, '#faq-default-2')]
There are some data on this page :
$ scrapy shell "https://partsouq.com/en/catalog/genuine/unit?c=Toyota&ssd=%24HQwdcgcAAwFNa3YjVR92aVB7C10ZDko%24&vid=4463&cid=&uid=2535&q="
and there are numbers on the left hand-side of the page, After clicking on any one of them a table with contents appears like in the attachement, but after making "inspect element" on any item on this table, i get empty set !!
response.xpath('//*[#id="gf-result-table"]/tr[2]/td[2]/div').extract()
[ ]
this shows the tabe and the html code for it
You are giving wrong xpath. correct xpath is
response.xpath('//*[#id="gf-result-table"]/tbody/tr[2]/td[2]/div')
https://partsouq.com/en/search/search?q=0910112012&qty=1
this is the url of the attachement, the pop-up window is rendered by JavaScript, you can not do JS things in the scrapy.
And the xpath for the a tag is simple:
//a[#id]
I want to write an ImportXML function in a Google Spreadsheet to return the document name of the same spreadsheet. For example, my spreadsheet is titled "Kimchi". I want' to return that name in cell "A1" to automate a series of functions within the spreadsheet based on the document name. I'm too lazy to type the value into the cell for each of the hundred or so spreadsheets I'll copy from the original template and rename.
I can't seem to nail a correct query structure.
This bit of XML looked promising, but I can't seem to get the query to pull it:
<span class="docs-title" id="docs-title" role="button"><div class="docs-title-inner" id="docs-title-inner">kimchi</div></span>
I've tried so far...
=ImportXML("SOME URL HERE", "//div[#class=’docs-title-inner’]/#content")
It returns...
Error: Imported Xml content can not be parsed.
I've tried all kinds of variations, some probably equally poorly formed. Following is some of the XML structure that looked juicy:
<html>
<head>
<title>kimchi - Google Sheets</title>
But this XPath query within the ImportXML function didn't work either
=ImportXML("SOME URL HERE", "/html/head/title")
It returned...
Error: Import Internal Error.
I'm stumpted.
Here's the spreadsheet with variations.
PS This ended up working after I shared the document with the world:
=ImportXml("THE URL", "//meta[#itemprop='name']/#content")
You dont have to do any of that.
Go to tools-> script editor -> blank project
replace the contents of the edit window with the code below:
function BookName() {
return SpreadsheetApp.getActiveSpreadsheet().getName();
}
Ctrl-S, put BookName in the name box, click ok, wait for the yellow "saving" bar to dissapear. Close the tab with the code editor.
In your sheet you can now simply type =BookName() and the cell will display the workbook title.
Working on creating automated tests with selenium I expired problem when I try to locate text input by text inside it. I use Xpath and FirePath plugin for checking it's result.
My xpath is:
//input[#context='#name' and contains(#class, 'edit-string') and contains(#value, 'testSection')]
and it returns nothing, while when I remove last condition:
//input[#context='#name' and contains(#class, 'edit-string')]
i found required div.
As I undestand the problem is in that fact that value attribute doesn't contains typed text. Firebug console proves this theory(jQuery library present on the page):
>>>> $("input.edit-string")[0].getAttribute("value")
null
>>>> $("input.edit-string")[0].value
"testSection"
>>>> $("input.edit-string")[0].getAttribute("context")
"#name"
Any ideas how can I use typed text in xpath expression?
Please try it with following instructions..
Command ---> Target ---> Value
focus --->class=edit-string
typeKeys --->class=edit-string--->any text
Approach:
From the Given Info the XPath is
//input[#context='#name' and contains(#class, 'edit-string')]
Step 1:
So the Input is having the Class Name that contains the string "edit-string". We can convert the above XPath into CSS selector
css=.edit-string
Step 2:
Try getting the Text
$(".edit-string").getText();