One common thing to want to do in the Yahoo Pipes YQL element is pass in a Pipes value to the YQL query. For example:
select * from html.tostring where url='<someurl>' and xpath='//div[#id="foo"]'
and you want to pass in a dynamic value for <someurl>. Let's say that it's an RSS feed item's URL called item.link. Attempting to simply replace the quoted someurl with item.link gives you this error:
Invalid identifier item.link. me is the only supported identifier in this context
How can I pass this value in?
You'll need to create a separate Pipe that takes the item URL as user input, passes it into a string builder which substitues the actual URL value into the YQL query string, and pass that as input to the YQL widget. Then in your main pipe, pass the item.link value as the input to your subpipe.
Specifically:
Create a User inputs -> URL Input item. You can give it any name and prompt. It's helpful to set the "Debug" value for testing.
Create a String builder, with 3 fields. In the first field, put the string up to where the value should be substituted. In the example in the question, select * from html.tostring where url='. In the second field, connect the output from the URL input element to here. In the third field, add the rest of the output: ' and xpath='//div[#id="foo"]'. When the string is built, it will be a complete YQL query string, with the provided URL substituted in.
Create a YQL element, and connect the output from the string builder to the query field.
Connect the YQL elements output to the Pipe Output element.
Save your new pipe.
In your main pipe, create an element of your new pipe (My Pipes -> whatever-you-named-it). Typically you'll be dragging it into a Loop element. Set the input to be item.link and you'll get the proper output.
Related
https://i.stack.imgur.com/k1bZD.png
I am using sys.person to capture the person's name. Is there any way for just 'john' and not the entire {"name": "john"} to show instead? I want to avoid sys.given-name and sys.last-name since dialogflow says they are deprecated.
Do you have your entity configured as a list? If it is a list, you probably have to output the parameter like $session.params.name[0].name.
If it's not a list, $session.params.name.name would work, but in that case I suggest you change the entity name to person, as it would be more readable to see $session.params.person.name
From your screenshot, it looks like your parameter is an isList parameter.
I have checked today the behavior of static responses when referencing isList parameters and it seems that we can now display the list of values for a certain parameter without the array index in the responses using the format: $session.params.parameter-id.
In your case, it would be:
$session.params.name.
In addition, if you want to reference a specific array index in an isList parameter, you can use the format $session.params.parameter-id[i]. In your case, it would be $session.params.name[0]
or if you want to get the original value you can reference it as $session.params.name[0].original
I am trying to extract data from a search result that is partially build via ajax:
https://www.vitalsana.com/catalogsearch/result/?q=ibuprofen
The wanted data PZN: 16336937 is somehow injected after page onload:
xpath does return an empty result:
//[#id="maincontent"]/div[3]/div[1]/div[2]/div[4]/ol/li[1]/form/div/div[2]/p[2]/span[2]/span
Sanme goes for the data verfügbar. It is loaded after pageload via this API I guess:
https://www.vitalsana.com/catalogsearch/searchTermsLog/save/?q=ibuprofen
I noticed that some info is within inline JS, but it is difficult to get just this JS. I tried last, but this seems to be ignored. It gets all JS including the desired info:
response.xpath('//script[last()]/text()').extract()
I am using scrapy 2.1.0. Is there a way to retrieve this data?
PZN: 16336937 is not present in the search results (Vitamin D3 != Ibuprofen).
To get the PZN number of a product (8 digits), you can extract it from the img element of each product. For example, for the first search result ([1]) :
response.xpath('substring(substring-before((//img[#class="product-image-photo img-fluid"])[1]/#src,"_"),string-length(substring-before((//img[#class="product-image-photo img-fluid"])[1]/#src,"_"))-7,8)').extract()
Output : 07728561
You could also extract the value directly from the script element, but you'll have to figure out how to escape single quotes in scrapy. The XPath :
substring-after(substring-before(//script[contains(.,"Suche")],'",'),'"id": "')
Output : 07728561
Note : using regex instead of substring functions might be cleaner.
What you could try also is to "rebuild" the json from the script element, load the json, then query on it. Something like this should work :
import json
products = response.xpath('substring(substring-after(//script[contains(.,"Suche")],"] ="),1,string-length(substring-after(//script[contains(.,"Suche")],"] ="))-1)').extract()
result = json.loads(products)
for i in result:
print i['id']
Last option : request directly the data from the API (with a well-formed payload, a valid token and the appropiate method).
I am trying to use '*' as the wildcard in the fileid parameter in the Files.get request in the google api's explorer, for the google-drive api.
I want to get all the files in a folder, I have a list of the id's but need the files_get to pull the file_id's as well so they can match.
I've tried inputing a start at the fileid parameter but I keep getting the error file not found.
I have left a screenshot of my results below.
screenshot of result
file.get
It looks like you are using the files.get method
Gets a file's metadata or content by ID.
The key word in the above method from the documentation is that it gets A file you need to pass the file id.
Fields
The parameters filed is used to for partial response. If you set it to name then only the name field will be returned in the response and the rest of the fields in the response will be null.
file.list
You should be using files.list you can use page size to return a max of 1000 rows per page after that you will need to request the next page of data.
pageSize integer The maximum number of files to return per page. Partial or empty result pages are possible even before the end of the files list has been reached. Acceptable values are 1 to 1000, inclusive. (Default: 100)
I am trying to pass a URL with a specific parameter having dynamic value for eg. email value will be
${__Random(0000,9999)}+$tester#testing.com.
This generate a random value+email id and get passed for request no.1.
I want to get that exact value for the email id parameter which get passed for that request and again pass that value to another request.
Use third parameter in __Random to save random value:
A reference name for reusing the value computed by this function.
${__Random(0000,9999, rnd)}+$tester#testing.com
And then use the variable concatenated with email suffix later:
${rnd}+$tester#testing.com
I am using SoapUI Pro 4.5.2 to read data from spreadsheets, put them into Soap requests to my web service, and get responses back to write to a spreadsheet. It's working.
I have two fields in the input data pertinent to my question:
Middle name is defined as a string of 4 characters, and as minOccurs = 1 and maxOccurs = 1.
Postal extension code (the 4-digit number that is optional after the normal 5-digit code) is defined as a string of 4 digits. This field is optional, so it is marked as minOccurs=0 and maxOccurs=1.
When I use the SoapUI UI interface to send a request, this works fine; if there's no value for middle name, SoapUI generates an empty tag and sends it (I guess because of the minOccurs=1). If there's no value for PostalCode, it does not send any tag at all (I guess because of minOccurs=0).
When SoapUI reads data from an Excel spreadsheet, however, the response to the same data is an error indicating that the extended postal code value of '' is not legal, because it must be 4 digits. It appears that SoapUI generates an empty tag for the extended postal code when reading data from the spreadsheet, and sends it.
I found the "Remove Empty Content" option for SoapUI requests, default to false. I set it to true, and now get an error back from validation indicating that middle name is required but not found. I'm guessing that the remove empty content removed all the empty content (reasonable enough), and middle name has to be there, even if empty, because of the minOccurs=1.
Do I have any way out of this tail-chasing problem? I suppose I'm looking for something like a conditional for the output of the postal extension code, so I can eliminate it if it's empty, even if reading values from the spreadsheet.
I am also curious if there are XSD fixes, but I greatly prefer a fix that doesn't involve changing the XSD -- that becomes a political matter.
EDIT FOR DETAIL:
To put input into the request: I have used the SoapUI UI to choose "properties" from the input spreadsheet for each of the input fields; when that's done, one ends up with values in the request fields like:
${SpreadsheetInput#FrstNm}
Where SpreadsheetInput is the name of the datasource step reading the spreadsheet, and FrstNm is one of the properties. I do this with the "Get Data" option off the popup menu you get by right-clicking the request input field, but there may be other ways.
So first your problem:
Remember that internally to SoapUI almost everything is a string. Doing something like:
<postCode>${SpreadsheetInput#PostCode}</postCode>
in your SOAP request, assuming PostCode is either blank or does not exist outright, will expand to:
<postCode></postCode>
and SoapUI will even optimize it to:
</postCode>
Then your validation kicks in, which says you do not need to provide this element, but if you do, it had better be 4-characters long. Which the above fails.
The solution:
You need to pragmatically (meaning you will have to write Groovy code) create this node in your request. There are several ways to handle this. The quick and dirty is with a Groovy step, that goes something like:
def postCode = context.expand('${SpreadsheetInput#PostCode}').trim()
if (postCode != null && postCode != '')
testRunner.testCase.setpropertyValue("postCodeNode", "<postCode>" + postCode + "</postCode>")
else
testRunner.testCase.setpropertyValue("postCodeNode", "")
Then in your request replace the original:
<postCode>${SpreadsheetInput#PostCode}</postCode>
with just:
${#TestCase#postCodeNode}
Notice, the XML node elements are part of the SoapUI property! Again: everything in SoapUI is just a plain string.
If you want something more hard-core, have a look at
dynamically create elements in a SoapUI request. This is mine.