I have one REST endpoint which returns a response like this
[
{
"id": "dbfff519-e8f6-4db3-9e26-a4e9014dc360",
"code": "123456789012345678901234567890123456789012345678901234567890",
"name": "client-code-with-character-length-sixty",
},
{
"id": "a673fb54-3503-4996-ba9b-a4e9014dc3ea",
"code": "18MTH",
"name": "18 Month",
},
{
"id": "60b781e3-4515-40f5-81ee-a4e9014dc400",
"code": "2periods",
"name": "I Have 2 Periods",
}...
etc.
I would like to be able to retrieve the ID where, for example, the code="2periods" UI can retrieve using either ResponseAsxml (with xpath) or Response with JSONPath. If I use the former I have managed to get the following to nearly retrieve my ID
//Response[1]/e[code='2periods']/id
BUT this looks like this
<id>0bc4aa5f-f8ab-4efe-b788-a4e9014dc45f</id>
And I don't know how to remove the start and end tags, the id has to be just the GUID.
I can't work out how to do something similar in the JSONPath - I've only managed to get something like
$[3].id
to work, but the order of the entities is not guaranteed in the test environment.
For ResponseAsXml using the XPath to get only the id value without <id> tag just add /text() to your actual expression:
//Response[1]/e[code='2periods']/id/text()
If you want to do the same using Response property with JSONPath try with:
$..[?(#.['code']=='2periods')].id[0]
Hope it helps,
Related
During developing pipeline which will use Elasticsearch as a source I faced with issue related paging. I am using SQL Elasticsearch API. Basically, I've started to do request in postman and it works well. The body of request looks following:
{
"query":"SELECT Id,name,ownership,modifiedDate FROM \"core\" ORDER BY Id",
"fetch_size": 20,
"cursor" : ""
}
After first run in response body it contains cursor string which is pointer to next page. If in postman I send the request and provide cursor value from previous request it return data for second page and so on. I am trying to archive the same result in Azure Data Factory. For this I using copy activity, which store response to Azure blob. Setup for source is following.
copy activity source configuration
This is expression for body
{
"query": "SELECT Id,name,ownership,modifiedDate FROM \"#{variables('TableName')}\" WHERE ORDER BY Id","fetch_size": #{variables('Rows')}, "cursor": ""
}
I have no idea how to correctly setup pagination rule. The pipeline works properly but only for the first request. I've tried to setup Headers.cursor and expression $.cursor but this setup leads to an infinite loop and pipeline fails with the Elasticsearch restriction.
I've also tried to read document at https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support but it seems pretty limited in terms of usage examples and difficult for understanding.
Could somebody help me understand how to build the pipeline with paging abilities utilization?
Responce with the cursor looks like:
{
"columns": [
{
"name": "companyId",
"type": "integer"
},
{
"name": "name",
"type": "text"
},
{
"name": "ownership",
"type": "keyword"
},
{
"name": "modifiedDate",
"type": "datetime"
}
],
"rows": [
[
2,
"mic Inc.",
"manufacture",
"2021-03-31T12:57:51.000Z"
]
],
"cursor": "g/WuAwFaAXNoRG5GMVpYSjVWR2hsYmtabGRHTm9BZ0FBQUFBRUp6VGxGbUpIZWxWaVMzcGhVWEJITUhkbmJsRlhlUzFtWjNjQUFBQUFCQ2MwNWhaaVIzcFZZa3Q2WVZGd1J6QjNaMjVSVjNrdFptZDP/////DwQBZgljb21wYW55SWQBCWNvbXBhbnlJZAEHaW50ZWdlcgAAAAFmBG5hbWUBBG5hbWUBBHRleHQAAAABZglvd25lcnNoaXABCW93bmVyc2hpcAEHa2V5d29yZAEAAAFmDG1vZGlmaWVkRGF0ZQEMbW9kaWZpZWREYXRlAQhkYXRldGltZQEAAAEP"
}
I finally find the solution, hopefully, it will be useful for the community.
Basically, what needs to be done it is split the solution into four steps.
Step 1 Make the first request as in the question description and stage file to blob.
Step 2 Read blob file and get the cursor value, set it to variable
Step 3 Keep requesting data with a changed body
{"cursor" : "#{variables('cursor')}" }
Pipeline looks like this:
pipeline
Configuration of pagination looks following
pagination . It is a workaround as the server ignores this header, but we need to have something which allows sending a request in loop.
I am using the composer to publish a bot to fetch data from an azure storage table.
In short, the bot composer needs to construct a bot to iterate through an XML deserialized JSON object returned by the azure storage rest API.
In my code generated by the composer, the bot does a "set property" step immediately following the successful return of the REST API (storage table query). Given the deserialized object returned by the storage REST API, how should the "set property" statement be constructed so the bot can print our the individual data field,
Another way to phrase the question: how can I use the composer to construct the bot to iterate through a returned deserialized object (coded in XML JSON format)?
Where can I find a document that can shed some light on this matter?
Is there any place I can find a good example? Can it be done via composer?
Thanks in advance.
Yes, it can be done. If the API returns XML, make sure you configure your api call to ask for content type application/xml.
Then you can use use the xPath built in function. Make note that it will return an array if results in more than value matches the expression, in which you can use the foreach function to iterate over it with. I needed to run the nightly build of Composer (with bot-builder 4.12.0) to get it to work for me. See here for some more info:
https://github.com/microsoft/botbuilder-js/pull/3093
Here's an example that worked for me:
"actions": [
{
"$kind": "Microsoft.SendActivity",
"$designer": {
"id": "rGv7XC"
},
"activity": "${SendActivity_rGv7XC()}"
},
{
"$kind": "Microsoft.HttpRequest",
"$designer": {
"id": "TDA1wO"
},
"method": "GET",
"url": "http://www.geoplugin.net/xml.gp?ip=157.54.54.128",
"resultProperty": "dialog.api_response",
"contentType": "application/xml"
},
{
"$kind": "Microsoft.SetProperty",
"$designer": {
"id": "ipNhfY"
},
"property": "dialog.timezone",
"value": "=xPath(dialog.api_response.content,'/geoPlugin/geoplugin_timezone/text()')"
},
{
"$kind": "Microsoft.SendActivity",
"$designer": {
"id": "DxohEx"
},
"activity": "${SendActivity_DxohEx()}"
}
]
You can (if needed/you wish) use the json and jPath built in functions to convert xml to json and then query with. Something like:
${json(user.testXml)} and then
${jPath(user.testJson , "automobiles")}
Is there an API call which will returns all recent public tweets regarding for specific location?
I tried GET trends/place but it's WOEID not worked for Sri Lanka and Cities.
(Sri Lanka WOEID is 23424778)
I don't want to use tweeter GET search/tweets.json endpoint because search based on certain key words.
Is there any solution for this?
Finally, I got the answer. We could not get trends on each and every location using tweeter API 1.1.
you have to check our closest trending place using this API call. For Austalia
https://api.twitter.com/1.1/trends/closest.json?lat=37.781157&long=-122.400612831116
the response like be this.
[
{
"country": "Australia",
"countryCode": "AU",
"name": "Australia",
"parentid": 1,
"placeType": {
"code": 12,
"name": "Country"
},
"url": "http://where.yahooapis.com/v1/place/23424748",
"woeid": 23424748
}
]
after that, you can use GET trends/place.
even though you cannot use GET trends/place for geo-based filtering you can use tweeter search endpoint. if you do not want to filter it by keyword ignore the q parameter and use geocodeparameter. for example,
https://api.twitter.com/1.1/search/tweets.json?result_type=recent&geocode=5.954920,80.554956,12mi
In here mi is miles.
I want to use MailChimp's API (v3.0) to get a list of my subscribers. Specifically, I want to only get subscribers with a first name of "Ken". This is the URL I've constructed to do that, but I'm not getting what I want. It returns a list of emails but they do not match my criteria.
Am I doing something wrong or is this not possible with 3.0?
https://us2.api.mailchimp.com/3.0/lists/<list_id>/members?members.merge_fields.FNAME=Ken
Here is the result I'd LIKE to get.
{
"members": [
{
"email_address": "kenjones#email.com",
"unique_email_id": "7cd36cf93c",
"status": "unsubscribed",
"merge_fields": {
"FNAME": "Ken",
"LNAME": "Jones"
}
]
}
Let's say I have an endpoint user/1/results and I want to upload multiple results at a time.
So I send it JSON like:
{
"data": [
{
"date": "2014-02-14 03:15:41",
"score": 18649,
"time": 42892
},
{
"date": "2013-11-18 09:21:46",
"score": 7856,
"time": 23568.8
}]
}
Let's say time needs to be an integer, so the second entity fails validation.
What's the best thing to do:
Fail both, nothing saves, respond with error message..
Save first entity, respond with error message.
In either case, what would an error message look like? i.e. how/does it specify that it's the second entity that fails validation.
I think you should fail both and respond with an error message because it might be cumbersome again to track the remaining results.
Error message should give the details of failing location. for example if it fails at the second one then specify it in json response.