PUT vs POST when adding documents in elastic search - elasticsearch

I am new to Elasticsearch and trying to add documents in elastic index. I have got confused between PUT and POST here as both are producing same results in below scenario:
curl -H "Content-Type: application/json" -XPUT "localhost:9200/products/mobiles/1?pretty" -d"
{
"name": "iPhone 7",
"camera": "12MP",
"storage": "256GB",
"display": "4.7inch",
"battery": "1,960mAh",
"reviews": ["Incredibly happy after having used it for one week", "Best iPhone so far", "Very expensive, stick to Android"]
}
"
vs
curl -H "Content-Type: application/json" -XPOST "localhost:9200/products/mobiles/1?pretty" -d"
{
"name": "iPhone 7",
"camera": "12MP",
"storage": "256GB",
"display": "4.7inch",
"battery": "1,960mAh",
"reviews": ["Incredibly happy after having used it for one week", "Best iPhone so far", "Very expensive, stick to Android"]
}
"

POST :used to achieve auto-generation of ids.
PUT :used when you want to specify an id.
see this

They both are among safe methods of HTTP.
usually we use POST to create a resource and PUT to modify that. besides if you're free to set-up the server side, you can use both of them because they both have similar properties like: they both have body, they are safe, data is not shown in URL, and ....
though it is better to consider standard rules that I said one of them before:
usually we use POST to create a resource and PUT to modify that. this way your code is more readable, changeable ...
for going deeper you can consider these tips according to put-versus-post:
Deciding between POST and PUT is easy: use PUT if and only if the endpoint will follow these 2 rules:
The endpoint must be idempotent: so safe to redo the request over and over again;
The URI must be the address to the resource being updated.
When we use PUT, we’re saying that we want the resource that we’re sending in our request to be stored at the given URI. We’re literally “putting” the resource at this address.

The only different between POST and PUT is that you cannot use PUT to create documents with auto ID generation.
The following query will create a document and auto generate an ID:
POST /products/_doc
{
"name": "Shoes",
"price": 100,
"in_stock": 64
}
Trying the same with PUT results to an "Incorrect HTTP method".
PUT /products/_doc
{
"name": "Shoes",
"price": 100,
"in_stock": 64
}
Unless I didn't experiment hard enough, this is the only difference between POST and PUT when creating documents.
Other than this, POST and PUT will get you to achieve the same things.

Related

LinkToken with account filters - assets/investment/brokerage - not working

I am currently trying to retrieve the link token using a curl command in my terminal. I ultimately want to obtain the user's investment data in my app so I decided to use the assets product with account_filter of investment and subtype brokerage. However, this is not working.
Here is the request I am using (minus the IDs):
curl -X POST https://sandbox.plaid.com/link/token/create -H 'Content-Type: application/json' -d '{
"client_id": "...",
"secret": "...",
"user": { "client_user_id": "unique-per-user"
},
"client_name": "...",
"products": ["assets"],
"country_codes": ["US"],
"language": "en",
"redirect_uri": "...",
"account_filters": {
"investment": {
"account_subtypes": ["brokerage"]
}
}
}'
I then get the error:
{
"display_message": null,
"documentation_url": "https://plaid.com/docs/#create-link-token",
"error_code": "INVALID_FIELD",
"error_message": "account_filters must be nonempty Map\u003caccount type, Map\u003c\"account_subtypes\", Array\u003caccount subtype\u003e\u003e\u003e where account type and account subtype are strings. There must also be at least 1 valid product and account type combination.",
"error_type": "INVALID_REQUEST",
"request_id": "...",
"suggested_action": null
}%
UPDATE: I've confirmed with the Assets team that appears to be a bug / unintended behavior, they are looking into it.
I'm not entirely sure why, but it does seem like the Assets product does not support using account filters for anything other than depository accounts. Note that the Assets API is designed for lending use cases and gets a snapshot of a single point in time of a user's account balances and transactions. If you're looking to understand someone's investments, you in most cases probably want to use the Plaid Investments API, which allows you to view what a user is invested in, has automatic account filtering for investments accounts built in, and will keep you up-to-date with a user's current investments.

Perspective API: Proper way to send requests with auto-detection of language

I am a bit confused on the proper way to send requests using Google's Perspective API.
Sending the following request works:
{"comment":{"text":"yo hamburger"},"languages":["en"],"requestedAttributes":{"TOXICITY":{}}}
In the documentation, it says, "...If you are using a production attribute, language is auto-detected if not specified in the request." So, I tried:
{"comment":{"text":"yo hamburger"},"requestedAttributes":{"TOXICITY":{}}}
And in response, I got a HTTP/1.0 400 Bad Request.
I also tried including all of the languages listed on the documentation page, like this:
{"comment":{"text":"yo hamburger"},"languages":["en","fr","es","de","it","pt"],"requestedAttributes":{"TOXICITY":{}}}
But that also gave me a response of HTTP/1.0 400 Bad Request.
Another attempt was made leaving the array of languages empty, like this:
{"comment":{"text":"yo hamburger"},"languages":[],"requestedAttributes":{"TOXICITY":{}}}
However, it still gave me a response of HTTP/1.0 400 Bad Request.
I was wondering, what is the proper way to send a request to the API and have it auto-detect language?
User x00 provided the path to the solution in the question's comment section. By using curl, I was able to see what was going on.
Here's what was happening:
In this first example, the system worked without error.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages: ["en"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"attributeScores": {
"TOXICITY": {
"spanScores": [
{
"begin": 0,
"end": 12,
"score": {
"value": 0.050692778,
"type": "PROBABILITY"
}
}
],
"summaryScore": {
"value": 0.050692778,
"type": "PROBABILITY"
}
}
},
"languages": [
"en"
],
"detectedLanguages": [
"tr",
"ja-Latn",
"de",
"en"
]
}
In this second example, the system was indeed auto-detecting language, but since "yo hamburger" was detected as Turkish, it could not provide a solution and instead sent a 400 as the response code.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: tr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"detectedLanguages": [
"tr"
],
"attribute": "TOXICITY"
}
}
]
}
}
This next example is more mysterious to me, as the language field for the request is plural, "languages," so it seems you can provide more than one language. However, it said it couldn't support that.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages:["en","fr","es","de","it","pt"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: en,fr,es,de,it,pt",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"requestedLanguages": [
"en",
"fr",
"es",
"de",
"it"
],
"attribute": "TOXICITY"
}
}
]
}
}
In this next example, leaving the languages array empty also provided the auto-detection of language, but again, "yo hamburger" was detected as Turkish, so it could not provide a response.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages:[],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: tr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"detectedLanguages": [
"tr"
],
"attribute": "TOXICITY"
}
}
]
}
}
Noticing that Perspective API would not allow me to choose all of the languages that are provided for the TOXICITY report, I decided to try two languages. The response was the same. Apparently Perspective API rejects the request if multiple languages are specified. Perhaps naming the field "languages" was a thought for the future.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages: ["en","fr"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: en,fr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"requestedLanguages": [
"en",
"fr"
],
"attribute": "TOXICITY"
}
}
]
}
}
Maybe you're using bad client library or other issue is causing the problem, Here is documentation about client library that in example language is auto-detected without problem. Check that and if not successful provide more details for further investigations.
As I said in the comments, general approach to these kind of issues: use curl. It helps a lot.
To sum up you findings:
auto-detection with a set of languages doesn't seem to work.
the correct way to send a request with auto-detection enabled is
{comment: {text: "some text"}, requestedAttributes: {TOXICITY:{}} }
but sometimes it fails on short texts, especially with slang inside.
So what can be done about it?
The easyest way is to assign some weight to Bad Requests (probably something around 0.5). Anyway, as a response you get the probability and not a definitive answer. So
toxicity score = 1 means "definitely toxic"
toxicity score = 0 means "not toxic at all"
and toxicity score = 0.5 means "we have no idea"
same thing goes for Bad Request - "you have no idea"
and you will get 0.5 from time to time, so you must deal somehow with comments of that score anyway. As well as with network errors etc.
But I would say that a probability of toxicity of a comment that result in LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE is higher than 0.5. But it's up to you to decide on the exact number.
As auto-detection doesn't work well with short texts you can bump up probability of correct auto-detection by adding some context into you request: a couple of other comments in the thread, or better yet, a couple of other comments from the same user. Not too big ones and not too small ones.
Make three requests specifying a language. As far as I can tell TOXICITY works only with English, Spanish, and French. On github I've got this reply:
"TOXICITY is currently supported in English (en), Spanish (es), French (fr), German (de), Portuguese (pt), and Italian (it). We will work to remove the contradictions you identified."
Auto-detect by yourself before sending a request. That'll require some effort, but it shouldn't be too hard, given you have much more context available to you than is available to Perspective API (or any other third-party API)
Also
These kind of APIs are not supposed to stay unattended. Fine tuning and moderation on your part is required. Or else we'll end up in the worst-case scenario of algocracy :).
And I think it's a good idea in general to store statistics of toxicity of comments for a user... as well as some manual coefficient. Because for example: Mathematical formulas give high toxicity
I've posted an couple of issues on github, but no reply yet (whating for reply on the second issue). When/If I'll get them I'll update my answer with details.

Is pagination for Youtube API Channel Memberships (sponsors.list) broken?

I'm trying to paginate through a list of results using the youtube API for Channel Memberships (sponsors.list), but the paging and PageTokens don't seem to be working as they are supposed to.
I'm currently developing an application for a user to generate a list of all Members to their channel (using the api for sponsors.list: https://developers.google.com/youtube/v3/live/docs/sponsors/list)
I have a test account, and I've been able to successfully pull the list. However, the test account only has 5 memberships. Since the API can only pull a maximum of 50 results per page, I want to make sure that my app can account for the possibility that the channel will have 50+ sponsors.
So, I've set the results per page to give me just 1, theoretically giving me 5 pages I can then sift through to simulate 50+ members.
The problem arises when I try to page through the results... as the API says, I grab the nextPageToken from the results, and pass it in the next call in the pageToken parameter. However, when I do so, even when testing in the API explorer, I get back an empty list, and no nextPageToken for the next page.
{
"kind": "youtube#sponsorListResponse",
"etag": "\"XpPGQXPnxQJhLgs6enD_n8JR4Qk/UCSC321uKOiUT6GNkcPmkqoH1sY\"",
"pageInfo": {
"totalResults": 0,
"resultsPerPage": 0
},
"items": []
}
Additionally, if I pass a fake pageToken, the results come as if I'd passed no token at all, so it is at least recognizing the nextPageToken I'm passing it.
My google searches have failed me, other than just turning up pages talking about how the pagination is supposed to work... which it obviously isn't. Am I doing something wrong? Or is it indeed broken?
Edit
Here are the API calls I made.
Initial member list pull (After getting the authorization token, etc).
'https://www.googleapis.com/youtube/v3/sponsors?part=snippet&filter=all&maxResults=1' \
--header 'Authorization: Bearer [SECRET_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
Which results in: (I've edited out sensitive info, like [CHANNEL_ID], etc).
{
"kind": "youtube#sponsorListResponse",
"etag": "\"XpPGQXPnxQJhLgs6enD_n8JR4Qk/PRgb6wjx--gdhgTtZ1auDKOony0\"",
"nextPageToken": "GLiawvDS6uEC",
"pageInfo": {
"totalResults": 5,
"resultsPerPage": 1
},
"items": [
{
"kind": "youtube#sponsor",
"etag": "\"XpPGQXPnxQJhLgs6enD_n8JR4Qk/LoD6jhrr94l_4soca-7lx14kyRQ\"",
"snippet": {
"channelId": "[CHANNEL_ID]",
"sponsorDetails": {
"channelId": "[CHANNEL_ID]",
"channelUrl": "[CHANNEL_URL]",
"displayName": "[DISPLAY_NAME]",
"profileImageUrl": "[PROFILE_IMAGE_URL]"
},
"sponsorSince": "2019-04-25T06:36:11.677Z"
}
}
]
}
So I grab the nextPageToken "GLiawvDS6uEC", and drop that into my next call in the pageToken field, as the API instructs.
'https://www.googleapis.com/youtube/v3/sponsors?part=snippet&filter=all&maxResults=1&pageToken=GLiawvDS6uEC' \
--header 'Authorization: Bearer [SECRET_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
And wind up with this depressing result:
{
"kind": "youtube#sponsorListResponse",
"etag": "\"XpPGQXPnxQJhLgs6enD_n8JR4Qk/UCSC321uKOiUT6GNkcPmkqoH1sY\"",
"pageInfo": {
"totalResults": 0,
"resultsPerPage": 0
},
"items": []
}
So, turns out this was an actual problem with the API. Had a friend who knew someone at google, they looked into it, got the problem fixed! It works as intended now! Yay!
That said, if I hadn't had that connection, who knows if this would ever have been solved ;_;
As far as I can see, a nextPageToken of value GLiawvDS6uEC is invalid.
All the page tokens I came across were of a pattern described e.g. by Youtube Data API v3 pageToken for arbitrary page.
The API's documentation itself says nothing about how a page token should look like!
Maybe someone else has better inside on this issue. In any case, I suggest to file a report with Google.

Last Login Date to Application

I am looking for an API to find the user's last login date to a particular application. Not sure if it exists, please let me know your thoughts on getting this information through API calls.
You can get the last date the user was authenticated to any application on the User profile:
GET https://dev-12345.oktapreview.com/api/v1/users/xyz
{
"id": "xyz",
"status": "ACTIVE",
"created": "2017-02-22T02:33:02.000Z",
"activated": null,
"statusChanged": "2017-02-22T17:50:43.000Z",
"lastLogin": "2017-09-28T16:19:57.000Z",
"lastUpdated": "2017-08-08T20:11:46.000Z",
"passwordChanged": "2017-02-22T17:50:43.000Z",
"profile": {
"lastName": "Barbettini",
"firstName": "Nathanael",
// ...
}
The lastLogin field contains the timestamp of the last time the user logged in.
This isn't tracked on a per-app basis, though. You can get the app-specific App User:
GET https://dev-12345.oktapreview.com/api/v1/apps/abc/users/
{
"created": "2017-09-22T00:54:49.000Z",
"lastUpdated": "2017-09-22T00:54:49.000Z",
"scope": "GROUP",
"status": "ACTIVE",
"statusChanged": "2017-09-22T00:54:49.000Z",
"passwordChanged": null,
"syncState": "DISABLED",
"profile": {
"zoneinfo": "America/Los_Angeles",
"region": null,
// ...
}
But it doesn't have lastLogin. One way you could solve this is by saving a timestamp in the App User profile in your application code.
You could use filters to specify which logs do you want from the system logs API endpoint, limit the answer to 1, set the response order to start from the most recent ones, and user filters again to filter for your user.
So, you need to filter the request using the following filters:
actor.alternateId: which identifies the user, in my case the pattern "username#" is enough
event_type: this one should be user.authentication.sso which means that the user SSO to an app (any of the ones available for that user)
outcome.result: success, which means that the user successfully SSO to the app
Example query would be something like this:
curl -X GET -H "Content-Type: application/json" -H "Authorization: SSWS super-secret-token-here" -H "Accept: application/json" "https://<your.domain.here.com>/api/v1/logs?until=2018-07-17T11%3A30%3A55.664541&limit=1&sortOrder=DESCENDING&filter=event_type+eq+%22user.authentication.sso%22+and+outcome.result+eq+%22SUCCESS%22+and+actor.alternateId+sw+%22<user name here>%40%22"
Remember to replace the token, the URL and the before testing this query.
Also, remember that this example is URL encoded, so this is why it looks so weird.

Pull public event data from Google Calendar

I may be over thinking this a bit. On my web site, I would like to user certain data from my public google calendar. My plan is to pull it on the server side so I can do things like process it, cache it and format it the way I want.
I've been looking at using the Google Api libraries, but I can't get past any of the authorization hurdles. A service account sounds like what I really want, but I'm having trouble wrapping my head around how that works in this situation.
The old GDATA apis would be ok, but I'm not very keen on using them because they look fairly deprecated at this point by the newer libraries.
Since it is all public data, I'm hoping there is a simpler way to get to the event data that I'm looking for.
In case it matters, my site is Asp.Net (MVC).
Update
Ok, I was definitely way over thinking it. See my answer.
Now that RSS has been removed from Google Calendar, I've been in search of an easy replacement. I dug around and found the following in the Google Calendar API that seems to do the trick: calendar.events.list
Calendar Events List in Google API Explorer is a good place to get started with the different parameters and options - and it'll build you an example request string. You can see that I specified a minimum time of 2/5/2016, sort it by the start time, and show deleted events.
GET https://www.googleapis.com/calendar/v3/calendars/[CALENDAR ID HERE]/events?
orderBy=startTime&showDeleted=true&singleEvents=true&
timeMin=2016-02-05T00%3A00%3A00Z&key={YOUR_API_KEY}
Results are in JSON so you can parse it in your favorite programming language, ASP.NET or whatever. Result looks like:
{
"kind": "calendar#events",
"etag": "\"123456789123456\"",
"summary": "My Public Calendar",
"updated": "2016-01-29T14:38:29.392Z",
"timeZone": "America/New_York",
"accessRole": "reader",
"defaultReminders": [ ],
"items": [ {
"kind": "calendar#event",
"etag": "\"9876543210987654\"",
"id": "sfdljgsdkjgheakrht4sfdjfscd",
"status": "confirmed",
"htmlLink": "https://www.google.com/calendar/event?eid=sdgtukhysrih489759sdkjfhwseihty7934hyt94hdorujt3q95uy689u9yhfdgnbiwe5hy",
"created": "2015-07-06T16:21:59.000Z",
"updated": "2015-07-06T16:21:59.329Z",
"summary": "In-Service Day",
"location": "Maui, HI",
"creator": {
"email": "abra#cadabra.com",
"displayName": "Joe Abra"
},
"organizer": {
"email": "cadabra.com_sejhrgtuiwerghwi4hruh#group.calendar.google.com",
"displayName": "My Public Calendar",
"self": true
},
"start": {
"date": "2016-02-08"
},
"end": {
"date": "2016-02-09"
},
"transparency": "transparent",
"iCalUID": "isdt56y784g78ty86tgsfdgh#google.com",
"sequence": 0
},
{
...
}]
}
One good answer to this (the one I'm going with) is to simply use the calendar's public address to get the data. This is an option that I had forgotten about and it works fine for this particular situation.
You can find the url for the data if you go to the settings for a particular calendar and pick the format you want (I went with xml for this situation.)
The data that you get out of this service is very human-reader optimized, but I can make it work for what I'm doing.

Resources