AutoML Vision: Error: ImportData stopped due to too many errors - google-cloud-automl

I selected to create a dataset in the AutoML Vision UI like so:
I get this: Error: ImportData stopped due to too many errors.
Notice that it tells me that I have errors in my csv file, but it does not tell me what errors are, so how can I debug?
I tried this:
>cat operation-get-status.sh
#!/bin/bash
if [ $# -ne 1 ]; then
printf "usage: get-training-status.sh [operation-id]\n"
exit 1
fi
export OPERATION_ID="$1"
#export OPERATION_ID="ICN21697762462531584"
source set-vars.sh
curl -X GET \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
https://automl.googleapis.com/v1beta1/projects/$PROJECT_ID/locations/$REGION_NAME/operations/operation-id
But the response is not very helpful:
>./operation-get-status.sh IOD2963854538949263360
{
"error": {
"code": 400,
"message": "List of found errors:\t1.Field: name; Message: Required field is invalid\t",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"field": "name",
"description": "Required field is invalid"
}
]
}
]
}
}
Which required field is invalid?

I go the same error on my GUI with uploaded CSVs. Tried to upload a single annotation from one video. Got the following error:
Cannot move files across regions. Please use a regional bucket in the
same location and with same storage class as AutoML. Required
Location: us-central1, required location type: Region, required
storage class: Standard.
After changing the type of bucket to the recommended settings, the upload was successful.

Related

Laravel validation error message show only short form

I got used to request to validate before entering the API side function itself, but recently, when I moved the controller function to services then, the error message became like the below sample.
{
"message": "validation.min.string",
"errors": {
"password": [
"validation.min.string"
]
}
}
But when I use the same way build in the new project, the error message show as usual as below
{
"message": "The name must only contain letters.",
"errors": {
"name": [
"The name must only contain letters."
]
}
}
I found that I had added an en language file under res resources\lang\en, which caused this issue, so may I know how I can change back to this version without removing the en file? Thanks

laptop curl Youtube API v3 request always returns 'Exceeded Quota' error

Every time I attempt to use the following simple bash script command on my Macbook Pro laptop, I get an 'Exceeded Quota' error. I do supply a valid API key and I'm under the impression that I don't need any other tokens because I am only requesting Public information (a simple list of songs).
Bash script:
#!/bin/bash
curl \
'https://youtube.googleapis.com/youtube/v3/search?part=snippet&q=music&type=video&key=[My API key]' \
--header 'Accept: application/json' \
--compressed
Reply:
{
"error": {
"code": 403,
"message": "The request cannot be completed because you have exceeded your \u003ca href=\"/youtube/v3/getting-started#quota\"\u003equota\u003c/a\u003e.",
"errors": [
{
"message": "The request cannot be completed because you have exceeded your \u003ca href=\"/youtube/v3/getting-started#quota\"\u003equota\u003c/a\u003e.",
"domain": "youtube.quota",
"reason": "quotaExceeded"
}
]
}
}
I don't have any quota problems pulling song lists via the YouTube Data API 'list' page:
https://developers.google.com/youtube/v3/docs/search/list

Perspective API: Proper way to send requests with auto-detection of language

I am a bit confused on the proper way to send requests using Google's Perspective API.
Sending the following request works:
{"comment":{"text":"yo hamburger"},"languages":["en"],"requestedAttributes":{"TOXICITY":{}}}
In the documentation, it says, "...If you are using a production attribute, language is auto-detected if not specified in the request." So, I tried:
{"comment":{"text":"yo hamburger"},"requestedAttributes":{"TOXICITY":{}}}
And in response, I got a HTTP/1.0 400 Bad Request.
I also tried including all of the languages listed on the documentation page, like this:
{"comment":{"text":"yo hamburger"},"languages":["en","fr","es","de","it","pt"],"requestedAttributes":{"TOXICITY":{}}}
But that also gave me a response of HTTP/1.0 400 Bad Request.
Another attempt was made leaving the array of languages empty, like this:
{"comment":{"text":"yo hamburger"},"languages":[],"requestedAttributes":{"TOXICITY":{}}}
However, it still gave me a response of HTTP/1.0 400 Bad Request.
I was wondering, what is the proper way to send a request to the API and have it auto-detect language?
User x00 provided the path to the solution in the question's comment section. By using curl, I was able to see what was going on.
Here's what was happening:
In this first example, the system worked without error.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages: ["en"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"attributeScores": {
"TOXICITY": {
"spanScores": [
{
"begin": 0,
"end": 12,
"score": {
"value": 0.050692778,
"type": "PROBABILITY"
}
}
],
"summaryScore": {
"value": 0.050692778,
"type": "PROBABILITY"
}
}
},
"languages": [
"en"
],
"detectedLanguages": [
"tr",
"ja-Latn",
"de",
"en"
]
}
In this second example, the system was indeed auto-detecting language, but since "yo hamburger" was detected as Turkish, it could not provide a solution and instead sent a 400 as the response code.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: tr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"detectedLanguages": [
"tr"
],
"attribute": "TOXICITY"
}
}
]
}
}
This next example is more mysterious to me, as the language field for the request is plural, "languages," so it seems you can provide more than one language. However, it said it couldn't support that.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages:["en","fr","es","de","it","pt"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: en,fr,es,de,it,pt",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"requestedLanguages": [
"en",
"fr",
"es",
"de",
"it"
],
"attribute": "TOXICITY"
}
}
]
}
}
In this next example, leaving the languages array empty also provided the auto-detection of language, but again, "yo hamburger" was detected as Turkish, so it could not provide a response.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages:[],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: tr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"detectedLanguages": [
"tr"
],
"attribute": "TOXICITY"
}
}
]
}
}
Noticing that Perspective API would not allow me to choose all of the languages that are provided for the TOXICITY report, I decided to try two languages. The response was the same. Apparently Perspective API rejects the request if multiple languages are specified. Perhaps naming the field "languages" was a thought for the future.
CURL:
curl -H "Content-Type: application/json" --data \
'{comment: {text: "yo hamburger"},
languages: ["en","fr"],
requestedAttributes: {TOXICITY:{}} }' \
https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=[API_KEY]
RESPONSE:
{
"error": {
"code": 400,
"message": "Attribute TOXICITY does not support request languages: en,fr",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.commentanalyzer.v1alpha1.Error",
"errorType": "LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE",
"languageNotSupportedByAttributeError": {
"requestedLanguages": [
"en",
"fr"
],
"attribute": "TOXICITY"
}
}
]
}
}
Maybe you're using bad client library or other issue is causing the problem, Here is documentation about client library that in example language is auto-detected without problem. Check that and if not successful provide more details for further investigations.
As I said in the comments, general approach to these kind of issues: use curl. It helps a lot.
To sum up you findings:
auto-detection with a set of languages doesn't seem to work.
the correct way to send a request with auto-detection enabled is
{comment: {text: "some text"}, requestedAttributes: {TOXICITY:{}} }
but sometimes it fails on short texts, especially with slang inside.
So what can be done about it?
The easyest way is to assign some weight to Bad Requests (probably something around 0.5). Anyway, as a response you get the probability and not a definitive answer. So
toxicity score = 1 means "definitely toxic"
toxicity score = 0 means "not toxic at all"
and toxicity score = 0.5 means "we have no idea"
same thing goes for Bad Request - "you have no idea"
and you will get 0.5 from time to time, so you must deal somehow with comments of that score anyway. As well as with network errors etc.
But I would say that a probability of toxicity of a comment that result in LANGUAGE_NOT_SUPPORTED_BY_ATTRIBUTE is higher than 0.5. But it's up to you to decide on the exact number.
As auto-detection doesn't work well with short texts you can bump up probability of correct auto-detection by adding some context into you request: a couple of other comments in the thread, or better yet, a couple of other comments from the same user. Not too big ones and not too small ones.
Make three requests specifying a language. As far as I can tell TOXICITY works only with English, Spanish, and French. On github I've got this reply:
"TOXICITY is currently supported in English (en), Spanish (es), French (fr), German (de), Portuguese (pt), and Italian (it). We will work to remove the contradictions you identified."
Auto-detect by yourself before sending a request. That'll require some effort, but it shouldn't be too hard, given you have much more context available to you than is available to Perspective API (or any other third-party API)
Also
These kind of APIs are not supposed to stay unattended. Fine tuning and moderation on your part is required. Or else we'll end up in the worst-case scenario of algocracy :).
And I think it's a good idea in general to store statistics of toxicity of comments for a user... as well as some manual coefficient. Because for example: Mathematical formulas give high toxicity
I've posted an couple of issues on github, but no reply yet (whating for reply on the second issue). When/If I'll get them I'll update my answer with details.

Model is not deployed and cannot predit

On Google Cloud Automl - Language Classification, I put in the following request for prediction.
I am getting - The model is not deployed, hence not supported for prediction yet.
How do we deploy a model ? I do not see any documentation if deploying a model
I am running this command shown in code in Google Cloud Shell
curl -X POST -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" -H "Content-Type: application/json" https://automl.googleapis.com/v1beta1/projects/[projectid]/locations/us-central1/models/[Modelname]:predict -d '{
"payload" : {
"textSnippet": {
"content": "YOUR TEXT HERE",
"mime_type": "text/plain"
},
}
}'
{
"error": {
"code": 400,
"message": "The model is not deployed, hence not supported for prediction yet.",
"status": "FAILED_PRECONDITION"
}
}
Expected prediction
You can deploy a successfully trained model using the UI on GCP.
Under HOME go to your respective AI and find your model in its dashboard.

Google Genomics API Callsets Search "Unknown field name: datasetIds"

I've been using the Google Genomics API for about a day now. I've successfully called many of the APIs like Datasets.list, Datasets.get, and even Readsets.search but I'm having a problem with the Callsets.search.
I'm making POST request to:
POST https://www.googleapis.com/genomics/v1beta/callsets/search?key=MY_KEY_HERE
And my request body is:
{
"datasetIds" : [
"376902546192"
]
}
But the response I'm getting back is:
{
"error" : {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Unknown field name: datasetIds",
"locationType": "other",
"location": ""
}
],
"code": 400,
"message": "Unknown field name: datasetIds"
}
}
According to the documentation: https://developers.google.com/genomics/v1beta/reference/callsets/search datasetIds is a perfectly valid parameter.
The crazy thing that's perplexing me is this identical request works just fine on the readsets/search endpoint but not the callsets/search endpoint? I'm almost wondering if it's a bug in the API. Can anyone help?
Received this from Google:
The variants and callsets APIs just went
through some breaking changes so that they'll be compliant with GA4GH
v0.5 when they go fully public.
All breaking changes should be done now - and I'll try to get all the
docs and code samples updated today or tomorrow.
Until then, you can see the real parameters in the API explorer (it
can't lie :) - in this case, the datasetId field has now changed to
"variantSetIds" (still using that same value, just a rename)
I just tested it, and it works. Below are the results:
$ java -jar target/genomics-tools-client-java-v1beta.jar searchcallsets --dataset_id 376902546192
Getting call sets from: 1000 Genomes
{"created":"1410541777431","id":"376902546192-0","name":"HG00345","sampleId":"HG00345","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-1","name":"HG00369","sampleId":"HG00369","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-2","name":"HG01085","sampleId":"HG01085","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-3","name":"HG01107","sampleId":"HG01107","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-4","name":"NA12347","sampleId":"NA12347","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-5","name":"NA18579","sampleId":"NA18579","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-6","name":"HG00372","sampleId":"HG00372","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-7","name":"HG01134","sampleId":"HG01134","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-8","name":"NA18532","sampleId":"NA18532","variantSetIds":["376902546192"]}
{"created":"1410541777431","id":"376902546192-9","name":"NA18597","sampleId":"NA18597","variantSetIds":["376902546192"]}
Hope it helps,
Paul

Resources