I am using Watson's Speech-To-Text Lite service and I am trying to find a way to automate the loading of new audio files to transcribe. I am very new to Bash and so I'm unclear of even the more rudimentary terms - so I'm finding the problem hard to find a solution for.
For a single use-case, I run the following file (my API key omitted with 'MY APIKEY')
curl -X POST -u "apikey: MY APIKEY" --header "Content-Type: audio/flac" --data-binary "#audiofile_1.flac" "https://gateway-lon.watsonplatform.net/speech-to-text/api/v1/recognize?model=en-US_BroadbandModel&speaker_labels=true" > C:/Users/outputpath/output_1.txt
What I am essentially trying to achieve is to overcome having to manually type and retype the names of the audio files and output. So if I had three (or more) audio files (i.e. audiofile_1, 2, and 3.flac), i would like to create an output file corresponding to each audio file - Some psuedo-code that might help explain what I mean would be
files = [file_1, file_2, file_3]
for file_x in files:
run curl command
save as output_x
You almost got it. You just need to learn some shell syntax:
files=("file_1" "file_2" "file_3")
for file_x in "${files[#]}"
do
curl -X POST -u "apikey: MY APIKEY" --header "Content-Type: audio/flac" --data-binary "#${file_x}" "https://gateway-lon.watsonplatform.net/speech-to-text/api/v1/recognize?model=en-US_BroadbandModel&speaker_labels=true" > "C:/Users/outputpath/${file_x}.txt"
done
First you create the files array with your list of files. Then, you iterate over those files and run the curl command on each of them.
Trying to lunch a job workflow via REST API and passing extra variables for the playbook to consume, but returned body shows that provided variables are put in ignored_fields section.
Used POSTMAN and CURL to run the templates both returned the same result
CURL command
curl -X POST http://172.16.0.97/api/v2/job_templates/8/launch/ -H "Content-Type: application/json" -H "Authorization: Bearer Je
gxwfQrdKQXoRUtNWtWFz62FX5bTy" -d "{\"extra_vars\": {\"vendor\":\"juniper\"}}"
Returned body
{"job":34,"ignored_fields":{"extra_vars":{"vendor":"juniper"}},"id":34,"type":"job","url":"/api/v2/jobs/34/","related":{"created_by":"/api/v2/users/1/","modified_by":"/api/v2/users/1/","labels":"/api/v2/jobs/34/labels/","inventory":"/api/v2/inventories/1/","project":"/api/v2/projects/7/","extra_credentials":"/api/v2/jobs/34/extra_credentials/","credentials":"/api/v2/jobs/34/credentials/","unified_job_template":"/api/v2/job_templates/8/","stdout":"/api/v2/jobs/34/stdout/","job_events":"/api/v2/jobs/34/job_events/","job_host_summaries":"/api/v2/jobs/34/job_host_summaries/","activity_stream":"/api/v2/jobs/34/activity_stream/","notifications":"/api/v2/jobs/34/notifications/","job_template":"/api/v2/job_templates/8/","cancel":"/api/v2/jobs/34/cancel/","create_schedule":"/api/v2/jobs/34/create_schedule/","relaunch":"/api/v2/jobs/34/relaunch/"},"summary_fields":{"inventory":{"id":1,"name":"Demo Inventory","description":"","has_active_failures":true,"total_hosts":1,"hosts_with_active_failures":1,"total_groups":0,"groups_with_active_failures":0,"has_inventory_sources":false,"total_inventory_sources":0,"inventory_sources_with_failures":0,"organization_id":1,"kind":""},"project":{"id":7,"name":"Cox-Phase3","description":"","status":"successful","scm_type":"git"},"job_template":{"id":8,"name":"Port Flap","description":""},"unified_job_template":{"id":8,"name":"Port Flap","description":"","unified_job_type":"job"},"created_by":{"id":1,"username":"admin","first_name":"","last_name":""},"modified_by":{"id":1,"username":"admin","first_name":"","last_name":""},"user_capabilities":{"delete":true,"start":true},"labels":{"count":0,"results":[]},"extra_credentials":[],"credentials":[]},"created":"2019-05-14T09:43:16.115516Z","modified":"2019-05-14T09:43:16.177517Z","name":"Port Flap","description":"","job_type":"run","inventory":1,"project":7,"playbook":"main.yml","forks":0,"limit":"","verbosity":1,"extra_vars":"{}","job_tags":"","force_handlers":false,"skip_tags":"","start_at_task":"","timeout":0,"use_fact_cache":false,"unified_job_template":8,"launch_type":"manual","status":"pending","failed":false,"started":null,"finished":null,"elapsed":0.0,"job_args":"","job_cwd":"","job_env":{},"job_explanation":"","execution_node":"","controller_node":"","result_traceback":"","event_processing_finished":false,"job_template":8,"passwords_needed_to_start":[],"ask_diff_mode_on_launch":false,"ask_variables_on_launch":false,"ask_limit_on_launch":false,"ask_tags_on_launch":false,"ask_skip_tags_on_launch":false,"ask_job_type_on_launch":false,"ask_verbosity_on_launch":false,"ask_inventory_on_launch":false,"ask_credential_on_launch":false,"allow_simultaneous":false,"artifacts":{},"scm_revision":"","instance_group":null,"diff_mode":false,"job_slice_number":0,"job_slice_count":1,"credential":null,"vault_credential":null}
According to the fine manual, AWX (and thus Tower) version 3.0 and greater has made extra_vars more strict: https://docs.ansible.com/ansible-tower/latest/html/userguide/job_templates.html#ug-jobtemplates-extravars
If you are running a version greater than 3.0, you will need to either turn on playbook survey or set ask_variables_on_launch=True for that template
In my case, I'm using curl -L ... and the payload got lost after the redirect. Be sure to double check that if you find the extra_vars still gets ignored after ensuring ask_variables_on_launch=True.
Tangentially related to the API when utilizing the AWX and Tower CLI I ran into a similar issue of variables not being taken when launching jobs. The solution was that on the Job Template in Tower the "Prompt on Launch" setting needed to checked for the variable to pass through. So much time wasted on such a simple miss.
I made a script that downloads several files located in my professional OneDrive. This script works perfectly from a French computer, a US computer but it can't work from a Japanese computer.
To permit you understand the problem, I will detail the program:
1- I establish the token system (I got inspired by Jay Lee detailed answer) and retrieve the token in the access_token variable.
2- To download the file, in my case I cannot use
curl -w %{time_total} https://graph.microsoft.com/v1.0/me/drive/items/01M...WU/content -H "Authorization: Bearer $access_token"
Thus, this how I proceed:
#I get the item properties
itemProperties=$(curl ${ODf1Mb} -H "Authorization: Bearer $access_token")
#In these properties I select the downloadUrl that will permit me to download the file
downloadUrl=$(echo -e "$itemProperties" | grep "#microsoft.graph.downloadUrl" | awk -F'[",]' '{ print $9 }')
#Finally I execute this URL storing the download time in a variable (I do all this stuff for this)
dload=$(curl -w %{time_total} ${downloadUrl} -H "Authorization: Bearer $access_token")
As I said at the begin, for French and US computers it will work but on the Japanese machine it doesn't. I do get the itemProperties and the downloadUrl but when I call the downloadUrl with CURL it seems that it cannot reach the server because I have this:
As we can see we do not even have the Total weight to be downloaded. As an element of comparison, this is the result in a French machine:
I know, there is a warning relating to command substitution but I haven't tried to fix it yet because it makes its job.
Note -> the downloadUrl has this format:
https://lpl-my.sharepoint.com/personal/{user}_{company infra domain}_com/_layouts/15/download.aspx?
I just cannot figure out what is the problem. I can access to the https://lpl-my.sharepoint.com through the browser so I don't think the server IP is banned.
Check your ping / traceroute to see if lpl-my.sharepoint.com resolves to the same network location.
Also, I have seen other folks run curl with -v to see verbose traces and see if what the difference is.
Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters
Having a many-to-many relationship between users and groups. I would like to know how to update this relationship with SDR. This is what I've tried so far after reading the docs.
curl -X POST -H 'Content-Type: text/uri-list' -d 'http://localhost:8080/rest/users/5' http://localhost:8080/rest/groups/1/users
Expected result: Add user 5 to group 1.
Actual result: 405 Method Not Allowed.
curl -X PUT -H 'Content-Type: text/uri-list' -d 'http://localhost:8080/rest/users/5' http://localhost:8080/rest/groups/1/users
Expected result: Replace all members of group 1 with user 5.
Actual result: Works as expected.
curl -X PUT -H 'Content-Type: text/uri-list' -d #members.txt http://localhost:8080/rest/groups/1/users
Where the file members.txt has:
http://localhost:8080/rest/users/5
http://localhost:8080/rest/users/6
http://localhost:8080/rest/users/7
Expected result: Replace all members of group 1 with the users 5, 6 and 7.
Actual result: Only last user (in this case 7) gets added.
Could someone provide an example on how to ADD a single URI to an association?. Also if possible, how to add or replace an association with multiple URIs?
After re-reading the documentation, it does indeed say POST should add to the collection.
My experience has been to use PATCH to add to the collection.
To further the answer: You should be able to use PUT CONTENT-TYPE: text/uri-list with a content body having multiple URIs. Each URI is separated by a line break "\n"
Try this:
curl -v -X POST -H "Content-Type: text/uri-list" -d "http://localhost:8080/rest/users/5" http://localhost:8080/rest/groups/1/users