I'm trying to run an elasticsearch query with scrolling. Here's what I'm trying to do, but it is not working. I'm following the instructions here: https://www.elastic.co/guide/en/elasticsearch/reference/6.7/search-request-scroll.html
$ curl -u $(whoami) -XPOST 'https://myhost.io/elasticsearch/_msearch?scroll=1m' --data-binary #query.json -H "kbn-version:6.7.1" | jq
Enter host password for user '<first.last>':
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 973 0 96 100 877 321 2933 --:--:-- --:--:-- --:--:-- 3254
{
"statusCode": 500,
"error": "Internal Server Error",
"message": "An internal server error occurred"
}
Does anyone know why this doesn't work? If I remove the '?scroll=1m', this query works just fine.
You just need to expand the size of the scroll which is set probably to a smaller size
Related
Hi, I am trying to create a Jira issue through Jenkins on a windows slave. The console output is not showing any error, however, the Jira issue is not getting created. Below is the code:
pipeline {
agent { label 'windows'}
stages {
stage('Build') {
steps {
bat script {"""curl -u ${jira_username}:${jira_password} -X POST -H 'Content- Type:application/json' -d '{"fields":{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":{"key":"KEY"},"summary":"summary","description":"description","issuetype":{"name":"Test"}}}' http://localhost:8080/rest/api/2/issue/"""}
}
}
}
}
The console output is:
D:\workspace\TestJob>curl -u username:password -X POST -H 'Content-Type:application/json' -d
'{"fields":{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":
{"key":"KEY"},"summary":"summary","description":"description","issuetype":{"name":"Test"}}}'
http://localhost:8080/rest/api/2/issue/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 136 0 0 100 136 0 323 --:--:-- --:--:-- --:--:-- 326
It doesn't throw any errors but the issue is also not getting created.
However, The same code if I run on Linux slave then I get the below response and the issue is getting created.
+ curl -u 'username:password' -X POST -H Content-Type:application/json -d '{"fields":
{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":
{"key":"KEY"},"summary":"summary","description":"description","issuetype":
{"name":"Test"}}}' http://localhost:8080/rest/api/2/issue/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 255 0 87 100 168 115 223 --:--:-- --:--:-- --:--:-- 223
100 255 0 87 100 168 115 223 --:--:-- --:--:-- --:--:-- 223
{"id":"648","key":"KEY-35","self":"http://localhost:8080/rest/api/2/issue/648"}
As you can see the issue is getting created on Linux but not on windows. Requesting help to resolve this issue.
Resolved the issue
Firstly 'Content-Type: application/json' should be in double-quotes.
Putting it in single quote 'Content-Type: application/json' was actually giving 415 error code which I got to know after using -k -D- in the curl command
Secondly, I had to use double backslash in the data.
The command which actually worked is:
bat script {"""curl -u username:password --header "Content-Type:application/json" -X POST --data "{\\"fields\\":{\\"project\\":{\\"key\\":\\"KEY\\"},\\"summary\\":\\"summary\\",\\"description\\":\\"description\\",\\"issuetype\\":{\\"name\\":\\"Test\\"},\\"components\\":[{\\"id\\":\\"1\\"}],\\"fixVersions\\":[{\\"id\\":\\"2\\"}]}}" http://localhost:8080/rest/api/2/issue/"""}
I run following script in gitlab's job:
after-merge-to-release:
script:
- set -x
- >
curl\
--X POST --header 'PRIVATE-TOKEN:xyz' --header 'Content-Type:application/json'\
--data '{"id":"'$PROJECT_ID'","name":"'${MOST_RECENT_SPRINT_BRANCH//'origin/'/}'","allowed_to_push":[{"access_level":0}],"allowed_to_merge":[{"access_level":0}]}'\
https://git.xyz.net/api/v4/projects/"$PROJECT_ID"/protected_branches
I can confirm this command works (it adds a new protected branch to my project), but gitlab's output doesn't confirm it:
$ curl\ # collapsed multi-line command
++ echo '$ curl\ # collapsed multi-line command'
++ curl --X POST --header PRIVATE-TOKEN:xyz --header Content-Type:application/json --data '{"id":"123","name":"origin/sprint/48","allowed_to_push":[{"access_level":0}],"allowed_to_merge":[{"access_level":0}]}' https://git.xyz.net/api/v4/projects/123/protected_branches
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: POST
100 530 100 413 100 117 3152 893 --:--:-- --:--:-- --:--:-- 4045
Why it doesn't return the response from curl request and throws Could not resolve host: POST error?
The error is here:
--X POST
Use only a single dash before X to select the http method:
-X POST
I am writing a small bash script that gets the profile of a Minecraft user from their username. However, when I run the following script, the profile_json variable comes out blank while the uuid_json and uuid variables have the content they should.
#!/bin/bash
# settings
username="Notch"
# script
uuid_json=$(curl https://api.mojang.com/users/profiles/minecraft/$username)
echo $uuid_json
uuid=$(jq '.id' <<< "$uuid_json")
echo $uuid
profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid)
echo $profile_json
Moreover, the output from running the script in a terminal shows the Current Speed of profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid) as 0. Any ideas as to why this would be?
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 56 100 56 0 0 160 0 --:--:-- --:--:-- --:--:-- 160
{"name":"Notch","id":"069a79f444e94726a5befca90e38aaf5"}
"069a79f444e94726a5befca90e38aaf5"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
AS you can see by executing echo $uuid you got "069a79f444e94726a5befca90e38aaf5"
instead of 069a79f444e94726a5befca90e38aaf5 there should be no quotes.
jq has option to remove them it's -r with that everything works as expected(I think).
So change it to:
#!/bin/bash
# settings
username="Notch"
# script
uuid_json=$(curl https://api.mojang.com/users/profiles/minecraft/$username)
echo $uuid_json
uuid=$(jq -r '.id' <<< "$uuid_json") echo $uuid
profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid)
echo $profile_json
Your echo $uuid outputs this:
"069a79f444e94726a5befca90e38aaf5"
But it really should be:
069a79f444e94726a5befca90e38aaf5
for the curl command to work. So, tweak your jq line to strip off quotation marks.
I am using the Qualtrics API to retrieve survey data at scheduled intervals. Below is my shell script (bash) get_responses.sh which posts the export, measures the download completion rate, gets the export, and stores/unzips the file.
STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"
result=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
"surveyId": "SV_000000000000",
"startDate": "startDate": "'"$STARTDATESTRING"'",
"endDate": "endDate": "'"$ENDDATESTRING"'",
"format": "csv",
"useLocalTime": true,
"useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")
es_id=$(echo "$result" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')
curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}"
curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"
unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA/"
I intermittently get the following error-
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 339 100 133 100 206 165 256 --:--:-- --:--:-- --:--:-- 256
{"result":{"percentComplete":0.0,"file":null,"status":"in progress"},"meta":{"httpStatus":"200 - OK","requestId":"2c55524c-03aa-495c-8de7-b54d5b441b34"}} % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 129 100 129 0 0 405 0 --:--:-- --:--:-- --:--:-- 405
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile
directory in one of /Users/myname/Desktop/response.zip or
/Users/myname/Desktop/WEA/response.zip.zip, and cannot find /Users/myname/Desktop/WEA/response.zip.ZIP, period.
The odd part is that this error usually only occurs when I first run the script. If I rerun it, without making any changes, it will go through with no errors. To my understanding this error message means a corrupted zip file. Am I corrupting the POST request by saving it into a variable? I need to capture the post request output somehow as it provides the es_id that I need for the subsequent GET request. I would hard code it but the es_id refreshes weekly.
Using #T.Gibbons advice I was able to eliminate the error on the initial run by incorporating a while loop into the script.
STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"
post_response=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
"surveyId": "SV_00000000000000",
"startDate": "'"$STARTDATESTRING"'",
"endDate": "'"$ENDDATESTRING"'",
"format": "csv",
"useLocalTime": true,
"useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")
es_id=$(echo "$post_response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')
percent_complete=0
while [ $percent_complete -ne 100 ]
do
response=$(curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}")
percent_complete=$(echo "$response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.percentComplete')
done
curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"
unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA"
rm "/Users/myname/Desktop/WEA/response.zip"
Here's an example ip proxy checker shell script:
#!/bin/sh
while read IP
do
CURL=$(curl -x http://$IP -L http://icanhazip.com)
echo "$CURL"
done < ip.txt
But instead of a simple result like:
0.0.0.0
1.1.1.1
I get:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0.0.0.0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
1.1.1.1
How can I quiet the extra stuff?
-s/--silent
Silent mode. Don't show progress meter or error messages. Makes Curl mute.
If this option is used twice, the second will again disable mute.
CURL=$(curl -s -x http://$IP -L http://icanhazip.com)