Performing multiple requests using curl, with different options - bash

I'm trying to use curl to upload a file to sharepoint. I can do this successfully in three steps (ie. 3 separate invocations of curl to check the file out, upload it, and check it back in), using the suggestions in the following post:
How to check-out a file from sharepoint document library using curl?
My individual requests look like:
# Checkout the index.html file
curl --ntlm --user ${USER} \
--data #- \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckOutFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx << EOF
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<CheckOutFile xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<pageUrl>${FILE}</pageUrl>
<checkoutToLocal>false</checkoutToLocal>
<lastmodified/>
</CheckOutFile>
</soap:Body>
</soap:Envelope>
EOF
# upload the file
curl --ntlm -u ${USER} \
-T HTML/2015/index.html \
${FOLDER}
curl --ntlm --user ${USER} \
--data #- \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckInFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx << EOF
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<CheckInFile xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<pageUrl>${FILE}</pageUrl>
<comment>Automagic update</comment>
<checkinType>0</checkinType>
</CheckInFile>
</soap:Body>
</soap:Envelope>
EOF
Unfortunately, this results in cUrl asking me for my password 3 times (and it's a long password! :-) ). I also don't like the idea of a .netrc file, since writing passwords to disk is not a great idea.
So, what I thought I might be able to do is combine all of the requests into a single command line, setting and deleting headers as necessary, supplying the request bodies as appropriate using bash process substitution, etc.
curl --ntlm --user ${USER} \
--trace-ascii publish.log \
--data #<(echo "$CHECKOUT") \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckOutFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx \
-H "SOAPAction:" \
-H "Content-Type:" \
-T HTML/2015/index.html \
${FOLDER} \
--data #<(echo "$CHECKIN") \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckInFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx
Unfortunately, what happens is that cUrl seems to process all of the options at once, and only then attempt to request the URL's, resulting in options for one URL overwriting options for another URL, and ultimately nothing working. A snippet from the log file:
> 0000: PUT /xxx/xxx/_vti_bin/Lists.asmx HTTP/1.1
> 0033: Authorization: NTLM AAAAAAAAAAA=
> 0075: User-Agent: curl/7.30.0
> 008e: Host: example.com
> 00a8: Accept: */*
> 00b5: SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckOu
> 00f5: tFile
> 00fc: Content-Type: text/xml; charset=utf-8
> 0123: SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckIn
> 0163: File
> 0169: Content-Type: text/xml; charset=utf-8
> 0190: Content-Length: 0
> 01a3: Expect: 100-continue
Notice the duplicated SOAPAction header, while I was hoping to only have the first options applied.
Is there any way to say "stop processing options now, do this URL, then carry on"?

If typing password three times is your only concern, you can prompt for the password and read it in a variable and use it in curl command as like below.
echo "Password: "
read -s PASSWORD
# Checkout the index.html file
curl --ntlm --user ${USER}:${PASSWORD} \
--data #- \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckOutFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx << EOF
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<CheckOutFile xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<pageUrl>${FILE}</pageUrl>
<checkoutToLocal>false</checkoutToLocal>
<lastmodified/>
</CheckOutFile>
</soap:Body>
</soap:Envelope>
EOF
# upload the file
curl --ntlm -u ${USER}:${PASSWORD} \
-T HTML/2015/index.html \
${FOLDER}
curl --ntlm --user ${USER}:${PASSWORD} \
--data #- \
-H "SOAPAction: http://schemas.microsoft.com/sharepoint/soap/CheckInFile" \
-H "Content-Type: text/xml; charset=utf-8" \
${SHAREPOINT}/_vti_bin/Lists.asmx << EOF
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<CheckInFile xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<pageUrl>${FILE}</pageUrl>
<comment>Automagic update</comment>
<checkinType>2</checkinType>
</CheckInFile>
</soap:Body>
</soap:Envelope>
EOF

Related

Variable in command substitution is expanded into multiple arguments

I have tried to create a method in a bash script which should be able to perform a curl operation with a variable number of headers, however I seem to get stuck in that the curl command consider the header arguments to be multiple arguments as they contain spaces.
When I run the following line in bash I get a 201 response:
response=$($executable -X POST localhost:9200/index-template/globalmetadata --write-out '%{http_code}' --silent --output /dev/null --verbose --data "#${full_path}" -H "Content-Type: application/json" )
If I run the following:
#!/bin/bash
submit_request () {
full_path=/home/mat/globalmetadata.json
header_option=""
header_options=""
for header in "${#:1}"; do # looping over the elements of the $# array ($1, $2...)
header_option=$(printf " -H %s" "$header")
header_options=$(printf '%s%s' "$header_options" "$header_option")
done
echo Headers: $header_options
executable=curl
#response=$($executable -X POST localhost:9200/index-template/globalmetadata --write-out '%{http_code}' --silent --output /dev/null --verbose --data "#${full_path}" -H "Content-Type: application/json" )
response=$($executable -X POST localhost:9200/index-template/globalmetadata --write-out '%{http_code}' --silent --output /dev/null --verbose --data "#${full_path}" $header_option )
echo $response
}
submit_request "\"Content-Type: application/json\""
I get this output:
Headers: -H "Content-Type: application/json"
======= 3
* Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /index-template/globalmetadata HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 3232
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
} [3232 bytes data]
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 406 Not Acceptable
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 97
<
{ [97 bytes data]
* Connection #0 to host localhost left intact
* Could not resolve host: application
* Closing connection 1
406000
What I have noticed is that even though the headers are -H "Content-Type: application/json, curl says Could not resolve host: application. I suspect that it splits the arguments into two due to space between Content-Type: and application/json.
I tried to mix and match quotes and double quotes in all kinds of combination but nothing really worked.
#GordonDavisson is right, you'll have to put your header_options in an array. Just be aware that using the array "${header_options[#]}" when it is empty will result in an empty argument in your curl command, but You can get rid of this problem by storing the whole command in an other array.
submit_request () {
local -a header_options
for arg; do header_options+=(-H "$arg"); done
local -a curl_command=( \
curl \
-X POST \
localhost:9200/index-template/globalmetadata \
--write-out '%{http_code}' \
--silent \
--output /dev/null \
--verbose \
--data '#/home/mat/globalmetadata.json' \
"${header_options[#]}" \
)
local response=$( "${curl_command[#]}" )
printf '%s\n' "$response"
}

Loop through list for curl requests in bash

I have a bash script that sends a curl request and displays the response.
#!/bin/bash
token=$(curl -k -X GET \
'https://v.mytesting.io/oauth/token?grant_type=password&username=user1&password=123' \
-H 'Authorization: Basic 12345678' \
-H 'Host: v.mytesting.io.io')
v=$( jq -r ".access_token" <<<"$token" )
ts=$(curl -k -X POST \
https://timeseries.mytimeseries.io/v5/time_series/query \
-H 'Authorization: Bearer '"$v" \
-H 'Content-Type: application/json' \
-H 'Host: timeseries.mytimeseries.io' \
-H 'tenant: 123-123-123' \
-d '{"operation" : "raw","responseFormat" : "kairosDB","startTime": "1d-ago","stopTime": "now","tagList" : [ {"tagId" : "V.S.23164117.AVG.10M"}]}')
p=$(jq '.queries[].sample_size, .queries[].results[].name' <<<"$ts")
echo "$p"
My current output is just a value and the name of the tagId.
My query only allows for 1 tagId ( you can see above )
I want to be able to set a list of tagId's.
Then when I run this script it should loop through the list of tagId's and execute the curl request replacing the V.S.23164117.AVG.10M with each value
in the list.
Then output the entire list of results into a file.
list would be like so - (I would love to be able to enter this list into a seperate file and the bash script calls that file. Sometimes this list can be a few hundred lines.
V.S.23164117.AVG.10M
V.S.23164118.AVG.10M
V.S.23164119.AVG.10M
V.S.23164115.AVG.10M
V.S.23164114.AVG.10M
output would like look so.
value tagId
value tagId
value tagId
100 V.S.23164117.AVG.10M
etc..
thank you for any help
You can loop over list of tags using a small script. I'm not 100% clean of the output format. You can change the 'echo' to match the required format.
Note minor change to quotes to allow variable expansion in the body.
The tags will be stored in a file, for examples, tags.txt
V.S.23164117.AVG.10M
V.S.23164118.AVG.10M
V.S.23164119.AVG.10M
And the script will be use the file
#! /bin/bash
# Use user defined list of tags
tags=tags.txt
token=$(curl -k -X GET \
'https://v.mytesting.io/oauth/token?grant_type=password&username=user1&password=123' \
-H 'Authorization: Basic 12345678' \
-H 'Host: v.mytesting.io.io')
v=$( jq -r ".access_token" <<<"$token" )
for tag in $(<$tags) ; do
ts=$(curl -k -X POST \
https://timeseries.mytimeseries.io/v5/time_series/query \
-H 'Authorization: Bearer '"$v" \
-H 'Content-Type: application/json' \
-H 'Host: timeseries.mytimeseries.io' \
-H 'tenant: 123-123-123' \
-d '{"operation" : "raw","responseFormat" : "kairosDB","startTime": "1d-ago","stopTime": "now","tagList" : [ {"tagId" : "'"$tag"'"}]}')
p=$(jq '.queries[].sample_size, .queries[].results[].name' <<<"$ts")
echo "$tag $p"
done

Using bombardier for server benchmarking - but can't add body parameters

So, I'm using bombardier on MacOS Sierra
for server benchmarking from command line.
In this example i'm using 1 connection and 1 request, and two headers:
"Authorization" and "Content-Type" and body: "{isTemplate:1}"
But server does not receiving body.
./bombardier -c 1 -n 1 -m PATCH -H "Authorization: Bearer MYBEARERGOESHERE" -H "Content-Type: application/x-www-form-urlencoded" -b "{isTemplate:1}" http://localhost:8082/presentation/6525/update
I tried:
-b "{isTemplate:1}"
-b "isTemplate:1"
Any thoughts?
Just found the solution, - you need to write body variables like this:
-b "isTemplate=1"
So final request:
./bombardier -c 1 -n 1 -m PATCH -H "Authorization: Bearer MYBEARERGOESHERE" -H "Content-Type: application/x-www-form-urlencoded" -b "isTemplate=1" http://localhost:8082/presentation/6525/update

Curl POST call for server API which uploads a file

I want to translate a series of POSTMAN calls into bash in order to create a script. Super easy till now where I want to POST an xlsx file with roles with form-data.I use this script:
curl -i -X POST \
-H 'externalTenantId: 326c1027-bf20-4cd6-ac83-33581c50249b' \
-H "uid: user" \
-H "Content-Type: multipart/form-data" \
-F 'payload={
"importMode": "OVERWRITE",
"tenantId": "326c1027-bf20-4cd6-ac83-33581c50249b",
"file": "roles.xlsx"
}' \
-F 'file=#roles.xlsx' \
"http://server:8080/iamsvc/batchImport/v2/direct/roles"
This is the postman call which works:
POST http://server:8080/iamsvc/batchImport/v2/direct/roles
Headers:
uid: user#domain.com
externalTenantId: 4cd6-ac83-33581c50249b-327522
Payload:
{
"file": [Excel file to be uploaded],
"importMode": "OVERWRITE",
"tenantId": "4cd6-ac83-33581c50249b-327522"
}
This is the error that I get:
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=0BA814182C258E1DFE62ACF98409F9CD; Path=/iamsvc/;
Secure; HttpOnly
Content-Length: 0
Date: Mon, 26 Sep 2016 12:59:50 GMT
Connection: close
The answer was in curl --manual and it's working like this:
curl -i -X POST -H "uid: user" -H "externalTenantId: 326c1027-bf20-4cd6-ac83-33581c50249b" -F "file=#/home/user/zscripts/iamapi/roles.xlsx" -F "importMode=OVERWRITE" -F "tenantId=326c1027-bf20-4cd6-ac83-33581c50249b" http://server:8080/iamsvc/batchImport/v2/direct/roles

How to programmatically set username and password on Geoserver at intall time

I am writing a bash script to install Geoserver by following the steps outlined here Install Instructions . I am trying to use CURL to post a custom data store config file. But I am having an authentication error.
* upload completely sent off: 42out of 42 bytes
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< Set-Cookie: SPRING_SECURITY_REMEMBER_ME_COOKIE=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/geoserver
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="GeoServer Realm"
< Content-Type: text/html;charset=utf-8
< Content-Length: 1299
< Date: Thu, 31 Jan 2013 05:16:17 GMT
I belive it is becuase I haven't set the username and password for Geoserver. The only way I can seem to find is to do it via the web admin interface. I would like to set it via my bash script.
Is there a way to achieve this?
Bash script section here
echo 'Downdloading geoserver'
wget http://downloads.sourceforge.net/project/geoserver/GeoServer/2.2.4/geoserver-2.2.4-war.zip
unzip geoserver-2.2.4-war.zip
sudo cp geoserver.war /var/lib/tomcat7/webapps/
sleep 120
echo 'setting up geoserver'
curl -u $U_NAME:$PASSWORD -v -XPOST -H 'Content-type: text/xml' \
-d '<workspace><name>catami</name></workspace>' \
http://localhost:8080/geoserver/rest/workspaces ;
curl -u $U_NAME:$PASSWORD -XPOST -T datastore-config.xml -H 'Content-type: text/xml' \
http://localhost:8080/geoserver/rest/workspaces/catami/datastores ;
curl -u $U_NAME:$PASSWORD -XPOST -H 'Content-type: text/xml' \
-d '<featureType><name>Force_image</name></featureType>' \
http://localhost:8080/geoserver/rest/workspaces/catami/datastores/CatamiImagePoints/featuretypes ;
curl -u $U_NAME:$PASSWORD -XPOST -H 'Content-type: application/vnd.ogc.sld+xml' \
-d #catami-colour-by-depth.sld http://localhost:8080/geoserver/rest/styles ;
curl -u $U_NAME:$PASSWORD -XPUT -H 'Content-type: text/xml' \
-d '<layer><defaultStyle><name>catami-colour-by-depth</name></defaultStyle><enabled>true</enabled></layer>' \
http://localhost:8080/geoserver/rest/layers/catami:Force_image
Apparently you can edit the username and password inside the GeoServer Data Directory. You can read more about it here.
And after you found what file to change what it, then you just need to sed that data to your liking.

Resources