I'm building a lambda function that needs to receive several parameters, including eventually files.
curl -v -H "Authorization: blah" -F "index_name=test" -F "document=#./spec/fixtures/config/indexes/video.json" -X POST https://blah.execute-api.us-east-1.amazonaws.com/dev/index_document
I can see from inside my event['body'] that I'm receiving my data, but I can't find a way to parse the data and using Rack feels a bit weird.
<--------------------------298624d5b63e53bf
Content-Disposition: form-data; name="index_name"
test
--------------------------298624d5b63e53bf
Content-Disposition: form-data; name="document"; filename="video.json"
Content-Type: application/octet-stream
{
"settings": {
"index": {
"number_of_shards": 5
}, ...
Is there any other way to have my data back in an orderly fashion, like:
>>> event['body']['index_name']
"test"
I would recommend you have a look at the curb ruby gem
Related
I am writing shell scripts in macOS 11.1 Big Sur to test several endpoints thru cUrl request. To improve my scripts, I would love to do the following:
send the curl request
store the response code to a variable in an array
store part of the response payload to another variable
I can not share the exact endpoint since that would be a security breach for others to test their solutions, unfortunately.
In example, using a modified request from one of my scripts:
#!/bin/sh
EMAIL_USERNAME=`date "+%Y%m%d%H%M%S"`
curl -iX POST 'http://test.someclientcloud.com/api/online/customer/register' -H 'Connection: keep-alive' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Authorization: XXXXXXXXXXXXXXX' -H 'Accept: application/json' -H 'X-Origin-Source: WWW' -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.9,pt;q=0.8' --data-binary $'{ "emailAddress": "'$EMAIL_USERNAME'#xxxxxx.com", "firstName": "Terminate", "lastName": "ME", "opt": true, "password": "Password7654!", "phoneNumber": "555229'$TODAY'", "regCity": "Anytown", "regCountryCode": "1", "regState": "TX", "regZip": "86753", "username": "'$EMAIL_USERNAME'#xxxxx.com" }'
The curl response is:
HTTP/1.1 201 Created
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: close
Date: Wed, 06 Jan 2021 18:40:38 GMT
Pragma: no-cache
Transfer-Encoding: chunked
Location: /api/customer/354190751
Expires: 0
requestDuration: 67
X-Frame-Options: DENY
X-Origin-Source: WWW
X-Content-Type-Options: nosniff
proxyUser: WWW
X-XSS-Protection: 1; mode=block
X-Application-Context: application:production
The two things I would like from that response are 201 for the response code, from the first line of the response.
And 354190751 from the end of line 7 of the response above.
I can use these responses to form a report at the end of the script of each response per request. And I can use the newly created customer number in further tests within the script to modify the account further or delete the account for clean up at the end of the test.
I suspect this can be done with clever regex? My regex level is only moderately dangerous.
To be clear, I will mention, this is Apple Shell Scripting in macOS 11.1 Big Sur. Variants of shell scripting are many.
Hopefully the question was clear, thank you for your help.
Added that this is macOS 11.1 Big Sur
Unfortunately, while curl lets you ask it to directly write the status code, that doesn't extend to individual headers -- so you need to do the pattern matching yourself. One way to do that is as follows (note the use of process substitution syntax to avoid the issues discussed in BashFAQ #24, which makes this incompatible with /bin/sh; that said, because your code uses ANSI C-like string syntax, aka $'', it isn't compliant with the sh standard regardless):
#!/usr/bin/env bash
# put your actual curl code here; using a stub here so others can test
getCurlOutput() {
cat <<'EOF'
HTTP/1.1 201 Created
Transfer-Encoding: chunked
Location: /api/customer/354190751
Expires: 0
EOF
}
code=
customer=
{
read _ code _
prefix="Location: /api/customer/"
while IFS= read -r line; do line=${line%$'\r'}
case $line in
"$prefix"*) customer=${line#$prefix};;
esac
done
} < <(getCurlOutput)
echo "Retrieved exit status $code"
if [ -n "$customer" ]; then
echo "Customer id retrieved is $customer"
fi
I am hitting a wall trying to build a script to save myself quite a good bit of time. I am working in a system in which I need to run a curl POST against a list of values. The list is about 400 lines long, so I am hoping to find a way of scripting this in Bash instead of running that call manually for each entry. Below are some details to help understand what I'm trying to accomplish:
If I were to be doing this task manually, each call would be formatted like the below:
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d #json_payload.json
This points to my JSON in the listed file which shows as the below:
{
"groupId": "12345678987654321",
"type": "serial",
"memberInfo": "apple"
}
If I run the above, the call works, and the expected operation occurs. The issue is that I need to run this against roughly 400 values for that "memberInfo" field in the JSON payload. I'm trying to identify a way to run a single bash script, which will run this curl command over and over, and update the JSON payload to use each row in a file as the below:
memberList.txt
apple
banana
peach
pear
orange
.
.
And then maybe insert a pointer in my JSON for the "memberInfo" field over to this file.
Any and all help/suggestions are greatly appreciated!
.
This will do as you intend. Its a little convoluted but you might polish it a bit.
#!/bin/bash
function getString(){
echo $1 | python3 -c '
import json
import sys
payload="""
{
"groupId": "12345678987654321",
"type": "serial",
"memberInfo": ""
}
"""
obj = json.loads(payload)
obj["memberInfo"] = sys.stdin.read().strip()
print(json.dumps(obj, indent = " "))
'
}
while read member
do
getString "$member" > json_payload.json
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d #json_payload.json
done <<< "$( cat fruits.txt )"
Hope it helps!
while read member; do
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d '{"groupId": "12345678987654321","type": "serial","memberInfo": "$member"}'
done <members.txt
This will work if you only care about the memberInfo field, another method could be writing your json line by line to payloads.txt file.
payloads.txt
{"groupId": "12345678987455432","type": "stereo","memberInfo": "apple"}
{"groupId": "34532453453453465","type": "serial","memberInfo": "banana"}
...
then use this as the script
while read payload; do
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d '$payload'
done <payloads.txt
here is a collection of bash scripting common uses I've had to use
https://github.com/felts94/advanced-bash/blob/master/bash_learn.sh
I want to send an email with attached pdf file through the Sparkpost API with curl post.
To insert the pdf I use (my test.pdf is ~ 200KB)
"data":"'$(cat test.pdf} | base64 --wrap=0)'"
But somehow this doesn't work out showing the following error:
/usr/bin/curl: Die Argumentliste ist zu lang (original)
/usr/bin/curl: Argument list is too long
EDIT:
curl command
curl -X POST https://api.eu.sparkpost.com/api/v1/transmissions -H 'Authorization: <APIKEY>' -H 'Content-Type: application/json' -d '{
"options":{
"open_tracking":false,
"click_tracking":false,
"inline_css":false
},
"recipients":[
{
"address":{
"email":"user#domain.tld",
"name":"user"
}
}
],
"content":{
"from":{
"name":"sender",
"email":"sender#domain.tld"
},
"reply_to":"replyto#domain.tld",
"subject":"subject",
"text":"textbody",
"attachments":[
{
"name":"attachmentname.pdf",
"type":"application/pdf",
"data":"'$(cat test.pdf | base64 --wrap=0)'"
}
]
}
}'
This is coming up because you are trying to pass the entirety of the base64'd content on the command line. curl has the ability to load in data to POST from a file, which I'd recommend doing. More information can be found in the man page, but the basic format is this:
curl -X POST -d #filename.txt https://website.com/path
According to the curl manual, the -F option allows you to encode a file for base64, but limits the output to 76 characters.
Ex:
-F '=#localfile;encoder=base64'
I want to send a json request and embedd a variable in the post data.
I did a little research and I came up with the single quotes around the variable.
#!/bin/bash
FILENAME="/media/file.avi"
curl -i -X POST -H "Content-Type: application/json" —d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":{"file":"'$FILENAME'"}}}' http://192.167.0.13/jsonrpc
Unfortunately I get some errors:
curl: (6) Couldn't resolve host '—d'
curl: (3) [globbing] nested braces not supported at pos 54
HTTP/1.1 200 OK
Content-Length: 76
Content-Type: application/json
Date: Wed, 29 Jan 2014 19:16:56 GMT
{"error":{"code":-32700,"message":"Parse error."},"id":null,"jsonrpc":"2.0"}
Appearently there are some problems with the braces and the http answer states, that the command could not be executed. What's wrong with my code here?
Thanks!
This is my curl version:
curl 7.30.0 (mips-unknown-linux-gnu) libcurl/7.30.0 OpenSSL/0.9.8y
Protocols: file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL
Update: use the simpler
request_body=$(cat <<EOF
{
"jsonrpc": "2.0",
"method": "Player.Open",
"params": {
"item": {
"file": "$FILENAME"
}
}
}
EOF
)
rather than what I explain below. However, if it is an option, use jq to generate the JSON instead. This ensures that the value of $FILENAME is properly quoted.
request_body=$(jq -n --arg fname "$FILENAME" '
{
jsonrpc: "2.0",
method: "Player.Open",
params: {item: {file: $fname}}
}'
It would be simpler to define a variable with the contents of the request body first:
#!/bin/bash
header="Content-Type: application/json"
FILENAME="/media/file.avi"
request_body=$(< <(cat <<EOF
{
"jsonrpc": "2.0",
"method": "Player.Open",
"params": {
"item": {
"file": "$FILENAME"
}
}
}
EOF
))
curl -i -X POST -H "$header" -d "$request_body" http://192.167.0.13/jsonrpc
This definition might require an explanation to understand, but note two big benefits:
You eliminate a level of quoting
You can easily format the text for readability.
First, you have a simple command substitution that reads from a file:
$( < ... ) # bash improvement over $( cat ... )
Instead of a file name, though, you specify a process substitution, in which the output of a command is used as if it were the body of a file.
The command in the process substitution is simply cat, which reads from a here document. It is the here document that contains your request body.
My suggestion:
#!/bin/bash
FILENAME="/media/file 2.avi"
curl -i -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":{"file":"'"$FILENAME"'"}}}' http://192.167.0.13/jsonrpc
The differences are hyphen in -d (instead of a dash) and double quotes around $FILENAME.
Here is another way to insert data from a file into a JSON property.
This solution is based on a really cool command called jq.
Below is an example which prepares request JSON data, used to create a CoreOS droplet on Digital Ocean:
# Load the cloud config to variable
user_data=$(cat config/cloud-config)
# Prepare the request data
request_data='{
"name": "server name",
"region": "fra1",
"size": "512mb",
"image": "coreos-stable",
"backups": false,
"ipv6": true,
"user_data": "---this content will be replaced---",
"ssh_keys": [1234, 2345]
}'
# Insert data from file into the user_data property
request_data=$(echo $request_data | jq ". + {user_data: \"$user_data\"}")
I am trying to implement a simple server in Ruby, but somehow I can't get the data from a put request.
curl request that I am making:
curl -v -X PUT localhost:2016/api/kill -d {"connId" : 1}
The server seems to be reading the request alright.
The code:
while line = socket.gets
puts line.chomp
request << line.chomp
break if line =~ /^\s*$/
end
produces the output:
PUT /api/kill HTTP/1.1
User-Agent: curl/7.35.0
Host: localhost:2016
Accept: */*
Content-Length: 7
Content-Type: application/x-www-form-urlencoded
But I don't see the data anywhere?
Am I supposed to see it?
Is something wrong with the curl request?
You need single quotes around the body.
curl -v -X PUT localhost:2016/api/kill -d '{"connId" : 1}'