curl set upload chunk size - bash

I want to upload a big file with curl.
For that, I want to split it, without saving it to disk (like with split). I tried to use --continue-at with Content-Length.
curl -s \
--request PATCH \
--header "Content-Type: application/offset+octet-stream" \
--header "Content-Length: ${length}" \
--header "Upload-Offset: ${offset}" \
--continue-at "${offset}" \
--upload-file "${file}" \
"${dest}"
But curl "overshoots" and ignores Content-Length. Is there something like --stop-at? Alternatively, I have to use dd, if necessary.
EDIT
dd solution:
curl -s \
--request PATCH \
--header "Content-Type: application/offset+octet-stream" \
--header "Content-Length: ${length}" \
--header "Upload-Offset: ${offset}" \
--data-binary "#-" \
"${dest}" < <(dd if=${file} skip=${offset} count=${length} iflag=skip_bytes,count_bytes 2>/dev/null)
but if possible I would like to use only cURL..

Related

Delete Oldest Snapshot through API | Digital Ocean

Hi There,
I'm working on bash script to take snapshots of Digital Ocean volumes and droplets through cURL api. I can get all snapshot list and post new snapshot through the script. However I can delete the snapshot through API by manual selecting the ID, but I want to automatation. For that I have to pick the ID of oldest snapshot from get response of snapshot list. can someone please help me in this regard?
My script:
#! /bin/bash
token= xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
get_list=$(curl -X GET \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $token" \
"https://api.digitalocean.com/v2/snapshots?page=1&per_page=100&resource_type=volume")
echo $get_list > get_list
d=$(date +'%Y%m%d%H%M')
echo -e "Get List of Snapshots before delete\n" $get_list "\n\n\n\n"
y=1
for ((x=21;x<=2602;x+=265))
do
id=${get_list:$x:36}
echo $id > id$y
((y += 1))
done
rm -rf id
#curl -X DELETE \
#-H 'Content-Type: application/json' \
#-H "Authorization: Bearer $token" \
#"https://api.digitalocean.com/v2/snapshots/${get_list:274:36}"
#get_list= $(curl -X GET -H "Content-Type: application/json" \
#-H "Authorization: Bearer $token" \
#"https://api.digitalocean.com/v2/droplets?page=1&per_page=100")
#echo -e "Get List of Snapshots after delete\n" $get_list "\n\n\n\n"
#post_snap=$(curl -X POST \
# -H 'Content-Type: application/json' \
# -H "Authorization: Bearer $token" \
# -d '{"name":"'"$d"'"}' \
# "https://api.digitalocean.com/v2/volumes/6d44e0e2-4fc4-11ed-b137-0a58ac14c316/snapshots")
#get_list=$(curl -X GET \
# -H 'Content-Type: application/json' \
# -H "Authorization: Bearer $token" \
# "https://api.digitalocean.com/v2/snapshots?page=1&per_page=100&resource_type=volume")
#echo -e "Get List of Snapshots after new snap\n" $get_list "\n\n\n\n"
The list of snapshots from api response is pasted below:
{"snapshots":[{"id":"0e24bf13-515e-11ed-b137-0a58ac14c316","name":"202210211633","regions":["lon1"],"created_at":"2022-10-21T16:33:12Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"2ef69541-5162-11ed-9450-0a58ac14c2c7","name":"202210211702","regions":["lon1"],"created_at":"2022-10-21T17:02:45Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"689047e8-517a-11ed-a0ab-0a58ac14c028","name":"202210211955","regions":["lon1"],"created_at":"2022-10-21T19:56:10Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0,"tags":[]},{"id":"6cc0b0f2-516d-11ed-9fc6-0a58ac14c1c9","name":"2022-10-21_18:23","regions":["lon1"],"created_at":"2022-10-21T18:23:13Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"7e20fcfa-5171-11ed-a0ab-0a58ac14c028","name":"2022-10-21_18:52","regions":["lon1"],"created_at":"2022-10-21T18:52:20Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"96a3a010-515f-11ed-9fc6-0a58ac14c1c9","name":"2022-10-21-16-44","regions":["lon1"],"created_at":"2022-10-21T16:44:10Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"a13c0105-5178-11ed-9f93-0a58ac14c247","name":"2022-10-21-19-43","regions":["lon1"],"created_at":"2022-10-21T19:43:26Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"c4dfd95b-5174-11ed-b137-0a58ac14c316","name":"2022-10-21_19:15","regions":["lon1"],"created_at":"2022-10-21T19:15:47Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"d517a7f2-5169-11ed-a0ab-0a58ac14c028","name":"2022-10-21_17:57","regions":["lon1"],"created_at":"2022-10-21T17:57:30Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]},{"id":"f7180242-515a-11ed-b137-0a58ac14c316","name":"2022-10-21-16-11","regions":["lon1"],"created_at":"2022-10-21T16:11:05Z","resource_id":"6d44e0e2-4fc4-11ed-b137-0a58ac14c316","resource_type":"volume","min_disk_size":1,"size_gigabytes":0.0248,"tags":[]}],"links":{},"meta":{"total":10}}

Extracting access token from curl request in shell script

Using a postman as a base, I have a curl request here and I'm trying to return the access token.
AUTHORIZATION=$(curl --location --request POST 'https://some.url/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode "grant_type=$GRANT_TYPE" \
--data-urlencode "client_id=$CLIENT_ID" \
--data-urlencode "client_secret=$CLIENT_SECRET"\)
When I echo I get an output like:
{"access_token":"16WkRKbVpHWXlZekJsWVd...","token_type":"Bearer","expires_in":14400}
I want to extract the access_token and use in other parts of my script. I've tried the adding jq .access_token -r as seen below, but I'm just returning a null variable.
AUTHORIZATION=$(curl --location --request POST 'https://some.url/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode "grant_type=$GRANT_TYPE" \
--data-urlencode "client_id=$CLIENT_ID" \
--data-urlencode "client_secret=$CLIENT_SECRET"\
-s \
| jq .access_token -r)
Solutions here: extract token from curl result by shell script advise saving to file and grepping on it. I don't really want to save a token to a file if I can avoid it.

It looks like you flipped the parameter name and value when calling jq. I think it should be:
jq -r .access_token
not jq .access_token -r
Other than that your solution looks fine.

Step 1. Save the response in a variable
AUTHORIZATION=$(curl --location --request POST 'https://some.url/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode "grant_type=$GRANT_TYPE" \
--data-urlencode "client_id=$CLIENT_ID" \
--data-urlencode "client_secret=$CLIENT_SECRET"\)
Step 2 use jq.
AUTHORIZATION = `jq '.access_token' <<< "$AUTHORIZATION"`
Step 3 eliminate the quotes.
AUTHORIZATION=`echo "$AUTHORIZATION" | tr -d '"'`.

malformed_path for uploading file into my dropbox

Local spurce file : /tmp/back/wp.bak.sql
Dst file : /
my access token is xxxxxx.
I want to upload the /tmp/back/wp.bak.sql in my local pc into dropbox root directory,and keep the name unchanged.
curl command 1:
curl -X POST https://content.dropboxapi.com/2/files/upload \
--header 'Authorization: Bearer xxxxxx' \
--header 'Content-Type: application/octet-stream' \
--header 'Dropbox-API-Arg: {"path":"/"}' \
--data-binary #'/tmp/back/wp.bak.sql'
Output info.
{"error_summary": "path/malformed_path/...", "error": {".tag": "path", "reason": {".tag": "malformed_path"}, "upload_session_id": ""}}
curl command 2:
cd /tmp/back
curl -X POST https://content.dropboxapi.com/2/files/upload \
--header 'Authorization: Bearer xxxxxx' \
--header 'Content-Type: application/octet-stream' \
--header 'Dropbox-API-Arg: {"path":"/"}' \
--data-binary #'wp.bak.sql'
Same error info.
How to fix the malformed_path for my curl command?

Enter the full path of your file in the path field :
--header 'Dropbox-API-Arg: {"path":"/wp.bak.sql"}'
So :
curl "https://content.dropboxapi.com/2/files/upload" \
-H 'Authorization: Bearer xxxxxx' \
-H 'Content-Type: application/octet-stream' \
-H 'Dropbox-API-Arg: {"path":"/wp.bak.sql"}' \
--data-binary #'wp.bak.sql'

This question previously answers are outdated and not working. Here is a working code which is tested and working perfectly
curl -X POST https://content.dropboxapi.com/2/files/upload \
--header "Authorization: Bearer <your token>" \
--header "Dropbox-API-Arg: {\"path\": \"/file_path.txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false}" \
--header "Content-Type: application/octet-stream" \
--data-binary "#file_path.txt"

Bash: Command output as variable to curl error

I trying to get this bash script to run a speedtest (speedtest-cli) then pass the output as a variable to pushbullet via curl.
#!/bin/bash
speed=$(speedtest --simple)
curl --header 'Access-Token: <-ACCESS-TOKEN->' \
--header 'Content-Type: application/json' \
--data-binary {"body":"'"$speed"'","title":"SpeedTest","type":"note"}' \
--request POST \
https://api.pushbullet.com/v2/pushes
Other commands have worked well using this method (eg. whoami) but speedtest and ifconfig just get an error like this:
{"error":{"code":"invalid_request","type":"invalid_request","message":"Failed to decode JSON body.","cat":"(=^‥^=)"},"error_code":"invalid_request"}

Your quoting is wrong:
speed=$(speedtest --simple)
curl --header 'Access-Token: o.4q87SC5INy6nMQZqVHJeymwRsvMXW74j' \
--header 'Content-Type: application/json' \
--data-binary "{\"body\":\"$speed\",\"title\":\"SpeedTest\",\"type\":\"note\"}" \
--request POST \
https://api.pushbullet.com/v2/pushes
Reading from a here document simplifies the quoting:
speed=$(speedtest --simple)
curl --header 'Access-Token: o.4q87SC5INy6nMQZqVHJeymwRsvMXW74j' \
--header 'Content-Type: application/json' \
--data-binary #- \
--request POST \
https://api.pushbullet.com/v2/pushes <<EOF
{ "body": "$speed",
"title": "SpeedTest",
"type": "note"
}
EOF
However, in general you should not assume that the contents of the variable are a properly encoded JSON string, so use a tool like jq to generate the JSON for you.
jq -n --arg data "$(speedtest --simple)" \
'{body: $data, title: "SpeedTest", type: "note"}' |
curl --header 'Access-Token: o.4q87SC5INy6nMQZqVHJeymwRsvMXW74j' \
--header 'Content-Type: application/json' \
--data-binary #- \
--request POST \
https://api.pushbullet.com/v2/pushes
This can be refactored easily:
post_data () {
url=$1
token=$2
data=$3
jq -n --arg d "$data" \
'{body: $d, title: "SpeedTest", type: "note"}' |
curl --header "Access-Token: $token" \
--header 'Content-Type: application/json' \
--data-binary #- \
--request POST \
"$url"
}
post_data "https://api.pushbullet.com/v2/pushes" \
"o.4q87SC5INy6nMQZqVHJeymwRsvMXW74j" \
"$(speedtest ---simple)"

Setting a variable within command substitution

I'm trying to get both the HTTP code and the output of a curl command as part of a shell script, but in turn I'm trying to set both of them as variables. My code (which uses the pushbullet API) looks like this:
CURL_OUTPUT="$(exec 3>&1; \
HTTP_CODE="$(curl -s -S \
-w "%{http_code}" -o >(cat >&3) \
--header 'Authorization: Bearer '"$ACCT_TOKEN" \
-X POST https://api.pushbullet.com/v2/pushes \
--header 'Content-Type: application/json' \
--data-binary "$JSON")" \
)"
which should theoretically set $CURL_OUTPUT to the JSON returned by curl and $HTTP_CODE to the status code I get. Instead, I get only $CURL_OUTPUT; $HTTP_CODE is empty.
However, if I don't do the outer nest, like so:
exec 3>&1; \
HTTP_CODE="$(curl -s -S \
-w "%{http_code}" -o >(cat >&3) \
--header 'Authorization: Bearer '"$ACCT_TOKEN" \
-X POST https://api.pushbullet.com/v2/pushes \
--header 'Content-Type: application/json' \
--data-binary "$JSON")" \
The command works as expected; I get the JSON redirected to stdout, and the status code ends up in $HTTP_CODE.
So is it possible to get both outputs? Can I assign variables within a command substitution line?

Resources