Download iCloud file from shared link using bash - bash

I want to download a file from iCloud. I can share a link to a file. However the file is not directly linked in the urls, but the "real" download url can be retrieved:
#!/bin/bash
# given "https://www.icloud.com/iclouddrive/<ID>#<Filename>
ID="...."
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \
--data-raw '{"shortGUIDs":[{"value":"$ID"}]}' --compressed \
jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')
curl "$URL" -o myfile.ext
Sorce: https://gist.github.com/jpillora/702ded79330043e38e8202b5c73835e5
"fileContent" : {
"value" : {
...
"downloadURL" : "https://cvws.icloud-content.com/B/CYo..."
},
This is, however not working:
rl: (6) Could not resolve host: jq
curl: (3) nested brace in URL position 17:
{
"results" : [ {
"shortGUID" : {
"value" : "$ID",
"shouldFetchRootRecord" : true
},
"reason" : "shortGUID cannot be null or empty",
"serverErrorCode" : "BAD_REQUEST"
} ]
}
Any ideas, what I can do to make this work?

as #dan mentioned, jq is not a curl argument, its a separate command. hence you would need to | pipe it instead of \.
So the command would look something like this:
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \ --data-raw '{"shortGUIDs":[{"value":"$ID"}]}' --compressed | jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')

I solved it by installing jq and adding the ID directly instead of using $id. Just installing jq was not sufficient.
brew install jq
#!/bin/bash
# given "https://www.icloud.com/iclouddrive/<ID>#<Filename>
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \
--data-raw '{"shortGUIDs":[{"value":"ID"}]}' --compressed | jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')
Echo $url
curl "$URL" -o myfile.ext

Related

Jenkins pipeline stage build is green even though an error is present

I have a Jenkins Pipline with three stages: "Build" "Test" and "Deploy".
Here is the problem I have with "Build":
The build step ensures that structure of the Control-M Automation API json files are valid.
To do this, I utilize the $endpoint/build service provided by Automation API in the build step:
stage('Build') {
environment {
CONTROLM_CREDS = credentials('xxx')
ENDPOINT = 'xxx'
}
steps {
sh '''
username=$CONTROLM_CREDS_USR
password=$CONTROLM_CREDS_PSW
# Login
login=$(curl -k -s -H "Content-Type: application/json" -X POST -d \\{\\"username\\":\\"$username\\",\\"password\\":\\"$password\\"\\} "$ENDPOINT/session/login" )
token=$(echo ${login##*token\\" : \\"} | cut -d '"' -f 1)
# Build
curl -k -s -H "Authorization: Bearer $token" -X POST -F "definitionsFile=#ctmjobs/TestCICD.json" "$ENDPOINT/build"
curl -k -s -H "Authorization: Bearer $token" -X POST "$ENDPOINT/session/logout"
'''
}
}
<snip>
Everything works as expected, but if I intentionally put an error in the json file, Jenkins detects it and prints the error in the terminal, but "Build" still goes green. Can anyone identify the error? My expectation is that the stage "Build" goes to red as soon as there is an error in the JSON file.
Here is a Jenkins output from the terminal:
+ password=****
++ curl -k -s -H 'Content-Type: application/json' -X POST -d '{"username":"xxx","password":"****"}' /automation-api/session/login
+ login='{
"username" : "xxx",
"token" : "xxx",
"version" : "9.19.200"
}'
++ echo 'xxx",
' '"version"' : '"9.19.200"
' '}'
++ cut -d '"' -f 1
+ token=xxx
+ curl -k -s -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build
{
"errors" : [ {
"message" : "unknown type: Job:Dummmy",
"file" : "Test.json",
"line" : 40,
"col" : 29
}, {
"message" : "unknown type: Job:Dummmy",
"file" : "Test.json",
"line" : 63,
"col" : 29
} ]
}+ curl -k -s -H 'Authorization: Bearer xxx' -X POST /automation-api/session/logout
{
"message" : "Successfully logged out from session xxx"
} ``
Jenkins in order to consider a stage as failed, it will check the exit code of a command executed, in your case
curl -k -s -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build
The issue is that the curl, as a command, is executed successfully.
But the body of the curl indicates that the api call failed.
You could add --fail flag to your curl. This will force curl to return an erroneous exit code when the response status is > 400
(HTTP) Fail silently (no output at all) on server errors. This is
mostly done to enable scripts etc to better deal with failed attempts.
In normal cases when an HTTP server fails to deliver a document, it
returns an HTML document stating so (which often also describes why
and more). This flag will prevent curl from outputting that and return
error 22.
curl --show-error --fail -k -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build

Editing GIST with cURL

#!/bin/bash
COMMIT=$(git log -1 --pretty=format:'{"subject": "%s", "name": "xxx", "date": "%cD"}')
curl -X PATCH -d'{"files": {"latest-commit": {"content": "$COMMIT"}}}' -u user:xxxx https://api.github.com/gists/xxx
This just shows $COMMIT in the Gist. I tried playing with ''' and stuff but cannot make this work.
Your $COMMIT variable is not expanded to its value, because it is enclosed in single-quotes.
About an actual implementation in Bash
The GitHub API require you send the file content as a string: https://developer.github.com/v3/gists/#input-1
When file content contains newlines, double quotes or other characters needing an escaping within a string, the most appropriate shell tool to fill-in and escape the content string is jq.
JavaScript provide a JSON.stringify() method, but here in the shell world, we use jq to process JSON data.
If you don't have jq available you can convert the content of the file, to a properly escaped JSON string with GNU sed this way:
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses sed to properly fill-in and escape the content string
read -r -d '' json_data_payload <<EOF
{
"description": "Updated from GitHub API call in Bash",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": "$(
sed ':a;N;$!ba;s/\n/\\n/g;s/\r/\\r/g;s/\t/\\t/g;s/"/\\"/g;' <<<"$latest_commit_json_content"
)"
}
}
}
EOF
This is how jq is used to fill the content string with proper escaping:
json_data_payload="$(
jq \
--arg content "$latest_commit_json_content" \
--compact-output \
'.files."latest-commit.json".content = $content' \
<<'EOF'
{
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": ""
}
}
}
EOF
)"
Detailed and tested ok implementation:
#!/usr/bin/env bash
# Set to the gist id to update
gist_id='4b85f310233a6b9d385643fa3a889d92'
# Uncomment and set to your GitHub API OAUTH token
github_oauth_token='###################'
# Or uncomment this and set to your GitHub username:password
#github_user="user:xxxx"
github_api='https://api.github.com'
gist_description='Gist update with API call from a Bash script'
filename='latest-commit.json'
get_file_content() {
# Populate variables from the git log of latest commit
# reading null delimited strings for safety on special characters
{
read -r -d '' subject
read -r -d '' author
read -r -d '' date
} < <(
# null delimited subject, author, date
git log -1 --format=$'%s%x00%aN%x00%cD%x00'
)
# Compose the latest commit JSON, and populate it with the latest commit
# variables, using jq to ensure proper encoding and formatting of the JSON
read -r -d '' jquery <<'EOF'
.subject = $subject |
.author = $author |
.date = $date
EOF
jq \
--null-input \
--arg subject "$subject" \
--arg author "$author" \
--arg date "$date" \
"$jquery"
}
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses jq to properly fill-in and escape the content string
# and compact the output before transmission
get_gist_update_json() {
read -r -d '' jquery <<'EOF'
.description = $description |
.files[$filename] |= (
.filename = $filename |
.content = $content
)
EOF
jq \
--null-input \
--compact-output \
--arg description "$gist_description" \
--arg filename "$filename" \
--arg content "$(get_file_content)" \
"$jquery"
}
# prepare the curl call with options for the GitHub API request
github_api_request=(
curl # The command to send the request
--fail # Return shell error if request unsuccessful
--request PATCH # The request type
--header "Content-Type: application/json" # The MIME type of the request
--data "$(get_gist_update_json)" # The payload content of the request
)
if [ -n "${github_oauth_token:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with a OAUTH token
--header "Authorization: token $github_oauth_token"
)
elif [ -n "${github_user:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with an HTTP auth user:pass
--user "$github_user"
)
else
echo 'GitHub API require either an OAUTH token or a user:pass' >&2
exit 1
fi
github_api_request+=(
-- # End of curl options
"$github_api/gists/$gist_id" # The GitHub API url to address the request
)
# perform the GitHub API request call
if ! "${github_api_request[#]}"; then
echo "Failed execution of:" >&2
env printf '%q ' "${github_api_request[#]}" >&2
echo >&2
fi
Here is the generated curl call with my token redacted out:
curl --fail --request PATCH --header 'Content-Type: application/json' \
--data '{"description":"Hello World Examples","files":{"latest-commit.json":{"filename":"latest-commit.json","content":"{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"}}}' \
--header 'Authorization: token xxxx-redacted-xxxx' \
-- \
https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92
And the JSON response it replied with:
"url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92",
"forks_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/forks",
"commits_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/commits",
"id": "4b85f310233a6b9d385643fa3a889d92",
"node_id": "MDQ6R2lzdDRiODVmMzEwMjMzYTZiOWQzODU2NDNmYTNhODg5ZDky",
"git_pull_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"git_push_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"html_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"type": "application/json",
"language": "JSON",
"raw_url": "https://gist.githubusercontent.com/leagris/4b85f310233a6b9d385643fa3a889d92/raw/7cb7f9d4a0170daf5083929858fb7eef706f8b59/latest-commit.json",
"size": 105,
"truncated": false,
"content": "{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"
}
},
...

Editing Gist with cURL: "Problems parsing JSON",

#!/bin/bash
curl -v \
--request PATCH \
--data "$(
printf '{"files": {"somefile.json": {"content": " {"field": "value"} "}}}' \
)" \
--user x:x \
https://api.github.com/gists/x
Tried adding --header "Content-Type: application/json", no luck.
I'm using this because the content is actually a command output but right now I'm testing the basics because this is not workig.
I believe is somethig related to double quote escaping in bash, tried for a couple of hours with no luck. This is a nightmare.
Any tip is welcomed. Thanks.
It looks like you have too many quotation marks. If you want the value of the "content" element to be an object, then instead of this:
"content": " {"field": "value"} "
try this:
"content": {"field": "value"}
On the off chance that you want it to be a string, then try this:
"content": " {\"field\": \"value\"} "
Instead of fighting against quote escaping, you could write your payload to a file and tell curl to use that file as data like so:
curl -v \
--request PATCH \
--data #/tmp/some/file \
--user x:x \
https://api.github.com/gists/x
Note the # sign in the --data argument, which tells curl that the rest of the argument is a file name to read data from.
Depending on how you create your payload, you could also pipe it to curl using - as filename (echo payload | curl --data #- ...)

curl: argument list too long

I want to send an email with attached pdf file through the Sparkpost API with curl post.
To insert the pdf I use (my test.pdf is ~ 200KB)
"data":"'$(cat test.pdf} | base64 --wrap=0)'"
But somehow this doesn't work out showing the following error:
/usr/bin/curl: Die Argumentliste ist zu lang (original)
/usr/bin/curl: Argument list is too long
EDIT:
curl command
curl -X POST https://api.eu.sparkpost.com/api/v1/transmissions -H 'Authorization: <APIKEY>' -H 'Content-Type: application/json' -d '{
"options":{
"open_tracking":false,
"click_tracking":false,
"inline_css":false
},
"recipients":[
{
"address":{
"email":"user#domain.tld",
"name":"user"
}
}
],
"content":{
"from":{
"name":"sender",
"email":"sender#domain.tld"
},
"reply_to":"replyto#domain.tld",
"subject":"subject",
"text":"textbody",
"attachments":[
{
"name":"attachmentname.pdf",
"type":"application/pdf",
"data":"'$(cat test.pdf | base64 --wrap=0)'"
}
]
}
}'
This is coming up because you are trying to pass the entirety of the base64'd content on the command line. curl has the ability to load in data to POST from a file, which I'd recommend doing. More information can be found in the man page, but the basic format is this:
curl -X POST -d #filename.txt https://website.com/path
According to the curl manual, the -F option allows you to encode a file for base64, but limits the output to 76 characters.
Ex:
-F '=#localfile;encoder=base64'

"Invalid credentials" while doing a curl POST

I have a curl request in below format
curl -v -H "Content-Type:application/json" -H "x-user-id:xxx" -H "x-api-key:yyy" --data '{"logs":"'"${TEST_OUTPUT}"'","pass":"true | false"}' https://razeedash.one.qqq.cloud.com/api/v1/clusters/zzz/api/test_results
This works fine while I do from my MAC terminal. But the same command throws
13:49:26 {
13:49:26 "status": "error",
13:49:26 "message": "Invalid credentials"
13:49:26 }
I saw this post but not sure how else would I send a json body without curly braces. I know that we can save it as a file.json and use the file as body.But for some reasons that cannot be implemented in my scenario
In general, you should avoid trying to build JSON using string interpolation. Use a tool like jq to handle any necessary quoting.
jq -n --argson o "$TEST_OUTPUT" '{logs: $o, pass: "true | false"}' |
curl -v -H "Content-Type:application/json" \
-H "x-user-id:xxx" \
-H "x-api-key:yyy" \
--data #- \
https://razeedash.one.qqq.cloud.com/api/v1/clusters/zzz/api/test_results
However, if you can manage to correctly generate your JSON as you are now, you can just replace the jq command with echo:
echo '{"logs": ...' | curl ...
The #- argument to --data says to read from standard input.

Resources