I want to transfer old db_backups to some cold storage(Azure blob storage). How can I automate the task to transfer the files when the size reaches some specific limit, all ths should be done on remote server which I access through ssh in terminal.
Also how to read the response from which I can extract access_token and
can use that in my next command to be run automatically.
can anyone give me some example regarding this or any article to read?
How to add curl post request in bash script?
curl -X POST 'https://server.domain.com/v2/jobs/28723316-9373-44ba-9229-7c796f21b099/runs?project_id=aff59748-260a-476e-9578-b4f4a93e7a92' -H 'Content-Type: application/json' -H "Authorization: Bearer $token" -d ''
output is something like this :
{
"keys": [
{
"keyName": "key1",
"value": "FD0o1y8eSPutze",
"permissions": "FULL"
},
{
"keyName": "key2",
"value": "RLQ59xAi8Eg6p8VpIYx",
"permissions": "FULL"
}
]
}
I want to know how the above command will be run in a shell script. I want to take key1 value from the response and echo it.
but right now I am having trouble in adding $token in the bearer authorization.
can anyone tell me the right format for this in shell script.
Related
I'm writing a bash script to interface with my ecobee (query info and change settings). I have the authorization all worked out (access token and refresh token) and am now trying to request info from the ecobee. This json parameter list is dynamically created. None of the curl examples in the Developers API Doc seem to work.
I've tried assigning the json to a variable (?json="$selection") and to a file (?json=#"$location"). My latest attempt (hard coding the json and escaping the braces) of the curl is as follows:
response=$(curl -s "https://api.ecobee.com/1/thermostat?json=\{"selection":\{"selectionType":"registered","selectionMatch":null,"includeRuntime":true,"includeSettings":true/}/}" -H "Content-Type: application/json;charset=UTF-8" -H "Authorization: Bearer $access_token")
and I received a null response declare -- response=""
If I have curl read from a file:
{"selection":{"selectionType":"registered","selectionMatch":null,"includeRuntime":true,"includeSettings":true}}
then I receive:
response='{
"status": {
"code": 4,
"message": "Serialization error. Malformed json. Check your request and parameters are valid."
}
}'
Which I assuming it' an escape issue?
Can anyone offer any insight? Again, I thought this would be the easy part. I'm using jq to build my request json. Other alternatives to curl that can better deal with json? I'm limited to bash (which is what I know)
Thanks
To integrate (arbitrary) data into a URL (or URI in general) you need to prepare it using percent-encoding first.
As you have mentioned to use jq to compose that data, you could add #uri to your jq filter to perform the encoding, and use the --raw-output (or -r) option to then output it properly.
For example, jq -r '#uri' applied to your JSON
{
"selection": {
"selectionType": "registered",
"selectionMatch": null,
"includeRuntime": true,
"includeSettings": true
}
}
would output
%7B%22selection%22%3A%7B%22selectionType%22%3A%22registered%22%2C%22selectionMatch%22%3Anull%2C%22includeRuntime%22%3Atrue%2C%22includeSettings%22%3Atrue%7D%7D
which you can use within the query string ?json=….
I am trying to post message to Microsoft Teams channel using Windows batch script but I could not make use of the Teams message card formats. I am able to post messages using the below commands but as plain texts. Is there anyway in which I can make use of the Message card JSON formats ?
I also have some command line arguments which need to be used for the batch script so that the message displayed uses the same arguments.
curl -H "Content-type: application/json" --data "{\"#type\": \"ActionCard\",\"title\": \"New Lab %2 deployed successfully\", \"text\": \"Status is %3\"}" %1
The above command worked just fine. But doesnt satisfy my exact requirement which is described above.
I also created a seperate json file which was called as below and this worked fine but couldnt make use of command line arguments to format the JSON values.
curl --data #message.json webhook_url
message.json is as below
{
"summary":"New Lab deployed",
"sections":[
{
"activityTitle":"A <b>new lab</b> has been added!"
},
{
"title":"Details:",
"facts":[
{
"name":"Lab Name",
"value":"REPLACE"
},
{
"name":"Status",
"value":"REPLACE"
}
]
}
]
}
cURL is able to read data from file:
--data-binary "#message.json"
do not forget prepend AT sign to identify the doublequoted string is a filename, not a data itself.
I am trying to upload a zip file to Google drive account using curl.
The file is uploaded successfully but the filename is not getting updated. It gets uploaded with default filename i.e. "Untitled".
I am using below command.
curl -k -H "Authorization: Bearer cat /tmp/token.txt" -F "metadata={name : 'backup.zip'} --data-binary "#backup.zip" https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart
You can use Drive API v3 to upload the zip file. The modified curl code is as follows.
curl -X POST -L \
-H "Authorization: Bearer `cat /tmp/token.txt`" \
-F "metadata={name : 'backup.zip'};type=application/json;charset=UTF-8" \
-F "file=#backup.zip;type=application/zip" \
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"
In order to use this, please include https://www.googleapis.com/auth/drive in the scope.
The answer above works fine and was the command I used in uploading my file to Google Drive using Curl. However, I didn't understand what scope was and all of the initial setup required to make this command work. Hence, for documentation purposes. I'll give a second answer.
Valid as at the time of writing...
Visit the Credentials page and create a new credential (this is assuming you have created a project). I created credentials for TVs and Limited devices, so the work flow was similar to:
Create credentials > OAuth client ID > Application Type > TVs and Limited Input devices > Named the client > Clicked Create.
After doing this, I was able to copy the Client ID and Client Secret when viewing the newly created credential.
NB: Only the variables with double asterisk from the Curl commands should be replaced.
Next step was to run the Curl command:
curl -d "client_id=**client_id**&scope=**scope**" https://oauth2.googleapis.com/device/code
Scope in this situation can be considered to be the kind of access you intend to have with the credential having the inputted client_id. More about scope from the docs For the use case in focus, which is to upload files, the scope chosen was https://www.googleapis.com/auth/drive.file.
On running the curl command above, you'll get a response similar to:
{ "device_code": "XXXXXXXXXXXXX", "user_code": "ABCD-EFGH",
"expires_in": 1800, "interval": 5, "verification_url":
"https://www.google.com/device" }
Next step is to visit the verification_url in the response in your browser, provide the user_code and accept requests for permissions. You will be presented with a code when all prompts have been followed, this code wasn't required for the remaining steps (but there may be some reasons to use it for other use cases).
Next step is to use the Curl command:
curl -d client_id=**client_id** -d client_secret=**client_secret** -d device_code=**device_code** -d grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code https://accounts.google.com/o/oauth2/token
You will get a response similar to:
{ "access_token": "XXXXXXXXX", "expires_in": 3599,
"refresh_token": "XXXXXXXXX", "scope":
"https://www.googleapis.com/auth/drive.file", "token_type": "Bearer"
}
Now you can use the access token and follow the accepted answer with a Curl command similar to:
curl -X POST -L \
-H "Authorization: Bearer **access_token**" \
-F "metadata={name : 'backup.zip'};type=application/json;charset=UTF-8" \
-F "file=#backup.zip;type=application/zip" \
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"
I have elasticsearch backup taken into S3. But I am not able to restore it using any of the commands mentioned below.
curl -XPOST http://localhost:9200/_snapshot/elasticsearch/snap-dev_1/_restore
curl -XPOST http://localhost:9200/_snapshot/snap-deliveryreports_june2016bk/elasticsearch/_restore
I can see the files in S3:
What is the command to restore the data shown in the image?
update:
The following command is successful (returns acknowleged: true)
It means access key, secret key, bucket name and region is correct.
curl -XPUT 'http://localhost:9200/_snapshot/s3_repository?verify=true&pretty' -d'
{
"type": "s3",
"settings": {
"bucket": "todel162",
"region": "us-east-1"
}
}'
I guess I only need to know how to use restore snapshot command.
You can use the cat recovery API to monitor your restore status, as restoring just piggybacks on the regular recovery mechanism of elasticsearch, so check if you see anything using those APIs.
I want to set the HOST env var to $HEROKU_APP_NAME.herokuapps.com on a preview app. It doesn't look like I can do this in app.json since this is a computed value.
I was hoping to do it in a "postdeploy" script like this
heroku config:set HOST="`heroku config:get HEROKU_APP_NAME -a neon-dev-pr-520`.herokuapps.com"
but it wants to authenticate me as a Heroku user. Alas, this doesn't work either:
export HOST=$HEROKU_APP_NAME.herokuapps.com
Any suggestions?
Worked out with the assistance of Heroku's awesome support team a few years ago. We needed to set a reflexive environment variable for a middleware (parse-server) to know what to connect to. It's set manually on our Staging and Production apps, but to get it set on our review apps:
My app.json incldues:
"scripts": {
"postdeploy": "bin/bootstrap"
},
...
"env": {
"HEROKU_APP_NAME": {
"required": true
},
"HEROKU_PARENT_APP_NAME": {
"required": true
},
...
bin/bootstrap is:
#!/usr/bin/env bash
echo $HEROKU_APP_NAME
export SERVER_URL=https://$HEROKU_APP_NAME.herokuapp.com/parse
SERVER_URL is available (and correct) in my review apps. It's been 👍no problems since we implemented.
I found this post which suggests that you can use the Heroku PlatformAPI.
In this case as a Rails Rake task which is run as the postdeploy:
desc 'Bootstrap review app'
task bootstrap: ['db:schema:load', 'db:seed'] do
heroku = PlatformAPI.connect_oauth(ENV['HEROKU_API_TOKEN'])
heroku.config_var.update(ENV['HEROKU_APP_NAME'], 'WWW_HOSTNAME' => "#{ENV['HEROKU_APP_NAME']}.herokuapp.com")
end
Take a look at the documentation for review apps https://devcenter.heroku.com/articles/github-integration-review-apps#heroku_app_name-and-heroku_parent_app_name
As long as you declare HEROKU_APP_NAME or HEROKU_PARENT_APP_NAME as required or optional in your app.json file, they will be available for you to use in your postdeploy script so you can just do:
HOST="$HEROKU_APP_NAME.herokuapp.com"
Using python requests within a postdeploy script:
import requests
result = requests.patch(f"https://api.heroku.com/apps/{os.environ.get('HEROKU_APP_NAME')}/config-vars",
data=json.dumps({"YOUR_KEY": "SOME_VALUE"}),
headers={"Content-Type": "application/json",
"Accept": "application/vnd.heroku+json; version=3",
"Authorization": f"Bearer {os.environ.get("HEROKU_API_KEY")}"}
)
Or using curl when logged:
curl -n -X PATCH https://api.heroku.com/apps/$HEROKU_APP_NAME/config-vars \
-d '{
"FOO": "bar",
"BAZ": "qux"
}' \
-H "Content-Type: application/json" \
-H "Accept: application/vnd.heroku+json; version=3"
And when not logged in add the header, where an app token has been generated:
-H "Authorization: Bearer $HEROKU_API_KEY"
https://2.python-requests.org/en/master/user/authentication/#netrc-authentication
https://devcenter.heroku.com/articles/platform-api-reference#curl-examples