docker curl socket inside container - bash

I have a bash post process script for rtorrent.
In it I try to create a Container, start it and on the end remove it.
All via curl commands to the docker socket which i mounted into the container.
The command is successfully executed from rtorrent. The curl command for pushover is working nicely.
But I get a curl: (7) Couldn't connect to server Error Message for the docker curl commands.
Hope someone could point me in the right direction.
Log:
^#
---
^#/usr/local/bin/rtorrent-postprocess.sh /Pathtothedownload Nameofthedownload label
---
^#{"status":1,"request":"ec5c3c9c-5744-48f4-909b-68d38ec5e659"}curl: (7) Couldn't connect to server
curl: (7) Couldn't connect to server
curl: (7) Couldn't connect to server
curl: (7) Couldn't connect to server
--- Success ---
Script:
#!/bin/bash
# rtorrent postprocess Script by Tobias
export LANG=de_DE.UTF-8
# The file for logging events from this script
LOGFILE="/config/rtorrent-postprocess.log"
#LOGFILE="./debug.log"
# Pfad des Downloads
FOLDER="$1"
# Name des Downloads
NAME="$2"
# Label des Downloads
LABEL="$3"
# Media Verzeichniss /data/Media
MEDIA="/data/Media"
# COMPLETE Verzeichniss mit label /data/torrent/completed/$3
COMPLETE="/data/torrent/completed/$3"
##############################################################################
function edate
{
echo "`date '+%Y-%m-%d %H:%M:%S'` $1" >> "$LOGFILE"
}
function pushover {
curl -s \
-F "token=xxxxxxxxxxxxxxxx" \
-F "user=xxxxxxxxxxxxxxxxx" \
-F "message=$1 finished $2 $3 on `date +%d.%m.%y-%H:%m`" \
https://api.pushover.net/1/messages.json
}
edate " "
edate "Verzeichniss ist $COMPLETE"
edate "Name ist $NAME"
edate "Label ist $LABEL"
edate "rtorrent finished downloading $NAME"
pushover "rtorrent" "downloading" "$NAME"
edate "Starte Filebot - $COMPLETE/$NAME"
test_command() {
curl --unix-socket /var/run/docker.sock -X POST "http://localhost/containers/${NAME}/wait" -H "accept: application/json"
}
curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" -d '{ "Image": "rednoah/filebot", "Cmd": ["-script", "fn:amc", "--output", "/Media", "--action", "move", "-non-strict", "/volume1", "--log-file", "/opt/rtorrentvpn/config/filebot.log", "--conflict", "auto", "--def", "artwork=n", "seriesFormat=Serien/{localize.eng.n}/Season {s.pad(2)}/{localize.eng.n} - {s00e00} - {localize.deu.t}", "movieFormat=Filme/{localize.deu.n} ({y})/{localize.deu.n} ({y})", "musicFormat=Musik/{artist}/{album}/{fn}"], "HostConfig": { "Binds": ["'$COMPLETE/$NAME':/volume1", "data:/data", "/data/Media:/Media"]} }' "http://localhost/containers/create?name=${NAME}"
curl --unix-socket /var/run/docker.sock -X POST "http://localhost/containers/${NAME}/start" -H "accept: application/json"
if [ "$(test_command)" == "200" ]; then
edate "Status ist $test_command"
fi
curl --unix-socket /var/run/docker.sock -X DELETE "http://localhost/containers/${NAME}?force=true?v=true" -H "accept: application/json"
edate " "
edate "Filebot fertig"

I changed the PUID and GUID to the root id. Thanks to Robin479's comment. Now everything is running as expected.

Related

Jenkins pipeline stage build is green even though an error is present

I have a Jenkins Pipline with three stages: "Build" "Test" and "Deploy".
Here is the problem I have with "Build":
The build step ensures that structure of the Control-M Automation API json files are valid.
To do this, I utilize the $endpoint/build service provided by Automation API in the build step:
stage('Build') {
environment {
CONTROLM_CREDS = credentials('xxx')
ENDPOINT = 'xxx'
}
steps {
sh '''
username=$CONTROLM_CREDS_USR
password=$CONTROLM_CREDS_PSW
# Login
login=$(curl -k -s -H "Content-Type: application/json" -X POST -d \\{\\"username\\":\\"$username\\",\\"password\\":\\"$password\\"\\} "$ENDPOINT/session/login" )
token=$(echo ${login##*token\\" : \\"} | cut -d '"' -f 1)
# Build
curl -k -s -H "Authorization: Bearer $token" -X POST -F "definitionsFile=#ctmjobs/TestCICD.json" "$ENDPOINT/build"
curl -k -s -H "Authorization: Bearer $token" -X POST "$ENDPOINT/session/logout"
'''
}
}
<snip>
Everything works as expected, but if I intentionally put an error in the json file, Jenkins detects it and prints the error in the terminal, but "Build" still goes green. Can anyone identify the error? My expectation is that the stage "Build" goes to red as soon as there is an error in the JSON file.
Here is a Jenkins output from the terminal:
+ password=****
++ curl -k -s -H 'Content-Type: application/json' -X POST -d '{"username":"xxx","password":"****"}' /automation-api/session/login
+ login='{
"username" : "xxx",
"token" : "xxx",
"version" : "9.19.200"
}'
++ echo 'xxx",
' '"version"' : '"9.19.200"
' '}'
++ cut -d '"' -f 1
+ token=xxx
+ curl -k -s -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build
{
"errors" : [ {
"message" : "unknown type: Job:Dummmy",
"file" : "Test.json",
"line" : 40,
"col" : 29
}, {
"message" : "unknown type: Job:Dummmy",
"file" : "Test.json",
"line" : 63,
"col" : 29
} ]
}+ curl -k -s -H 'Authorization: Bearer xxx' -X POST /automation-api/session/logout
{
"message" : "Successfully logged out from session xxx"
} ``
Jenkins in order to consider a stage as failed, it will check the exit code of a command executed, in your case
curl -k -s -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build
The issue is that the curl, as a command, is executed successfully.
But the body of the curl indicates that the api call failed.
You could add --fail flag to your curl. This will force curl to return an erroneous exit code when the response status is > 400
(HTTP) Fail silently (no output at all) on server errors. This is
mostly done to enable scripts etc to better deal with failed attempts.
In normal cases when an HTTP server fails to deliver a document, it
returns an HTML document stating so (which often also describes why
and more). This flag will prevent curl from outputting that and return
error 22.
curl --show-error --fail -k -H 'Authorization: Bearer xxx' -X POST -F definitionsFile=#ctmjobs/Test.json /automation-api/build

backtick " ` " in request body in bash shell script while calling CURL

I'm writing CURL COMMAND
HTTP_Method : POST
API : api/databricks/query
Request body {"query":" GRANT SELECT,READ_METADATA,USAGE on DATABASE `DB` to `userID` " }
The query,I have execute using POSTMAN it's working
curl --location --request POST 'https://<databick_workspace-url>/api/sql/databricks/query' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data-raw '{"query":"GRANT SELECT,READ_METADATA,USAGE on DATABASE `DB-NAME` to `userID` " }
# Working, And user can access DB
I believe the problem is with backticks,
// Function
function grantReadAccess() {
local path="/api/sql/databricks/query"
local url="https://<databricks-workspace-url>/${path}"
printf 'curl %q' "${url}"
local DATABASENAME="DB-NAME"
local userID="userID"
local content="GRANT SELECT,READ_METADATA,USAGE on DATABASE \`${DATABASENAME}\` to \`${userID}\` "
echo "------------$content"
// OUTPUT Here : GRANT SELECT,READ_METADATA,USAGE on DATABASE `DB-NAME` to `userID`
// Same I have to pass in request body
local permissionToGroup=$(
curl -X POST "${url}" -H "Authorization: Bearer ${authtoken}" -H "Content-Type: application/json" -d '{ "query": \"'"${content}"'\" } ')
//Required in this {"query":"GRANT SELECT,READ_METADATA,USAGE on DATABASE `DB` to `userID` " }
echo "${permissionToGroup}"
}
// Tried In Postman insted of using backtick
1. Single Quote "'" {"query":"GRANT SELECT,READ_METADATA,USAGE on DATABASE 'DB' to 'userID' "
2. Blacket "(" {"query":"GRANT SELECT,READ_METADATA,USAGE on DATABASE (DB) to (userID) "
Error
{
"query": "grant SELECT, READ_METADATA, USAGE on DATABASE 'DB' to 'useID' ",
"data": null,
"error": " org.apache.spark.sql.catalyst.parser.ParseException: \nOperation not allowed: grant(line 1, pos 0)\n\n== SQL ==\ngrant SELECT, READ_METADATA, USAGE on DATABASE 'DB"
}
The above function is not working,
Always use a tool that understands JSON when you need to build JSON. In this context, the widely-accepted tool is for the job is jq.
curl -X POST "${url}" \
-H "Authorization: Bearer ${authtoken}" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg content "$content" '{"query": $content}')"
Your question is not very clear, but I'll guess you're having problems in this part:
-d '{ "query": \"'"${content}"'\" } '
That will get you literal backslashes before the double quotes around your content, since backslashes inside single quotes are inserted literally. You could swap that for -d '{ "query": "'"${content}"'" }' or -d "{ \"query\": \"${content}\" }"

Loop through list for curl requests in bash

I have a bash script that sends a curl request and displays the response.
#!/bin/bash
token=$(curl -k -X GET \
'https://v.mytesting.io/oauth/token?grant_type=password&username=user1&password=123' \
-H 'Authorization: Basic 12345678' \
-H 'Host: v.mytesting.io.io')
v=$( jq -r ".access_token" <<<"$token" )
ts=$(curl -k -X POST \
https://timeseries.mytimeseries.io/v5/time_series/query \
-H 'Authorization: Bearer '"$v" \
-H 'Content-Type: application/json' \
-H 'Host: timeseries.mytimeseries.io' \
-H 'tenant: 123-123-123' \
-d '{"operation" : "raw","responseFormat" : "kairosDB","startTime": "1d-ago","stopTime": "now","tagList" : [ {"tagId" : "V.S.23164117.AVG.10M"}]}')
p=$(jq '.queries[].sample_size, .queries[].results[].name' <<<"$ts")
echo "$p"
My current output is just a value and the name of the tagId.
My query only allows for 1 tagId ( you can see above )
I want to be able to set a list of tagId's.
Then when I run this script it should loop through the list of tagId's and execute the curl request replacing the V.S.23164117.AVG.10M with each value
in the list.
Then output the entire list of results into a file.
list would be like so - (I would love to be able to enter this list into a seperate file and the bash script calls that file. Sometimes this list can be a few hundred lines.
V.S.23164117.AVG.10M
V.S.23164118.AVG.10M
V.S.23164119.AVG.10M
V.S.23164115.AVG.10M
V.S.23164114.AVG.10M
output would like look so.
value tagId
value tagId
value tagId
100 V.S.23164117.AVG.10M
etc..
thank you for any help
You can loop over list of tags using a small script. I'm not 100% clean of the output format. You can change the 'echo' to match the required format.
Note minor change to quotes to allow variable expansion in the body.
The tags will be stored in a file, for examples, tags.txt
V.S.23164117.AVG.10M
V.S.23164118.AVG.10M
V.S.23164119.AVG.10M
And the script will be use the file
#! /bin/bash
# Use user defined list of tags
tags=tags.txt
token=$(curl -k -X GET \
'https://v.mytesting.io/oauth/token?grant_type=password&username=user1&password=123' \
-H 'Authorization: Basic 12345678' \
-H 'Host: v.mytesting.io.io')
v=$( jq -r ".access_token" <<<"$token" )
for tag in $(<$tags) ; do
ts=$(curl -k -X POST \
https://timeseries.mytimeseries.io/v5/time_series/query \
-H 'Authorization: Bearer '"$v" \
-H 'Content-Type: application/json' \
-H 'Host: timeseries.mytimeseries.io' \
-H 'tenant: 123-123-123' \
-d '{"operation" : "raw","responseFormat" : "kairosDB","startTime": "1d-ago","stopTime": "now","tagList" : [ {"tagId" : "'"$tag"'"}]}')
p=$(jq '.queries[].sample_size, .queries[].results[].name' <<<"$ts")
echo "$tag $p"
done

Replacing IP in curl command with bash variable

I'm currently trying to make a DDNS script that interacts with the Cloudflare API to catch changes in my ip address and automatically fix the ip address change for my web server. Everything is working correctly so far except I can't get $IP to be put properly in the curl statement. I first run a python script from within the bash script to get the ip address, then run the curl statement in the bash script. Here's what the python script looks like (it returns an ip address like "1.1.1.1" with quotations included because the curl command requires the quotations)
#!/usr/bin/python3
import subprocess as sp
def main():
command = "dig +short myip.opendns.com #resolver1.opendns.com";
ip = sp.check_output(command, shell=True).decode('utf-8').strip('\n');
ip_tmp = ip;
ip_tmp = '"' + ip + '"';
ip = ip_tmp;
print(ip);
if __name__ == "__main__":
main();
And the bash script looks like this:
#!/bin/bash
IP=$("./getIP.py")
curl -X PUT "https://api.cloudflare.com/client/v4/zones/zone_id/dns_records/dns_id" \
-H "X-Auth-Email: example.com" \
-H "X-Auth-Key: authkey" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"example.com","content":$IP,"ttl":120,"proxied":true}'
I've tried to have the python script only return numbers and then added the quotations in the bash script and now vice versa and I can't seem to get it to work. The last line should end up looking like this once the variable replaces with quotations around the ip address:
'{"type":"A","name":"example.com","content":"127.0.0.1","ttl":120,"proxied":true}'
The single quotes around your json structure prevent the variable from expanding.
You have a few options that are readily available.
Ugly quote escaping inside/around your json.
"{\"type\":\"A\",\"name\":\"example.com\",\"content\":$IP,\"ttl\":120,\"proxied\":true}"
Having the python write this data to a file and telling curl to use that file for the source of the post data.
curl -X PUT "https://api.cloudflare.com/client/v4/zones/zone_id/dns_records/dns_id" \
-H "X-Auth-Email: example.com" \
-H "X-Auth-Key: authkey" \
-H "Content-Type: application/json" \
--data #file_you_wrote_your_json_to.json
Using the python requests or urllib modules to issue the request to cloud flare.
Update your main() function to return the IP instead of print it.
my_ip = main()
url = "https://api.cloudflare.com/client/v4/zones/zone_id/dns_records/dns_id"
myheaders = {
"X-Auth-Email": "example.com",
"X-Auth-Key": "authkey",
"Content-Type": "application/json"
}
myjson = {
"type":"A",
"name":"example.com",
"content":my_ip,
"ttl":120,
"proxied":true
}
requests.put(url, headers=myheaders, data=myjson)
Better yet, just do it in bash. Cloudflare DDNS on github.
One shot to fetch the dynamic A-record ID:
curl -X GET "https://api.cloudflare.com/client/v4/zones/**Zone ID** \
/dns_records?type=A&name=dynamic" \
-H "Host: api.cloudflare.com" \
-H "User-Agent: ddclient/3.9.0" \
-H "Connection: close" \
-H "X-Auth-Email: example#example.com" \
-H "X-Auth-Key: "**Authorization key**" \
-H "Content-Type: application/json"
Cron job (* * * * *) to set the dynamic A-record:
#/usr/bin/env sh
AUTH_EMAIL=example#example.com
AUTH_KEY=** CF Authorization key **
ZONE_ID=** CF Zone ID **
A_RECORD_NAME="dynamic"
A_RECORD_ID=** CF A-record ID from cloudflare-dns-id.sh **
IP_RECORD="/tmp/ip-record"
RECORDED_IP=`cat $IP_RECORD`
PUBLIC_IP=$(curl --silent https://api.ipify.org) || exit 1
if [ "$PUBLIC_IP" = "$RECORDED_IP" ]; then
exit 0
fi
echo $PUBLIC_IP > $IP_RECORD
RECORD=$(cat <<EOF
{ "type": "A",
"name": "$A_RECORD_NAME",
"content": "$PUBLIC_IP",
"ttl": 180,
"proxied": false }
EOF
)
curl "https://api.cloudflare.com/client/v4/zones/$ZONE_ID \
/dns_records/$A_RECORD_ID" \
-X PUT \
-H "Content-Type: application/json" \
-H "X-Auth-Email: $AUTH_EMAIL" \
-H "X-Auth-Key: $AUTH_KEY" \
-d "$RECORD"

Using Curl to pull down Call Log Data

I want to pull down call data from RingCentral using a shell script and curl. I'm then going to put that into ELK to build a dashboard using Kibana. However, I don't know what I'm doing with the API. Does anyone have a place for me to start or some sample code to do this?
I'm currently struggling with just using curl to authenticate to get a token. At the moment I keep getting unsupported grant type. I setup the application in Sandbox and a "Server Only No UI".
I have run this from a Centos 7 box using a bash shell.
Here is the code have tried:
curl -X POST "https://platform.devtest.ringcentral.com/restapi/oauth/token"; \
-H "Accept: application/json" \
-H "Content-Type: application/x-www-form-urlencoded" \
-u "my client id:my client secret" \
-d "username=username&password=password&extension=<extension>&grant_type=password"
I left the username and password blank because I wasn't sure what that was.
The output is as follows:
{
"error" : "invalid_request",
"error_description" : "Unsupported grant type",
"errors" : [ {
"errorCode" : "OAU-250",
"message" : "Unsupported grant type"
} ]
}./rctest1.sh: line 2: -H: command not found
I've been able to reproduce your error and can resolve it by removing the semi-colon (;) after the URL in your command.
Explanation
A semi-colon creates two separate CLI commands instead of one, so in your call, you have two requests.
- Your Request 1
$ curl -X POST "https://platform.devtest.ringcentral.com/restapi/oauth/token"
- Your Request 2
$ -H "Accept: application/json" \
-H "Content-Type: application/x-www-form-urlencoded" \
-u "my client id:my client secret" \
-d "username=username&password=password&extension=&grant_type=password"
- Your Response 1
{
"error" : "invalid_request",
"error_description" : "Unsupported grant type",
"errors" : [ {
"errorCode" : "OAU-250",
"message" : "Unsupported grant type"
} ]
}
- Your Response 2
./rctest1.sh: line 2: -H: command not found
- Test Command
Here's a simple test showing the OS trying to process two commands:
$ hello;world
-bash: hello: command not found
-bash: world: command not found
Solution
- Working Request
Here is a working request without the semi-colon:
$ curl -X POST "https://platform.devtest.ringcentral.com/restapi/oauth/token" \
-H "Accept: application/json" \
-H "Content-Type: application/x-www-form-urlencoded" \
-u "my client id:my client secret" \
-d "username=username&password=password&extension=&grant_type=password"
- Working Response
Here is the working response:
{
"access_token" : "myAccessToken",
"token_type" : "bearer",
"expires_in" : 3600,
"refresh_token" : "myRefreshToken",
"refresh_token_expires_in" : 604800,
"scope" : "Meetings VoipCalling Glip SubscriptionWebhook Faxes Contacts RingOut SMS",
"owner_id" : "11111111",
"endpoint_id" : "22222222"
}

Resources