Command building & string escaping in Bash - bash

I am trying to build a Bash script that will take arguments from the command line as well as hard-coded arguments and pass them to a function, which in turn will construct a call to curl and perform an HTTP POST. Unfortunately, when I try passing strings with spaces in them, the script mangles the strings and the eventual call to curl fails to pass the arguments in the right order:
#!/bin/bash
API_URL="http://httpbin.org/"
docurl() {
arg1=$1
shift
arg2=$1
shift
args=()
while [ "$1" ]
do
arg_name="$1"
shift
arg_value="$1"
shift
args+=(-d $arg_name="$arg_value")
done
curl_result=$( curl -qSfs "$API_URL/post" -X POST -d arg1=$arg1 -d arg2=$arg2 ${args[#]} ) 2>/dev/null
echo "$curl_result"
}
docurl foo1 foo2 title "$1" body "$2"
The script invocation would be something like this:
test2.sh "Hello" "Body of the message"
The output of the script as it stands is this:
{
"form": {
"body": "Body",
"arg1": "foo1",
"arg2": "foo2",
"title": "Hello"
},
"headers": {
"Host": "httpbin.org",
"Connection": "close",
"Content-Length": "41",
"User-Agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"Content-Type": "application/x-www-form-urlencoded",
"Accept": "*/*"
},
"files": {},
"data": "",
"url": "http://httpbin.org/post",
"args": {},
"origin": "xxx.xxx.xxx.xxx",
"json": null
}
As you can see, in the form element, the fields "body" and "title" have been truncated. Can anybody let me know what on what I'm doing wrong here?

Use a bit more quotes!
args+=(-d "$arg_name"="$arg_value")
And:
curl_result=$( curl -qSfs "$API_URL/post" -X POST -d arg1="$arg1" -d arg2="$arg2" "${args[#]}" ) 2>/dev/null

Related

How to fix function which I'm trying to pass into "payload" in bash ? I've stuck with error "invalid payload"

I'm trying to configure custom slack notification from GitLab but keep getting error "invalid payload" in GitlabCI. I'm wondering what I'm doing wrong? Maybe that's an error in my function? Btw I've tested the weebhook itself and it's valid and working as expected
This is a piece from my .gitlab-ci-.yaml
ENVIRONMENT_NAME: "staging"
extends: .Deploy_image
only:
- master
after_script:
- if [ ${CI_JOB_STATUS} == "success" ]; then EXIT_STATUS=0; else EXIT_STATUS=1; fi
- envsubst </tmp/${CI_PIPELINE_ID}/${SCRIPT_PATH}/slackscript.sh> "slackscript.sh"
- source ./slackscript.sh; share_slack_update_deploy
and the script itself:
#!/bin/bash
set -euo pipefail
FAILURE=1
SUCCESS=0
SLACKWEBHOOKURL="my webhook url"
function print_slack_summary_deploy() {
local slack_msg_header
local slack_msg_body
local slack_channel
# Populate header and define slack channels
slack_msg_header=":x: *Deploy to ${ENVIRONMENT_NAME} failed*"
if [[ "${EXIT_STATUS}" == "${SUCCESS}" ]]; then
slack_msg_header=":heavy_check_mark: *Deploy to ${ENVIRONMENT_NAME} succeeded*"
slack_channel="channel-name"
fi
cat <<-SLACK
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${slack_msg_header}"
}
},
{
"type": "divider"
},
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": "*Stage:*\nDeploy"
},
{
"type": "mrkdwn",
"text": "*Pushed By:*\n${GITLAB_USER_NAME}"
},
{
"type": "mrkdwn",
"text": "*Job URL:*\nGITLAB_REPO_URL/${CI_JOB_ID}"
},
{
"type": "mrkdwn",
"text": "*Commit Message:*\nGITLAB_REPO_URL/${CI_COMMIT_MESSAGE}"
{
"type": "mrkdwn",
"text": "*Commit URL:*\nGITLAB_REPO_URL$(git rev-parse HEAD)"
},
{
"type": "mrkdwn",
"text": "*Commit Branch:*\n${CI_COMMIT_REF_NAME}"
}
]
},
{
"type": "divider"
}
]
}
SLACK
}
function share_slack_update_deploy() {
local slack_webhook
slack_webhook="$SLACKWEBHOOKURL"
curl -X POST \
--data-urlencode "payload=$(print_slack_summary_deploy)" \
"${slack_webhook}"
}
I've tested the webhook itself and it's valid and working as expected
If the webhook works locally (meaning the curl to ${slack_webhook} is successful), but the same webhook, when called by GitLab CI, does not work, that should probably mean one of the variables used in print_slack_summary_deploy() is not correctly valued (or is empty), resulting in an invalid payload.
I would display those variables in a debug print, and compare them with the one done during a manual (successful) test done locally.
As noted by the OP Vlad Tkachuk, the root cause was the JSON being invalid.
A tool like /usr/bin/json_pp (usually present when Perl is already installed) can help validate the variable (as seen here)
echo "${myJSONvar}" | json_pp 2>/dev/null || exit -1

CURL variables and problems parsing JSON [duplicate]

This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 1 year ago.
My script is creating pull requests using the github API:
./my-script.sh my-repo my-branch
#!/bin/bash
Repo=$1
Branch=$2
cd $Repo
get_data() {
cat <<EOF
{
"title": "PR title",
"head": $Branch,
"base": "development",
"body": "PR description"
}
EOF
}
echo $(get_data) # <----------------- I can see my value of the variable $Branch here
curl -X POST \
-H "Accept: application/vnd.github.v3+json" \
-d "$(get_data)" \ # <----------------- But here I'm facing "Problems parsing JSON"
-u my_user:my_token \
https://api.github.com/repos/my_user/$Repo/pulls
open https://github.com/my_user/$Repo/pulls
How can I set my variable into the curl correctly?
$Branch needs to be quoted as well:
get_data() {
cat <<EOF
{
"title": "PR title",
"head": "$Branch",
"base": "development",
"body": "PR description"
}
EOF
}

Bash get value of matching substring in a long string

I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty

I want to make some script to run the curl command

I'm trying to make some script to run the curl cmd.
IP=$1
WEBPORT=$2
STREAMPORT=$3
NAME=$4
curl -X POST -H "Content-Type:application/json"
-d '{"Image": "kerberos/kerberos", "ExposedPorts": {
"80/tcp": {}, "8889/tcp": {} }, "HostConfig": {
"PortBindings": { "80/tcp": [{ "HostPort": "'+$WEBPORT+ '" }],
"8889/tcp": [{ "HostPort": "'+$STREAMPORT+'" }] } } }'
http://${IP}:4243/containers/create?name=${NAME}
It' is my script
But it's not working...
is there any way to run this curl cmd in the script and make this script more comfortable by making function.
You could try the following code :
curl_post() {
curl -X POST -H "Content-Type:application/json" -d "{'Image': 'kerberos/kerberos', 'ExposedPorts': { '80/tcp': {}, '8889/tcp': {} }, 'HostConfig': { 'PortBindings': { '80/tcp': [{ 'HostPort': '$1' }],'8889/tcp': [{ 'HostPort': '$2' }] } } }" "http://$3:4243/containers/create?name=$4"
}
To invoke the function do :
curl_post $webport $streamport $ip $name
It probably needs some modification, but i hope it helps.

clone the server in zabbix using api

sorry guys, but i don't know how clone server in zabbix, so please help.
i didn't find the json request with this function in zabbix documentation, so try to create personal script, but when i get items from server and try to add it in new, have this error :
{"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid
params.","data":"Item uses host interface from non-parent
host."},"id":1}
thanks
if need of course my code :
create_host() {
a='{ "jsonrpc": "2.0", "method": "host.create", "params": { "host": "'$selected_name'", "interfaces": [ { "type": 1, "main": 1, "useip": 1, "ip": "'$IP'", "dns": "", "port": "10050" } ], "groups": [ { "groupid": "1" } ], "templates": [ { "templateid": "10001" } ], "inventory": { "macaddress_a": "01234", "macaddress_b": "56768" } }, "auth": "'$AUTH_TOKEN'", "id": 1 }'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$a"
}
push_items() {
id_of_host=`echo $3 | tr -d ']' | tr -d '[u' | sed -s "s/'//g"`
echo "name"$1
echo "key"$2
request_push='{
"jsonrpc": "2.0",
"method": "item.create",
"params": {
"name": "'$1'",
"key_": "'$2'",
"hostid": "'$id_of_host'",
"type": 10,
"value_type": 0,
"interfaceid": "2",
"delay": 300
},
"auth": "'$AUTH_TOKEN'",
"id": 1
}'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$request_push"
}
get_items() {
request='{
"jsonrpc": "2.0",
"method": "item.get",
"params": {
"output": ["name", "key_"],
"host": "'$ETALON_SERVER'",
"sortfield": "name"
},
"auth": "'$AUTH_TOKEN'",
"id": 1
}'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$request"
}
AUTH_TOKEN=$(authenticate)
if [ -z "$AUTH_TOKEN" ]; then
echo "Connection not established"
exit 1
else
echo "everything is ok"
echo $AUTH_TOKEN
fi
check_host=$(check_exist_host)
host_create=$(create_host)
#echo "$host_create"
id_host=`echo "$host_create" | python -c 'import json, sys; print json.load(sys.stdin)["result"]["hostids"]'`
items=$(get_items)
#echo $items
keys=`echo "$items" | python -c 'import json, sys; print ("".join(i["name"]+";"+i["key_"] +"|" for i in json.load(sys.stdin)["result"]))'`
while IFS='[;]' read -r s1 s2; do
name_item=$s1
item_key=$s2
push_items "$s1" "$s2" "$id_host"
done < <(printf "%b" "${keys//|/\\n}")
As old as this question is, it still comes up in the top search results...
The API doesn't allow host.clone() because it's supposed to be performed through host.get() and host.create(), depending on your needs. The following code clones
host name
host visible name
first IP address
groups, adding a new one (static ID)
macros
and changes
templates (statid ID)
The old host is renamed and disabled, in order to maintain data history.
from pyzabbix import ZabbixAPI
ZAPI = ZabbixAPI(api_url)
ZAPI.login(username, password)
batch_list = [
"host1-asdasd",
"host2-asdasd"
]
for hostname in batch_list:
try:
original_host = ZAPI.host.get(
filter={'host': hostname},
selectGroups='extend',
selectInterfaces='extend',
selectMacros='extend'
)[0]
disable = ZAPI.host.update(
hostid=original_host['hostid'],
status=1,
host=original_host['host'] + '-history',
name=original_host['name'] + ' (history)'
)
print(disable)
clone = ZAPI.host.create(
host=original_host['host'],
name=original_host['name'],
proxy_hostid=original_host['proxy_hostid'],
groups=original_host['groups'] + [{'groupid': 802}],
macros=original_host['macros'],
interfaces=[{'main': '1', 'type': '1', 'useip': '1', 'dns': '', 'port': '10050', 'bulk': '1',
'ip': original_host['interfaces'][0]['ip']}],
templates={'templateid': 25708}
)
print(clone)
except:
print('something went wrong with: ' + hostname)

Resources