when I execute a normal curl via a shell script functioniert es.
This work:
curl -s -v -X POST --data '{
"zoneConfig": {
"userID": "'$userid'",
"name": "'$myName'",
"id":"'$id'"
},
"delete": [
{
"id": "ID1"
},
{
"id": "ID2"
}
]
}' https://urlToAPI
But as soon as I put "delete" in a variable I get an undefined error from the API vendor
This is not working
delete='{
"id": "ID1"
},
{
"id": "ID2"
}'
curl -s -v -X POST --data '{
"zoneConfig": {
"userID": "'$userid'",
"name": "'$myName'",
"id":"'$id'"
},
"delete": [
'$deleteValues'
]
}' https://urlToAPI
But I don't understand the difference as both configurations are the same?
When interpolating, the value is split on whitespace.[1]
As such,
a='a b c'
prog $a
is equivalent to
prog 'a' 'b' 'c'
This splitting doesn't occur if the interpolation occurs inside of double-quotes.
As such,
a='a b c'
prog "$a"
is equivalent to
prog 'a b c'
Therefore, you need to change
$deleteValues
to
"$deleteValues"
To be precise, the IFS env var controls how the value is split. It's normally set such that splitting occurs on spaces, tabs and line feeds.
Related
I've got this json-file:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"23423565",
"23552352",
"08789089"
]
}
}
I need to iterate over every element of an list (upper_one), and replace it with other value.
I've tried this code:
#/bin/bash
for product in $(cat first-shop.json| jq -r '.shelves.upper_one[]')
do
cat first-shop.json| jq --arg id "$((1 + $RANDOM % 10))" --arg product "$product" -r '.shelves.upper_one[]|select(. == $product)|= $id'
done
But I got this kind of output:
1
23552352
08789089
23423565
10
08789089
23423565
23552352
7
Is it possible to iterate over list with jq, replace values with value from another function (like $id in the code), and print the whole final json with substituted values?
I need this kind of output:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
not just elements of "upper_one" list thrice.
You could try the following script :
#!/usr/bin/env bash
for product in $(jq -r '.shelves.upper_one[]' input.json)
do
id="$((1 + $RANDOM % 10))"
newIds+=("$id")
done
jq '.shelves.upper_one = $ARGS.positional' input.json --args "${newIds[#]}"
IMHO its better to use some scripting language and manipulate objects programmatically. If bash and jq is your only option - this do the job though not nice
$ jq '.shelves.upper_one[] |= (sub("23423565";"1") | sub("23552352";"10") | sub("08789089";"7"))' your.json
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
consider conversion to numbers with | tonumber
I am working on a shell script that would read filenames based on a topic name, read the file content, convert the text to a single line and post the message to kafka using kafka consumer.
For converting the contents of the json file, I am trying to use sed and seen many examples and tried them all. But none of them seem to convert the multiline text to single line so I could use it to post to kafka topic
This is the script I am trying to write
declare -A topicconfig
while read -r topic configfile; do
topicconfig["$topic"]="$configfile"
done <config_params.txt
for topic in "${!topicconfig[#]}"; do
echo "filename : ${topicconfig[$topic]}"
textFromFile=$(sed -e 's/[\n|\r]$//g' "${topicconfig[$topic]}")
echo "textFromFile : $textFromFile"
/bin/kafka-console-producer --bootstrap-server=${BOOTSTRAP_SERVER} \
--producer.config ~/client.properties --topic ${topic} <<< "${textFromFile}"
echo "configuration added to $topic"
done
What would be the reason sed doesn't convert the string to single line? Or how else could I change the call to sed to achieve the conversion from multiline to a single line?
The sed version is
Thank you
UPDATE
Sample Input
{
"eventType": "Dashboard_Configuration",
"version": 1,
"environments": [
"dev",
"qa",
"stage",
"prod"
],
"events": [
{
"sourceSystem": "FINTRX",
"events": [
{
"parentEventName": "Benefit",
"stageCount": 3,
"stages": [
{
"order": 1,
"stageName": "IngestStage",
"stageType": "INGEST"
},
{
"order": 2,
"stageName": "AvroStage",
"stageType": "AVRO"
},
{
"order": 3,
"stageName": "ReconciliationStage",
"stageType": "RECONCILIATION"
}
]
}
]
}
]
}
Expected Output
{ "eventType": "Dashboard_Configuration", "version": 1, "environments": [ "dev", "qa", "stage", "prod" ], "events": [ { "sourceSystem": "FINTRX", "events": [ { "parentEventName": "Policy Financial Transaction", "stageCount": 3, "stages": [ { "order": 1, "stageName": "IngestStage", "stageType": "INGEST" }, { "order": 2, "stageName": "AvroStage", "stageType": "AVRO" }, { "order": 3, "stageName": "ReconciliationStage", "stageType": "RECONCILIATION" } ] } ] } ] }
The Actual Output
Same as input - for the last iteration - some other iteration it was dropping the r and n from the string - but putting it on one line
Thanks
sed is designed to work on lines, it wouldn't do a good job working across lines in most cases.
A much easier way to delete newlines is tr -d '\r\n':
$ printf "PRE"; printf "%s\n" a b c; printf "POST"; echo
PREa
b
c
POST
$ printf "PRE"; printf "%s\n" a b c | tr -d '\r\n'; printf "POST"; echo
PREabcPOST
You can use bash's built-in string search and replace to delete newline characters
# Delete any newline character
read -r -d '' textFromFile <"${topicconfig[$topic]}"
textFromFile="${textFromFile//[$'\n\r']/}"
Now if you want to replace newlines with a space using the same feature
shopt -s extglob # Needed for extended glob patterns
read -r -d '' textFromFile <"${topicconfig[$topic]}"
# Replace one or more newlines by a single space
textFromFile="${textFromFile//+([$'\n\r'])/ }"
Replace: (sed)
textFromFile=$(sed -e 's/[\n|\r]$//g' "${topicconfig[$topic]}")
With: (awk)
textFromFile=$(awk '{printf $0}' "${topicconfig[$topic]}")
With: (root's tr solution)
textFromFile=$(tr -d '\r\n' < "${topicconfig[$topic]}")
NOTE: Neither of these ideas deal with the management of 'extra' white space, eg, removing leading spaces/tabs in a line and replacing with a single space (as shown in OP's expected output)
I have this output variable
OUTPUT=$(echo $ZONE_LIST | jq -r '.response | .data[]')
The Output:
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
How can I create a loop for this two values?
MyLoop
echo "$content - $id"
done
I tried this, but then I get a loop through every single value
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]'); do
echo $k
done
EDIT 1:
My complete JSON:
{
"errors": [],
"metadata": {
"transactionId": "",
},
"response": {
"data": [
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"abcd\"",
"id": "myID1",
"lastChangeDate": "2020-09-05T10:57:11Z",
},
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"abc\"",
"id": "myID2",
"lastChangeDate": "2020-09-05T07:08:15Z",
}
],
"limit": 10,
"page": 1,
"totalEntries": 2,
},
"status": "success",
"warnings": []
}
Now I need a loop for data, because I need it for a curl
The curl NOW:
curl -s -v -X POST --data '{
"deleteEntries": [
Data_from_json
]
}' https://URL_to_Update 2>/dev/null)
Now I want to create a new variable from my JSON data. My CURL should look like this at the end:
curl -s -v -X POST --data '{
"deleteEntries": [
{
"readID": "myID1",
"date": "2020-09-05T10:57:11Z", <--Value from addDate
"content": "abcd"
},
{
"readID": "myID2",
"date": "2020-09-05T06:58:52Z", <--Value from addDate
"content": "abc"
}
]
}' https://URL_to_Update 2>/dev/null)
Something like:
#!/usr/bin/env bash
while IFS=$'\37' read -r -d '' id content; do
echo "$id" "$content"
done < <(
jq -j '.response | .data[] | .id + "\u001f" + .content + "\u0000"' \
<<<"$ZONE_LIST"
)
jq -j: Forces a raw output from jq.
.id + "\u001f" + .content + "\u0000": Assemble fields delimited by ASCII FS (Hexadecimal 1f or Octal 37), and end record by a null character.
It then becomes easy and reliable to iterate over null delimited records by having read -d '' (null delimiter).
Fields id content are separated by ASCII FS, so just set the Internal Field Separator IFS environment variable to the corresponding octal IFS=$'37' before reading.
The first step is to realize you can turn the set of fields into an array like this using a technique like this:
jq '(.accountId + "," + .addDate)'
Now you can update your bash loop:
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]' | jq '(.content + "," + .id)'); do
echo $k
done
There is probably a way to combine the two jq commands but I don't have your original json data for testing.
UPDATE - inside the loop you can parse the comma-delimited string into separate fields. This are more efficient ways to handle this task but I prefer simplicity.
ID=$(echo $k | cut -d',' -f1)
PRIORITY=$(echo $k | cut -d',' -f2)
echo "ID($ID) PRIORITY($PRIORITY)"
Try this.
for k in $(echo $ZONE_LIST | jq -rc '.response | .data[]'); do
echo $k|jq '.content + " - " + .id' -r
done
I have this example test.json file:
{
"folder": {
"name": "Desktop",
"path": "Users/user/Desktop",
"executable": false
}
}
I am trying to read the object contained in 'folder' into a variable $var as an array. Desired output - $var contains the following:
[
{
"name": "Desktop",
"path": "Users/user/Desktop",
"executable": false
}
]
It seemed relatively straightforward. jq '.folder' test.json to read in the folder value, and then pipe the output using the -s flag to get it enclosed in an array. This command executes just fine:
jq '.folder' test.json | jq -s
However, when I try to save the result to a variable, it throws an error displaying the jq usage statement. What am I missing? Is there a different way to accomplish this?
var=$(jq '.folder' test.json | jq -s)
echo $var
I also found out that this works fine in zsh but throws an error in bash. Not sure if that's relevant and whether it has to do with different versions of jq
Is this what you were expecting ?
$ var="$(jq "[.folder]" test.json)"
$ echo "$var"
[
{
"name": "Desktop",
"path": "Users/user/Desktop",
"executable": false
}
]
i'm trying to build script that takes specific attribute value and store it in the array , this is the following JSON file:
[
{
"id": 1,
"name": "myna",
"description": "Simple Question",
"speaker": "USER",
},
{
"all_Id's": [
"11111"
],
"user": "me",
},
{
"id": 2,
"name": "mry",
"description": "Simple",
"speaker": "aaa",
}
]
as you see object in json file don't have the same attributes so i'm looking only on object has "name " attribute,the following script reads the Json file and return the values of attribute name only ,but i build something wrond as theERROR always on the "{" of the last object in file I don't know why , what i am i doing wrong?
the expected output is : [myna, mry]
#!/bin/bash
declare -a OB_I=()
declare counter1=0
jq -r '.name' file.json ; while read -r val ; do
if [[ ! $val ]]
then
OB_I[$counter]=$val ;
counter=$((counter+1));
fi
done;
$ printf '%s\n' "${OB_I[#]}"
The input of jq is a list, which doesn't have any keys, let alone one named name. You want
jq -r '.[].name'
instead.
Unrelated, you don't need the variable counter. You can simply append to your array with OB_I+=("$val").