sed to convert from multiline to single does not work - bash

I am working on a shell script that would read filenames based on a topic name, read the file content, convert the text to a single line and post the message to kafka using kafka consumer.
For converting the contents of the json file, I am trying to use sed and seen many examples and tried them all. But none of them seem to convert the multiline text to single line so I could use it to post to kafka topic
This is the script I am trying to write
declare -A topicconfig
while read -r topic configfile; do
topicconfig["$topic"]="$configfile"
done <config_params.txt
for topic in "${!topicconfig[#]}"; do
echo "filename : ${topicconfig[$topic]}"
textFromFile=$(sed -e 's/[\n|\r]$//g' "${topicconfig[$topic]}")
echo "textFromFile : $textFromFile"
/bin/kafka-console-producer --bootstrap-server=${BOOTSTRAP_SERVER} \
--producer.config ~/client.properties --topic ${topic} <<< "${textFromFile}"
echo "configuration added to $topic"
done
What would be the reason sed doesn't convert the string to single line? Or how else could I change the call to sed to achieve the conversion from multiline to a single line?
The sed version is
Thank you
UPDATE
Sample Input
{
"eventType": "Dashboard_Configuration",
"version": 1,
"environments": [
"dev",
"qa",
"stage",
"prod"
],
"events": [
{
"sourceSystem": "FINTRX",
"events": [
{
"parentEventName": "Benefit",
"stageCount": 3,
"stages": [
{
"order": 1,
"stageName": "IngestStage",
"stageType": "INGEST"
},
{
"order": 2,
"stageName": "AvroStage",
"stageType": "AVRO"
},
{
"order": 3,
"stageName": "ReconciliationStage",
"stageType": "RECONCILIATION"
}
]
}
]
}
]
}
Expected Output
{ "eventType": "Dashboard_Configuration", "version": 1, "environments": [ "dev", "qa", "stage", "prod" ], "events": [ { "sourceSystem": "FINTRX", "events": [ { "parentEventName": "Policy Financial Transaction", "stageCount": 3, "stages": [ { "order": 1, "stageName": "IngestStage", "stageType": "INGEST" }, { "order": 2, "stageName": "AvroStage", "stageType": "AVRO" }, { "order": 3, "stageName": "ReconciliationStage", "stageType": "RECONCILIATION" } ] } ] } ] }
The Actual Output
Same as input - for the last iteration - some other iteration it was dropping the r and n from the string - but putting it on one line
Thanks

sed is designed to work on lines, it wouldn't do a good job working across lines in most cases.
A much easier way to delete newlines is tr -d '\r\n':
$ printf "PRE"; printf "%s\n" a b c; printf "POST"; echo
PREa
b
c
POST
$ printf "PRE"; printf "%s\n" a b c | tr -d '\r\n'; printf "POST"; echo
PREabcPOST

You can use bash's built-in string search and replace to delete newline characters
# Delete any newline character
read -r -d '' textFromFile <"${topicconfig[$topic]}"
textFromFile="${textFromFile//[$'\n\r']/}"
Now if you want to replace newlines with a space using the same feature
shopt -s extglob # Needed for extended glob patterns
read -r -d '' textFromFile <"${topicconfig[$topic]}"
# Replace one or more newlines by a single space
textFromFile="${textFromFile//+([$'\n\r'])/ }"

Replace: (sed)
textFromFile=$(sed -e 's/[\n|\r]$//g' "${topicconfig[$topic]}")
With: (awk)
textFromFile=$(awk '{printf $0}' "${topicconfig[$topic]}")
With: (root's tr solution)
textFromFile=$(tr -d '\r\n' < "${topicconfig[$topic]}")
NOTE: Neither of these ideas deal with the management of 'extra' white space, eg, removing leading spaces/tabs in a line and replacing with a single space (as shown in OP's expected output)

Related

jq: iterate over every element of list and replace it with value

I've got this json-file:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"23423565",
"23552352",
"08789089"
]
}
}
I need to iterate over every element of an list (upper_one), and replace it with other value.
I've tried this code:
#/bin/bash
for product in $(cat first-shop.json| jq -r '.shelves.upper_one[]')
do
cat first-shop.json| jq --arg id "$((1 + $RANDOM % 10))" --arg product "$product" -r '.shelves.upper_one[]|select(. == $product)|= $id'
done
But I got this kind of output:
1
23552352
08789089
23423565
10
08789089
23423565
23552352
7
Is it possible to iterate over list with jq, replace values with value from another function (like $id in the code), and print the whole final json with substituted values?
I need this kind of output:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
not just elements of "upper_one" list thrice.
You could try the following script :
#!/usr/bin/env bash
for product in $(jq -r '.shelves.upper_one[]' input.json)
do
id="$((1 + $RANDOM % 10))"
newIds+=("$id")
done
jq '.shelves.upper_one = $ARGS.positional' input.json --args "${newIds[#]}"
IMHO its better to use some scripting language and manipulate objects programmatically. If bash and jq is your only option - this do the job though not nice
$ jq '.shelves.upper_one[] |= (sub("23423565";"1") | sub("23552352";"10") | sub("08789089";"7"))' your.json
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
consider conversion to numbers with | tonumber

Loop json Output in shell script

I have this output variable
OUTPUT=$(echo $ZONE_LIST | jq -r '.response | .data[]')
The Output:
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
How can I create a loop for this two values?
MyLoop
echo "$content - $id"
done
I tried this, but then I get a loop through every single value
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]'); do
echo $k
done
EDIT 1:
My complete JSON:
{
"errors": [],
"metadata": {
"transactionId": "",
},
"response": {
"data": [
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"abcd\"",
"id": "myID1",
"lastChangeDate": "2020-09-05T10:57:11Z",
},
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"abc\"",
"id": "myID2",
"lastChangeDate": "2020-09-05T07:08:15Z",
}
],
"limit": 10,
"page": 1,
"totalEntries": 2,
},
"status": "success",
"warnings": []
}
Now I need a loop for data, because I need it for a curl
The curl NOW:
curl -s -v -X POST --data '{
"deleteEntries": [
Data_from_json
]
}' https://URL_to_Update 2>/dev/null)
Now I want to create a new variable from my JSON data. My CURL should look like this at the end:
curl -s -v -X POST --data '{
"deleteEntries": [
{
"readID": "myID1",
"date": "2020-09-05T10:57:11Z", <--Value from addDate
"content": "abcd"
},
{
"readID": "myID2",
"date": "2020-09-05T06:58:52Z", <--Value from addDate
"content": "abc"
}
]
}' https://URL_to_Update 2>/dev/null)
Something like:
#!/usr/bin/env bash
while IFS=$'\37' read -r -d '' id content; do
echo "$id" "$content"
done < <(
jq -j '.response | .data[] | .id + "\u001f" + .content + "\u0000"' \
<<<"$ZONE_LIST"
)
jq -j: Forces a raw output from jq.
.id + "\u001f" + .content + "\u0000": Assemble fields delimited by ASCII FS (Hexadecimal 1f or Octal 37), and end record by a null character.
It then becomes easy and reliable to iterate over null delimited records by having read -d '' (null delimiter).
Fields id content are separated by ASCII FS, so just set the Internal Field Separator IFS environment variable to the corresponding octal IFS=$'37' before reading.
The first step is to realize you can turn the set of fields into an array like this using a technique like this:
jq '(.accountId + "," + .addDate)'
Now you can update your bash loop:
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]' | jq '(.content + "," + .id)'); do
echo $k
done
There is probably a way to combine the two jq commands but I don't have your original json data for testing.
UPDATE - inside the loop you can parse the comma-delimited string into separate fields. This are more efficient ways to handle this task but I prefer simplicity.
ID=$(echo $k | cut -d',' -f1)
PRIORITY=$(echo $k | cut -d',' -f2)
echo "ID($ID) PRIORITY($PRIORITY)"
Try this.
for k in $(echo $ZONE_LIST | jq -rc '.response | .data[]'); do
echo $k|jq '.content + " - " + .id' -r
done

Handling json object with special characters in jq

I have a json object with below element,
rsrecords="{
"ResourceRecords": [
{
"Value": "\"heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api""
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}"
And in my bash script,I have below code snippet,
jq -r '.[] | .Name ,.ResourceRecords[0].Value' <<< "$rsrecords" | \
while read -r name; read -r value; do
echo $name
Output is printed as,
\052.apiconsumer.alpha.sandbox.test.net.
But I am expecting it to print as \\052.apiconsumer.alpha.sandbox.test.net., which is , as it is "Name" from the json object..
How can this be done?
Before getting to the heart of the matter, please note that
the sample data as given is a bit of a mishmash, so I'll assume you meant something like:
rsrecords='
{
"ResourceRecords": [
{
"Value": "heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api"
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}
'
Your jq query does not match the above JSON, so I'll assume you intended the query to be simply:
.Name, .ResourceRecords[0].Value
In any case, with the above JSON, the bash commands:
jq -r '.Name, .ResourceRecords[0].Value' <<< "$rsrecords" |
while read -r name; read -r value; do
echo "$name"
done
yields:
\052.apiconsumer.alpha.sandbox.test.net.
This is correct, because the JSON string "\\X" is an encoding of the raw string: \X
If you want to see the JSON string, then invoke jq without the -r option. If you want to invoke jq with the -r option and want to see two backslashes, you will have to encode them as four backslashes in your JSON.

cannot call bash environment variable inside jq

In the below script, I am not able to successfully call the "repovar" variable in the jq command.
cat quayrepo.txt | while read line
do
export repovar="$line"
jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), $repovar"' severity.json > volume.csv
done
The script uses a text file to loop through the repo names
quayrepo.txt---> file has the list of names in this case the file has a value of "Reponame1"
sample input severity.json file:
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
desired output:
elfutils, 0.168-1, Medium, Reponame1
Required output: I need to retrieve the value of my environment variable as the last column in my output csv file
You need to surround $repovar with parenthesis, as the other values
repovar='qweqe'; jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), \($repovar)"' tmp.json
Result:
elfutils, 0.168-1, qweqe
There's no need for the export.
#!/usr/bin/env bash
while read line
do
jq -r --arg repovar "$line" '.data.Layer.Features[] | .Name + ", " + .Version + ", " + $repovar' severity.json
done < quayrepo.txt > volume.csv
with quayrepo.txt as
Reponame1
and severity.json as
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
produces volume.csv containing
elfutils, 0.168-1, Reponame1
To #peak's point, changing > to >> in ...severity.json >> volume.csv will create a multi-line csv instead of just overwriting until the last line
You don't need a while read loop in bash at all; jq itself can loop over your input lines, even when they aren't JSON, letting you run jq only once, not once per line in quayrepo.txt.
jq -rR --slurpfile inJson severity.json <quayrepo.txt >volume.csv '
($inJson[0].data.Layer | .Features[]) as $features |
[$features.Name, $features.Version, .] |
#csv
'
jq -R specifies raw input, letting jq directly read lines from quayrepo.txt into .
jq --slurpfile varname filename.json reads filename.json into an array of JSON objects parsed from that file. If the file contains only one object, one needs to refer to $varname[0] to refer to it.
#csv converts an array to a CSV output line, correctly handling data with embedded quotes or other oddities that require special processing.

why i get this error jq: error Cannot index array with string from json file?

i'm trying to build script that takes specific attribute value and store it in the array , this is the following JSON file:
[
{
"id": 1,
"name": "myna",
"description": "Simple Question",
"speaker": "USER",
},
{
"all_Id's": [
"11111"
],
"user": "me",
},
{
"id": 2,
"name": "mry",
"description": "Simple",
"speaker": "aaa",
}
]
as you see object in json file don't have the same attributes so i'm looking only on object has "name " attribute,the following script reads the Json file and return the values of attribute name only ,but i build something wrond as theERROR always on the "{" of the last object in file I don't know why , what i am i doing wrong?
the expected output is : [myna, mry]
#!/bin/bash
declare -a OB_I=()
declare counter1=0
jq -r '.name' file.json ; while read -r val ; do
if [[ ! $val ]]
then
OB_I[$counter]=$val ;
counter=$((counter+1));
fi
done;
$ printf '%s\n' "${OB_I[#]}"
The input of jq is a list, which doesn't have any keys, let alone one named name. You want
jq -r '.[].name'
instead.
Unrelated, you don't need the variable counter. You can simply append to your array with OB_I+=("$val").

Resources