I'm using jq to read some data from a JSON file.
after=`cat somefile.json | jq '.after[]'`
returns something like this:
"some value" "another value" "something else"
Basically a list of quoted strings. I now need to convert these strings into one string formatted like
"some value; another value; something else;"
I've tried a lot of combinations of for loops to try and get this working and nothing quite works.
Anyone know how this can be done? Cheers!
use sed:
sed -e 's/" /; /g; s/ "/ /g; s/"$/;"/' <<< '"some value" "another value" "something else"'
OUTPUT:
"some value; another value; something else;"
use sed s command for replacing the desire value
Thanks all! I actually decided to dig deeper into the jq docs to see if I could simply leverage it to do what I want.
after=`cat somefile.json | jq -c -r '.after[] + "; "'` | tr -d '\n'
This ended up working very well. Thanks for the sed version though! Always good to see another working solution.
Assuming .after[] returns the list of strings you describe, you can do this entirely with jq using join to format them as follows:
[ .after[] ] | join("; ") + ";"
Related
I got a big json file with text to be modify in each value. If we look a the first entry, it looks like :
$ cat description6fr.json | jq '.[0].fr'
Something\nSomethings\n\nSomething\n\nSomething
How to cut everything before the first \n\n ? I did try many things with sed and awk, but help is welcome. Thanks!
You could use index to find the match and then slice it:
jq '.[0].fr | .[index("\n\n") + 2:]' description6fr.json
If the string does not match \n\n, this will truncate the first two characters, so you might prefer something like:
jq '.[0].fr| if index("\n\n") then .[index("\n\n") + 2:] else . end' description6fr.json
you can parameterize that with:
jq --argjson s '"\n\n"' 'if index($s) then .[index($s) + 2:] else . end'
I have a json string like this,
rssample='{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}'
I want to update the NAME and Value in it using the below variables..
name="testname"
newvalue="heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager"
So I tried using sed,
newrs=$(sed -E "s/"NAME":""/"NAME":"$name"/g" <<< "$rssample")
newrs1=$(sed -E "s/"Value":""/"Value":"\$newvalue"/g" <<< "$newrs")
I am expecting below as output,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":"\"heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager\""}], "Type": "TXT","NAME":"testname","TTL": 300}}]}
But I am empty Name.And getting error for value as,
sed: 1: "s/Value:/Value:"heritag ...": bad flag in substitute command: 'o'
My output is,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}
Please let me know how to fix this? is using sed good idea or jq?
It's generally safer, and therefore often better, to use a JSON-aware tool rather than sed when editing JSON. Using jq, one possibility would be:
jq --arg name "$name" --arg newvalue "$newvalue" '
.Changes[0].ResourceRecordSet |=
(.NAME=$name
| .ResourceRecords[0].Value = $newvalue)' <<< "$rssample"
A free-form approach
jq --arg name "$name" --arg newvalue "$newvalue" '
walk(if type == "object"
then if has("NAME") then .NAME=$name else . end
| if has("Value") then .Value = $newvalue else . end
else . end)' <<< "$rssample"
I have a variable called TR_VERSION that is a JSON list of version numbers that looks something like this:
[
"1.0.1",
"1.0.2",
"1.0.3"
]
I would like to strip all of the JSON specific characters - [, ", , and ]. The following code works but it would be great to consolidate to one sed call instead of three.
TR_VERSION=$(echo $VERSION \
| sed 's|[",]||g' \
| sed 's/\[//' \
| sed 's/\]//')
Thanks for the answers!
Never ever use sed to parse json.
This is the way to go:
$ jq -r '.[]' < file.json
Output as expected
1.0.1
1.0.2
1.0.3
If you just want to remove all ", ,, [ and ] chars you may use
TR_VERSION=$(echo "$VERSION" | sed 's/[][",]//g')
Or,
TR_VERSION=$(sed 's/[][",]//g' <<< "$VERSION")
The [][",] pattern matches ], [, " or , chars.
If you really want to avoid a JSON parer, there is still no need to use sed. You could also do it by
TR_VERSION=$(tr -d '[]",' <<<$VERSION)
which, IMHO, is slightly better readable than the sed counterpart.
I would like to convert a list into JSON array. I'm looking at jq for this but the examples are mostly about parsing JSON (not creating it). It would be nice to know proper escaping will occur. My list is single line elements so the new line will probably be the best delimiter.
I was also trying to convert a bunch of lines into a JSON array, and was at a standstill until I realized that -s was the only way I could handle more than one line at a time in the jq expression, even if that meant I'd have to parse the newlines manually.
jq -R -s -c 'split("\n")' < just_lines.txt
-R to read raw input
-s to read all input as a single string
-c to not pretty print the output
Easy peasy.
Edit: I'm on jq ≥ 1.4, which is apparently when the split built-in was introduced.
--raw-input, then --slurp
Just summarizing what the others have said in a hopefully quicker to understand form:
cat /etc/hosts | jq --raw-input . | jq --slurp .
will return you:
[
"fe00::0 ip6-localnet",
"ff00::0 ip6-mcastprefix",
"ff02::1 ip6-allnodes",
"ff02::2 ip6-allrouters"
]
Explanation
--raw-input/-R:
Don´t parse the input as JSON. Instead, each line of text is passed
to the filter as a string. If combined with --slurp, then the
entire input is passed to the filter as a single long string.
--slurp/-s:
Instead of running the filter for each JSON object in the input,
read the entire input stream into a large array and run the filter
just once.
You can also use jq -R . to format each line as a JSON string and then jq -s (--slurp) to create an array for the input lines after parsing them as JSON:
$ printf %s\\n aa bb|jq -R .|jq -s .
[
"aa",
"bb"
]
The method in chbrown's answer adds an empty element to the end if the input ends with a linefeed, but you can use printf %s "$(cat)" to remove trailing linefeeds:
$ printf %s\\n aa bb|jq -R -s 'split("\n")'
[
"aa",
"bb",
""
]
$ printf %s\\n aa bb|printf %s "$(cat)"|jq -R -s 'split("\n")'
[
"aa",
"bb"
]
If the input lines don't contain ASCII control characters (which have to be escaped in strings in valid JSON), you can use sed:
$ printf %s\\n aa bb|sed 's/["\]/\\&/g;s/.*/"&"/;1s/^/[/;$s/$/]/;$!s/$/,/'
["aa",
"bb"]
Update: If your jq has inputs you can simply write:
jq -nR [inputs] /etc/hosts
to produce a JSON array of strings. This avoids having to read the text file as a whole.
I found in the man page for jq and through experimentation what seems to me to be a simpler answer.
$ cat test_file.txt | jq -Rsc '. / "\n" - [""]'
["aa","bb"]
The -R is to read without trying to parse json, the -s says to read all of the input as one string, and the -c is for one-line output - not necessary, but it's what I was looking for.
Then in the string I pass to jq, the '.' says take the input as it is. The '/ \n' says to divide the string (split it) on newlines. The '- [""]' says to remove from the resulting array any empty strings (resulting from an extra newline at the end).
It's one line and without any complicated constructs, using just simple built in jq features.
I have a files with many lines like:
lily weisy
I want to extract www.youtube.com/user/airuike and lily weisy, and then I also want to separate airuike from www.youtube.com/user/
so I want to get 3 strings: www.youtube.com/user/airuike, airuike and lily weisy
how to achieve this? thanks
do this:
sed -e 's/.*href="\([^"]*\)".*>\([^<]*\)<.*/link:\1 name:\2/' < data
will give you the first part. But I'm not sure what you are doing with it after this.
Since it is html, and html should be parsed with a html parser and not with grep/sed/awk, you could use the pattern matching function of my Xidel.
xidel yourfile.html -e '<a class="yt-uix-sessionlink yt-user-name " dir="ltr">{$link := #href, $user := substring-after($link, "www.youtube.com/user/"), $name:=text()}</a>*'
Or if you want a CSV like result:
xidel yourfile.html -e '<a class="yt-uix-sessionlink yt-user-name " dir="ltr">{string-join((#href, substring-after(#href, "www.youtube.com/user/"), text()), ", ")}</a>*' --hide-variable-names
It is kind of sad, that you also want to have the airuike string, otherwise it could be as simple as
xidel /yourfile.html -e '{$name}*'
(and you were supposed to be able to use xidel '{$name}*', but it seems I haven't thought the syntax through. Just one error check and it is breaking everything. )
$ awk '{split($0,a,/(["<>]|:\/\/)/); u=a[4]; sub(/.*\//,"",a[4]); print u,a[4],a[12]}' file
www.youtube.com/user/airuike airuike lily weisy
I think something like this must work
while read line
do
href=$(echo $line | grep -o 'http[^"]*')
user=$(echo $href | grep -o '[^/]*$')
text=$(echo $line | grep -o '[^>]*<\/a>$' | grep -o '^[^<]*')
echo href: $href
echo user: $user
echo text: $text
done < yourfile
Regular expressions basics: http://en.wikipedia.org/wiki/Regular_expression#POSIX_Basic_Regular_Expressions
Upd: checked and fixed