I use jq to process json str, but the shell does not retain double quotes. I can't add an escape character because the json str is externally sent.I want to keep the double quotes of the original string。
The json string is dynamically generated, the data content is undefined, I can't use sed to add double quotes
# The json_str is externally sent.
# Assume that the content is "{"name": "John", "age": 0}"
# I want get the name
echo "$json_str" | jq -r ".name"
I expect the output is "John", but the actual output is
parse error: Invalid literal at line 1, column 6
You can either use single quotes
json_str='{"name", "John", "age": 22}'
or escape the double quotes
json_str="{\"name\", \"John\", \"age\": 22}"
Note that this answer applies to the original version of the question.
I expect the output is "John"
Apart from the error introduced by your test case, the use of option -r is the issue:
· --raw-output / -r:
With this option, if the filter´s result is a string then it will
be written directly to standard output rather than being formatted
as a JSON string with quotes.
If you don't want colored output, you can use -M instead:
· --colour-output / -C and --monochrome-output / -M:
By default, jq outputs colored JSON if writing to a terminal. You
can force it to produce color even if writing to a pipe or a file
using -C, and disable color with -M.
The quote in the JSON example are part of the JSON layout, not the content of the files.
When you want to have the quotes, you can use
echo "${json_str}" | jq -r ".name" | sed 's/.*/"&"/'
or
name=$(echo "${json_str}" | jq -r ".name" | sed 's/.*/"&"/')
Related
The below code is producing the expected results with username.
es_eh="$(aws cloudtrail --region us-east-1 lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=route53.amazonaws.com --max-items 50 --start-time "${start_date}" --end-time "${end_date}" --output json)"
for row in $(echo "${es_eh}" | jq -r '.Events[] | #base64'); do
echo "${row}" | base64 --decode | jq -r '.Username'
done
I didn't understand the purpose of doing base64 encode and then doing decode of the same string inside loop to retrieve username.
This is not working when I remove base64 encode and decode.
for row in $(echo "${es_eh}" | jq -r '.Events[]'); do
echo "${row}"| jq -r '.Username'
done
Without the encoding, the output of the first jq is more than one row. The loop iterates over the lines and fails, as none of them contains a valid JSON. With the | #base64, each subobject is encoded to a single row, inflated back to a full JSON object by base64 --decode.
To see the rows, try outputting $row before processing it.
When you use $( ) without quotes around it, the result gets split into "words", but the shell's definition of a "word" is almost never what you want (and certainly has nothing to do with the json entries you want it split into). This sort of thing is why you should almost never use unquoted expansions.
Converting the output entries to base64 makes them wordlike enough that shell word splitting actually does the right thing. But note: some base64 encoders split their output into lines, which would make each line be treated as a separate "word". If jq's base64 encoding did this, this code would fail catastrophically on large events.
Transforming the for loop into a while loop should fix the problem :
while read -r row; do
echo "${row}" | jq -r '.Username'
done < <(echo "${es_eh}" | jq -c -r '.Events[]')
Note that in the outer jq, I used option -c to put output in a single ine.
How can I write a regex in a shell script that would target only the targeted substring between two given values? Give the example
https://www.stackoverflow.com
How can I match only the ":" in between "https" and "//".
If possible please also explain the approach.
The context is that I need to prepare a file that would fetch a config from the server and append it to the .env file. The response comes as JSON
{
"GRAPHQL_URL": "https://someurl/xyz",
"PUBLIC_TOKEN": "skml2JdJyOcrVdfEJ3Bj1bs472wY8aSyprO2DsZbHIiBRqEIPBNg9S7yXBbYkndX2Lk8UuHoZ9JPdJEWaiqlIyGdwU6O5",
"SUPER_SECRET": "MY_SUPER_SECRET"
}
so I need to adjust it to the .env syntax. What I managed to do this far is
#!/bin/bash
CURL_RESPONSE="$(curl -s url)"
cat <<< ${CURL_RESPONSE} | jq -r '.property.source' | sed -r 's/:/=/g;s/[^a-zA-Z0-9=:_/-]//g' > .env.test
so basically I fetch the data, then extract the key I am after with jq, and then I use sed to first replace all ":" to "=" and after that I remove all the quotations and semicolons and white spaces that comes from JSON and leave some characters that are necessary.
I am almost there but the problem is that now my graphql url (and only other) would look like so
https=//someurl/xyz
so I need to replace this = that is in between https and // back with the colon.
Thank you very much #Nic3500 for the response, not sure why but I get error saying that
sed: 1: "s/:/=/g;s#https\(.*\)// ...": \1 not defined in the RE
I searched SO and it seems that it should work since the brackets are escaped and I use -r flag (tried -E but no difference) and I don't know how to apply it. To be honest I assume that the replacement block is this part
#\1#
so how can I let this know to what character should it be replaced?
This is how I tried to use it
#!/bin/bash
CURL_RESPONSE="$(curl -s url)"
cat <<< ${CURL_RESPONSE} | jq -r '.property.source' | sed -r 's/:/=/g;s#https\(.*\)//.*#\1#;s/[^a-zA-Z0-9=:_/-]//g' > .env.test
Hope with this context you would be able to help me.
echo "https://www.stackoverflow.com" | sed 's#https\(.*\)//.*#\1#'
:
sed operator s/regexp/replacement/
regexp: https\(.*)//.*. So "https" followed by something (.*), followed by "//", followed by anything else .*
the parenthesis are back slashed since they are not part of the pattern. They are used to group a part of the regex for the replacement part of the s### operator.
replacement: \1, means the first group found in the regex \(.*\)
I used s###, but the usual form is s///. Any character can take the place of the / with the s operator. I used # as using / would have been confusing since you use / in the url.
The problem is that your sed substitutions are terribly imprecise. Anyway, you want to do it in jq instead, where you have more control over which parts you are substituting, and avoid spawning a separate process for something jq quite easily does natively in the first place.
curl -s url |
jq -r '.property.source | to_entries[] |
"\(.key)=\"\(.value\)\""' > .env.test
Tangentially, capturing the output of curl into a variable just so you can immediately cat it once to standard output is just a waste of memory.
Guys i have a string like this:
variable='<partyRoleId>12345</partyRoleId>'
what i want is to extract the value so the output is 12345.
Note the tag can be in any form:
<partyRoleId> or <ns1:partyRoleId>
any idea how to get the tag value using grep or sed only?
Use an XML parser to extract the value:
echo "$variable" | xmllint -xpath '*/text()' -
You probably should use it for the whole XML document instead of extracting a single line from it into a variable, anyway.
to use only grep, you need regexp to find first closing brackets and cut all digits:
echo '<partyRoleId>12345</partyRoleId>'|grep -Po ">\K\d*"
-P means PCRE
-o tells to grep to show only matched pattern
and special \K tells to grep cut off everything before this.
I am trying to get my tunnel ID from sauce labs... To get this I am doing the following:
SAUCE_TUNNELL_IDENTIFIER=$(curl https://saucelabs.com/rest/v1/myusername/tunnels -u myusername:mykey)
This returns ["my-tunnel-id"]
So I want to get rid of the [] and export my tunnel id..
Here is the export and removal of []
export SAUCE_TUNNELL_IDENTIFIER=$(echo $SAUCE_TUNNELL_IDENTIFIER | awk '{print substr($0, 2, length($0) -2)}')
When I do an export -p, I see it like this "\"my-tunnel-id\""
But if I just do an echo $SAUCE_TUNNELL_IDENTIFIER, I do not see the exta "\ in my variable. Where is this coming from? Also, any help making this into one better command would also be appreciated! :)
The server is returning a JSON array; you could use something like jq to extract just the single element in the array, but for something this simple, you can just use parameter expansion.
$ SAUCE_TUNNELL_IDENTIFIER=$(curl https://saucelabs.com/rest/v1/myusername/tunnels -u myusername:mykey)
$ SAUCE_TUNNELL_IDENTIFIER=${SAUCE_TUNNELL_IDENTIFIER//[]\"[]}
$ echo "$SAUCE_TUNNELL_IDENTIFIER"
my-tunnel-id
Update:
I assumed the identifier could not contain brackets or quotes. A safer, though longer, alternative is to strip each outer character one by one.
$ STI=$(curl ...)
$ STI=${STI#[}
$ STI=${STI#\"}
$ STI=${STI%]}
$ STI=${STI%\"}
and at this point I may as well show how to use jq to do this:
$ curl ... | jq -r '.[0]'
my-tunnel-id
. represents the incoming JSON; [0] extracts the first element; -r tells jq to output the raw string instead of the quoted JSON string (my-tunnel-id instead of "my-tunnel-id").
I'm using JQ https://stedolan.github.io/jq/ to work in bash with my json and when I read the json is throwing me an error
parse error: Invalid numeric literal at line 2, column 5=
Since my json has some comments
// comment
"spawn": {}
I've been seen looking the options and I cannot find any option to fix the problem. Any idea how to solve it?
JSON and thus jq do not support comments (in the usual sense) in JSON input. The jq FAQ lists a number of tools that can be used to remove comments, including jsonlint, json5, and any-json. I'd recommend one that can act as a filter.
See https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json for links and further details.
——
It might be worth pointing out that jq can be used to process JSON with #-style comments, at least if the JSON is not too large to be processed as a jq program. For example, you could use jq with the -f option to read a JSON file as a jq program.
Remove them; JSON does not support comments.
(JSON is defined here; you can see a briefer description of the grammar here.)
I found https://github.com/sindresorhus/strip-json-comments-cli which allows you to do:
cat my_json_with_comments.json | strip-json-comments | jq .
Can be stripped out using sed, eg to remove lines beginning with '//':
cat test.json | sed 's/^ *\/\/.*//' | jq <>commands>
sed is a pass-through/stream editor, in this case it's substituting nothing ( // ) for lines that begin with '//'; '//' must be escaped with a backslash character since the '/' is used by sed as a delimiter.
With this sed, you can remove:
Empty lines
Comments, even in the format "key": "value" //my comment
Eventually, json will be output, which can be processed without problems using jq
sed '/^[[:blank:]]*#/d;s/\/\/.*//' my.json | jq '.<your_block>'
reference: https://unix.stackexchange.com/a/157619