I'm trying to inline a jq construct that itself requires pipes. I suspect I'm running into issues because bash is treating them as bash-level pipes, rather than part of the jq.
Testing at jqplay.org, this .[1] | [.timeEnded, .lifecycleState] | flatten gets me the result I need.
Trying to embed that in bash, I am trying to do something like:
status=$(curl -X GET <URL> | jq -r -c '.[1] | [.timeEnded, .lifecycleState] | flatten' | awk -F, '{print $2}' | sed 's/"//g')
With no escaping the pipes within the jq, I get
[.timeEnded,: command not found
I tried to escape those pipes as jq -r -c '.[1] \| [.timeEnded, .lifecycleState] \| flatten' but that gets me a jq syntax error:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.[1] \| [.timeEnded, .lifecycleState] \| flatten
jq: 1 compile error
Wrapping the entire jq command in double quotes (as well as the escape chars) gave me the same syntax error. I'm sure there's probably an easy answer here, but jq is new to me.
Any help would be appreciated.
I clearly suspect that you have an unbreakable space in this part:
jq -r -c '.[1] | [...
So, edit the line manually, and replace all spaces with real spaces (taking care to not type unbreakable spaces again with AltGr+space)
Embedding jq in bash - what needs to be escaped?
Using bash and bash-like shells, jq programs can often be specified quite simply on the command line using single-quoted strings, e.g.
$ jq -n '"abc"'
"abc"
However, using this technique, single quotes are a headache since bash
does not allow single quotes within ordinary single-quoted strings. The workaround is quite horrible:
$ jq -n '"a'"'"'b"'
"a'b"
So if the jq program does have embedded single-quotes, then
it's probably time to use the -f option, but if that is not
an option, then using the form $'STRING' should be considered.
In this case, though, there are two characters that can occur in jq programs and
that will require attention: single-quotes and backslashes
For example:
$ jq -n $'"a\\tb" | "\'\\(.)\'"'
"'a\tb'"
If I'm not mistaken, the required escaping can be done using:
sed -e $'s/\'/\\\'/g' -e $'s/\\\\/\\\\\\\\/g'
Related
Im trying to create a variable using jq, let's say, for example:
firstName=($(curl -s https://www.easports.com/fifa/ultimate-team/api/fut/item | jq -r '.items[].firstName'))
The result I expected is "C. Ronaldo" but it gave me only "C." How can I fix it?
What about using .items[0] and command substitution instead of ($(...)) which is an array and is a subject to word-splitting, hence just the C.:
$ var=$(curl -s 'https://www.easports.com/fifa/ultimate-team/api/fut/item' | jq -r '.items[0].firstName')
$ echo "$var"
C. Ronaldo
I have used the below command, and It will get some substring in attribute value.
skipped=$(echo "$value" | jq -f '.[].output | scan("totalSkipped+: [[:digit:]]+")' | sed 's/"//g' )
I ran this script in shell through Jenkins job. and observed below error message:
/tmp/jenkins7615126817764256878.sh: command substitution: line 30: syntax error near unexpected token `"totalSkipped+: [[:digit:]]+"'
/tmp/jenkins7615126817764256878.sh: command substitution: line 30: `echo "$value" | jq .[].output | scan("totalSkipped+: [[:digit:]]+") | sed 's/"//g' )'
I have the entire json file which is stored in $value variable and echo "$value" returned the content of json file but not sure why its not working in jenkins.
I used the same command in jq online tool but It works as expected.
https://jqplay.org/s/7lBj_kDoB3
I'm using jq-1.6 version.
Can someone help me to resolve this?
skipped=$(jq -r '.[].output | scan("totalSkipped: [[:digit:]]+")' <<<"$value")
The pipeline is jq syntax, so it needs to be inside single quotes so the shell doesn't try to find a separate shell command named scan.
No reason for sed here -- using the -r argument to jq makes it emit raw strings as output, so they don't have syntactic quotes.
Cannot seem to find an answer to this one online...
I have a string variable (externally sourced) with new lines "\n" encoded as strings.
I want to replace those strings with actual new line carriage returns. The code below can achieve this...
echo $EXT_DESCR | sed 's/\\n/\n/g'
But when I try to store the result of this into it's own variable, it converts them back to strings
NEW_DESCR=`echo $EXT_DESCR | sed 's/\\n/\n/g'`
How can this be achieved, or what I'm I doing wrong?
Here's my code I've been testing to try get the right results
EXT_DESCR="This is a text\nWith a new line"
echo $EXT_DESCR | sed 's/\\n/\n/g'
NEW_DESCR=`echo $EXT_DESCR | sed 's/\\n/\n/g'`
echo ""
echo "$NEW_DESCR"
No need for sed, using parameter expansion:
$ foo='1\n2\n3'; echo "${foo//'\n'/$'\n'}"
1
2
3
With bash 4.4 or newer, you can use the E operator in ${parameter#operator}:
$ foo='1\n2\n3'; echo "${foo#E}"
1
2
3
Other answers contain alternative solutions. (I especially like the parameter expansion one.)
Here's what's wrong with your attempt:
In
echo $EXT_DESCR | sed 's/\\n/\n/g'
the sed command is in single quotes, so sed gets s/\\n/\n/g as is.
In
NEW_DESCR=`echo $EXT_DESCR | sed 's/\\n/\n/g'`
the whole command is in backticks, so a round of backslash processing is applied. That leads to sed getting the code s/\n/\n/g, which does nothing.
A possible fix for this code:
NEW_DESCR=`echo $EXT_DESCR | sed 's/\\\\n/\\n/g'`
By doubling up the backslashes, we end up with the right command in sed.
Or (easier):
NEW_DESCR=$(echo $EXT_DESCR | sed 's/\\n/\n/g')
Instead of backticks use $( ), which has less esoteric escaping rules.
Note: Don't use ALL_UPPERCASE for your shell variables. UPPERCASE is (informally) reserved for system variables such as HOME and special built-in variables such as IFS or RANDOM.
Depending on what exactly you need it for:
echo -e $EXT_DESCR
might be all you need.
From echo man page:
-e
enable interpretation of backslash escapes
This printf would do the job by interpreting all escaped constructs:
printf -v NEW_DESCR "%b" "$EXT_DESCR"
-v option will store output in a variable so no need to use command substitution here.
Problem with your approach is use of old back-ticks. You could do:
NEW_DESCR=$(echo "$EXT_DESCR" | sed 's/\\n/\n/g')
Assuming you're using gnu sed as BSD sed won't work with this approach.
I've a file that contains info that I'm retrieving such way
Command
cat 2018_02_15_09_01_08_result.tsv | grep -o [A-Z]\\*[0-9]*:[0-9]* | sort | uniq | sed -e 's/^/HLA-/' |tr '\n' ',' | sed '$ s/.$//'
Output
HLA-A*30:02,HLA-B*18:01,HLA-C*05:01
But I'm trying to save this in variable, the asterisk and a letter disappears, I've tried several ways, adding/removing commas etc and I'm yet not able to print it properly.
hla=`cat 2018_02_15_09_01_08_result.tsv | grep -o [A-Z]\\*[0-9]*:[0-9]* | sort | uniq | sed -e 's/^/HLA-/' |tr '\n' ',' | sed '$ s/.$//'`
echo $hla
HLA-05:01,HLA-18:01,HLA-30:02
echo "$hla"
HLA-05:01,HLA-18:01,HLA-30:02
There are multiple errors here, most of which will be aptly diagnosed by http://shellcheck.net/ without any human intervention.
You really should single-quote your regular expressions unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the regex before executing the command.
The obsolescent `command` in backticks introduces some unfortunate additional shell handling on the string inside the backticks. The solution since the 1990s is to prefer the $(command) syntax for command substitution, which does not exhibit this problem.
The cat is useless; grep knows full well how to read a file.
Try this refactored code:
hla=$(grep -o '[A-Z]*[0-9]*:[0-9]*' 2018_02_15_09_01_08_result.tsv |
sort -u | sed -e 's/^/HLA-/' |tr '\n' ',' | sed '$ s/.$//')
echo "$hla"
The double quotes around the variable interpolation in the echo are necessary and useful; notice also the line wraps for legibility and the use of sort -u in preference over sort | uniq (and generally try to reduce the number of processes -- once I understand what the sed | tr | sed does I can probably propose a simplification for that, too). Perhaps the simplest fix would be to refactor all of this into a single Awk script, but without access to the input, it's hard to tell you in more detail what that might look like.
(Also, are you really sure you need to capture the value to a variable? Often variable=value; echo "$variable" is just an obscure and inefficient way to say echo "value". And variable=$(command); echo "$variable" is better written simply command and capturing the command's standard output just so you can print it to standard output is a pure waste of cycles, unless you are planning to do something more with that variable's value.)
I've solved it by saving the output of the command with a redirection:
cat 2018_02_15_09_01_08_result.tsv |
grep -o [A-Z]\\*[0-9]*:[0-9]* |
sort | uniq |
sed -e 's/^/HLA-/' |tr '\n' ',' | sed '$ s/.$//' > out_file
hla=`cat out_file`
echo $hla
which gets me the expected HLA-A*30:02,HLA-B*18:01,HLA-C*05:01. Not the ideal solution, but it works.
I want to use the os.system command in order to get some information during a python script execution, but I get a syntax error, I'm literally using the same command on the terminal and the syntax it's correct.
the command is:
os.system(['df -h / | grep -E "\/$" | awk '{printf( $4)}''])
but the syntax error appears on the {
thanks in advance.
You're inconsistent with your single vs double quotes. Stack overflow's syntax coloring should make this clear.
You can fix your syntax error by escaping quotes as done by l'L'l, though I prefer wrapping the string in triple quotes. This avoids the need for escaped quotes (e.g. \').
os.system(["""df -h / | grep -E "\/$" | awk '{printf( $4)}'"""])
Then you'll just have a TypeError because the function doesn't expect a list. Fix it like this:
os.system("""df -h / | grep -E "\/$" | awk '{printf( $4)}'""")