I have a json file:
[{"outputType":{"type":"APK"},"apkInfo":{"type":"MAIN","splits":[],"versionCode":1},"path":"app-debug.apk","properties":{"packageId":"com.abc.def","split":"","minSdkVersion":"17"}}]
I want to use shell script to find the packageID (com.abc.def in this case) and copy it to a variable.
I don't really care about how it's done, as long as I get a variable holding the string, that will be perfect.
I don't do much shell script, but this task requires me to use shell script.
Thanks in advance.
v=$(sed -r 's/.*"packageId":"([^"]+)".*/\1/' file.json)
Look for the key "packageId" with colon and quotes, grab anything which is not a quote, followed by the quote. \1 is a reference to the matched expression in round parens.
Use a tool like jq to process JSON and don't rely on shell script ugly hacks.
In your case it would be: jq -r '.[0].properties.packageId'
It's easier, more readable and safer to use tools that parse JSON.
Related
We have an application that keeps some info in an encrypted file. To edit the file we have to put the text editor name in an environment variable in bash, for example, EDITOR=vi. Then we run the application and it opens the decrypted file in vi. I am trying to come up with a bash script that updates the encrypted file. The only solution that I can think of is passing sed command instead of vi to the EDITOR variable. It works perfectly for something like EDITOR='sed -i s#aaaa#bbbb#'.
Problem starts when I need space and regular expression. For example: EDITOR='sed -i -r "s#^(\s*masterkey: )(.*)#\1xxxxx#"' which return error. I tried running the EDITOR in bash with $EDITOR test.txt and I can see the problem. It doesn't like double quotes and space between them so I added a backslash before the double quotes and \s instead of space. Now it says unterminated address regex. For several hours I googled and couldn't find any solution. I tried replacing single quotes with double quotes and vice versa and everything that I could find on the internet but no luck.
How can I escape and which characters should I escape here?
Update:
Maybe if I explain the whole situation somebody could suggest an alternative solution. There is an application written by Ruby and it is inside a container. Ruby application has a secret_key_base for production and we supposed to change the key with EDITOR=vi rails credentials:edit --environment=production. I don't know Ruby and google did not return any ruby solution for automation so I could only think about sending sed instead of vi to Ruby.
How can I escape and which characters should I escape here?
That is not possible. Word splitting on the result of expansion cannot be escaped from inside the result of that expansion, it will always run. Note that filename expansion is also running over the result of the expansion.
Create an executable file with the script content and set EDITOR to it.
You could export a bash shell function, after some tries I got to:
myeditor() {
sed -i -E 's#^(\s*masterkey: )(.*)#\1xxxxx#' "$#"
}
export -f myeditor
EDITOR='bash -c "$#" _ myeditor'
I am trying to give an argument to my python program through the terminal.
For this I am using the lines:
import sys
something = sys.argv[1]
I now try to put in a string like this through the bash terminal:
python my_script.py 2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
This returns a bash error because some of the characters in the string are bash special characters.
How can I use the string exactly as it is?
You can put the raw string into a file, for example like this, with cat and a here document.
cat <<'EOF' > file.txt
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
EOF
and then run
python my_script.py "$(< file.txt)"
You can also use the text editor of your choice for the first step if you prefer that.
If this is a reoccurring task, which you have to perform from time to time, you can make your life easier with a little alias in your shell:
alias escape='read -r string ; printf "Copy this:\n%q\n" "${string}"'
It is using printf "%q" to escape your input string.
Run it like this:
escape
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
Copy this:
2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
You can use the escaped string directly in your shell, without additional quotes, like this:
python my_script.py 2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
In order to make life easier, shells like bash do a little bit of extra work to help users pass the correct arguments to the programs they instruct it to execute. This extra work usually results in predictable argument arrays getting passed to programs.
Oftentimes, though, this extra help results in unexpected arguments getting passed to programs; and sometimes results in the execution of undesired additional commands. In this case, though, it ended up causing Bash to emit an error.
In order to turn off this extra work, Bash allows users to indicate where arguments should begin and end by surrounding them by quotation marks. Bash supports both single quotes (') and double quotes (") to delimit arguments. As a last resort, if a string may contain single and double quotes (or double quotes are required but aren't aggressive enough), Bash allows you to indicate that a special- or whitespace-character should be part of the adjacent argument by preceding it with a backslash (\\).
If this method of escaping arguments is too cumbersome, it may be worth simplifying your program's interface by having it consume this data from a file instead of a command line argument. Another option is to create a program that loads the arguments from a more controlled location (like a file) and directly execs the target program with the desired argument array.
I am trying to study both bash and ruby and I am trying to pass a variable that contains a JSON object from ruby to be printed using a bash script.
I tried storing the JSON object in a variable then used that variable as an argument for my bash script but I am not getting the result that I wanted.
so in my ruby code I have this:
param = request.body
cmd = "ruby_to_bash #{param}"
exec cmd
and in my bash script, I am simply outputting the value of the argument:
echo $1
For example I have this JSON:
{"first":"one","second":"two","third":"three"}
my code only gives me this output:
{first:
I want to display the whole JSON object. How can I do this? Any kind of help will be much appreciated.
Both bash and ruby treat double quotes in a kinda special way: double-quoted strings are interpolated before passed to the receiver. I strongly encourage you to start with learning somewhat one in the first place instead of doing zero progress in both due to induced errors.
Answering the question stated, one must escape double quotes when passing back and forth since both ruby and bash treat them as content delimiters and discard when the content is already parsed in the resulting string.
This would work:
exec %|ruby_to_bash #{param.gsub('"', '\"')}|
I use %| and | as string delimiters to avoid clashes against inner quotes.
Sidenote: the above won’t work for the input containing spaces, but I purposely avoid to show how to deal with spaces since it leads to the dead end; once handled spaces with surrounding the interpolated param with single quotes, we are prone to be screwed up with inner single quotes in JSON object and so forth.
The code above is not someone should ever produce in production.
I think your best bet is, as usual in such cases, not to involve a shell at all.
When you say this:
cmd = "ruby_to_bash #{param}"
exec cmd
Your exec call is invoking a shell which parses cmd and then runs ruby_to_bash so you're doing this:
Build a command line.
Replace the current process with /bin/sh.
/bin/sh parses the command line that you should mashed together. This is where you need to worry about quoting and escaping to get past the shell.
/bin/sh exec's ruby_to_bash.
You could bypass the whole problem by using the multi-argument form of Kernel#exec:
exec(cmdname, arg1, ...)
command name and one or more arguments (no shell)
which doesn't bother with a shell at all. If you say:
exec 'ruby_to_bash', param
You won't involve a shell and the quotes or spacing in param won't matter to anyone other than ruby_to_bash.
I am new to scripting in Linux and I think I'm getting confused with using variables inside command substitution the more I learn and read about it. Can someone explain to me the following scenario?
In my ksh script, I am trying to use a ksh variable inside an sqlplus script as follows:
temp_var="'a', 'b'"
randomVar=$(sqlplus -s $con_details <<EOF
update table ABC
Set field1='val'
Where field2 NOT IN ("${temp_var}");
EOF)
But the above syntax leads to an error in the query and it fails with code 1.
However when I unquote the variable and simply write
Where field2 NOT IN (${temp_var});
The query runs fine. I have seen a lot of examples on SO and Unix and Linux advising to always quote your variables used inside command substitution, but it seems the opposite works for me.
I don't seem to get why using quotes inside $() give an error as opposed to not using them.
Also, the query runs fine when I don't use the ksh variable in it (i.e. Without the WHERE clause).
This is a different situation than where the usual advice applies -- you're using the variable in a here-document, rather than as part of the command line. The difference is in how it gets parsed.
When you use a variable on a command line (something like ls $file), the variable gets replaced by its value partway through the process of parsing the command, with weird and generally undesirable results. The standard solution is to double-quote the variable (ls "$file") to prevent it from being parsed at all, just used directly. The standard mistake people make is putting quotes in the variable's value, which doesn't work because the variable gets replaced after quotes have already been parsed.
But you're using the variable in a here-document, and those work a lot differently. What happens is that the shell just does variable expansion (and some escape parsing) in the here-document, but doesn't do any more extensive parsing. In particular, it doesn't parse quotes in the here-document, just treats them like any other characters. The document then gets passed as input to the command (sqlplus in your case), and it parses the document according to whatever its syntax rules are. Since the parsing happens after variable replacement, it doesn't matter if the quotes are in the variable or around it; they work the same either way. But you can't do both, which is what was happening with double-quotes around the variable. Essentially, you were sending this document to sqlplus:
update table ABC
Set field1='val'
Where field2 NOT IN ("'a', 'b'");
... and sqlplus doesn't like that double-quotes around single-quotes thing, and complains.
I'm trying to create a Here Document which is a shell script that includes the cat command. Of course, it fails when encountering the 2nd cat. I'm performing a lot of substitutions as well, so can't use the "DOC" escape trick.
myfile="/tmp/myipaddr"
cat >/usr/bin/setIPaddress <<_DOC_
...
OUT=`cat $myfile`
...
_DOC_
I supposed I could echo into a file, but that seems kludgy and I have a lot of quotes and backticks I'd need to escape?!? Any other thoughts?
Suppose the file contains
hello world
As written, the script you generate will contain the line
OUT=hello world
because the command substitution is performed immediately.
At the very least, you need to quote the line in the here document as
OUT="`cat $myfile`"
I suspect what you want is to include the literal command substitution in the resulting shell script. To do that, you would want to quote the backticks to prevent them from being evaluated immediately. Better still, use the recommended form of command substitution, $(...), and quote the dollar sign.
cat >/usr/bin/setIPaddress <<_DOC_
...
OUT=\$(cat $myfile)
...
_DOC_
/usr/bin/setIPaddress will then include the line
OUT=$(cat /tmp/myipaddr)