I have a py script which gets as an argument data stream char > and >>, for example
python script.py -o 'ls -la /tmp > /tmp/test'
When I'm executing this kind of command the > char perceived with my terminal. Could you help me, how can I pass the > as a symbol so that my console doesn't think of it as a command?
Single quotes already work exactly as you would wish. Probably you are doing something wrong with the string inside of your Python script.
In some more detail, the arguments when Bash is done parsing this command are (one per line)
python
script.py
-o
ls -la /tmp > /tmp/test
(The quotes were useful while Bash was parsing this command line, but they are gone now.)
You should easily be able to verify writhin Python that sys.argv[0] is script.py, sys.argv[1] is -o, and sys.argv[2] is the single-quoted string, sans quotes; all of these are strings (the shell really doesn't have any other primitive data type).
Backslash escaping all shell metacharacters individually would work as well here;
python script.py -o ls\ -la\ /tmp\ \>\ test
or double quoting the string; albeit in the shell, double quotes have slightly different semantics (variables will be interpolated and backticks evaluated, and a backslash can be used to escape backslashes, double quotes, dollar signs, and backticks, unlike in single quotes, where every character is preserved verbatim, and there is no way to escape a character.)
You don't reveal how exactly your code doesn't work, but I'd speculate that your problem is actually the opposite; you end up running something like ls '>' and get an error message that this file does not exist.
For the record, > is a redirection operator, not a command. The shell parses this into
ls
-la
/tmp
with standard oqtput redirected to the file /tmp/test. In terms of Python code,
with open('/tmp/test', 'w') as redirect:
subprocess.run(['ls', '-la', '/tmp'], stdout=redirect)
Of course, if you want to support arbitrary shell features, you need a shell. The simplest fix here is probably
subprocess.run(sys.argv[2], shell=True)
See Actual meaning of shell=True in subprocess but here, I don't see any easy way to avoid the shell, short of by writing your own reimplementation.
In general, when you want to use special characters literally, i.e., without their special meaning, you need to surround a string of characters with single quotation marks(or quotes) to strip all characters within the quotes of any special meaning they might have. It seems that you’re already applied quoting whilst passing your command line argument to the python script. So it should work as expected. For example let’s assume that your python script looks like this:
import argparse
parser = argparse.ArgumentParser(description='Your App')
parser.add_argument('-o', action="store", dest="command")
options = parser.parse_args()
print(options.command)
Invoking this script with python ./script.py -o 'ls -la /tmp > /tmp/test' will produce the below results as expected.
ls -la /tmp > /tmp/test
But you've already mentioned that it is not the behavior you're seeing. So the next thing you might check is the code in your actual python script to determine what’s going on there with the input value. It is most likely that the problem is in your python script. So, the next step I would do is inspecting your python script to understand that magic.
Try by escaping the character using a backslash \ before the greather than > symbol.
Related
I have a simple bash script that does something like the following:
#!/bin/bash
a=$(curl -s http://metadata/endpoint/internal)
echo "$a - bar"
(this is just a simplification). Note the use of the two $ signs to execute a command and resolve a variable.
Using Terraform, I want to write this file to a GCP instance during startup. Per Terraform's instructions, I'm attempting to avoid using the File Provisioner. Using the metadata_startup_script field in google_compute_instance2, I am including the script so that I can write it to a particular location.
E.g.
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<EOI
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
Terraform is interpolating the $ in the subscript somewhere (either in the loading into the metadata_startup_script, or in the writing out into the script to disk.
So, depending on what I use to try to escape the interpolation, it still fails to write. For example, I have tried (in the subscript):
echo "\$a - bar"
echo "${“$”}a - bar"
echo "$$a - bar"
According to the terraform docs, I'm supposed to use $$, but when I do it in the above, I get:
echo "1397a - bar"
All which fail to replicate the original script.
I’m just looking for the exact bash script, as written, to be written to disk.
My goal would be to do the above without extra escape sequences (as detailed here - Escaping dollar sign in Terraform) so that i can continue to run the original script (for debugging purposes).
I would also prefer not to build a packer image with the original script in it.
Thanks!
I don't think it's Terraform eating your variable interpolations here, because Terraform only understands ${ (a dollar sign followed by a brace) as starting an interpolation, whereas your example contains only $a.
However, you do seem to be embedded one bash script inside another, so it seems plausible to me that the outer bash is resolving your $a before the inner bash gets a chance to look at it. If so, you can use the literal variant of bash heredoc syntax, as described in answers to How to cat <> a file containing code?, so that the outer bash will take the content as literal and leave it to the inner bash to evaluate.
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<'EOI'
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
Notice that I wrote <<'EOI' instead of <<EOI, following the guidance from that other question in combination with the main Bash documentation on "here documents" (bold emphasis mine):
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input (or file descriptor n if n is specified) for a command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
If your machine image is configured to run cloud-init at startup -- this is often but not always what is responsible for executing metadata_startup_script -- you may be able to achieve a similar effect without so much Bash scripting indirection by using Cloud Config YAML instead of a shell script directly.
For example, if your intent is only to write the content of that file into the designated location in the filesystem, you could potentially follow the Writing out arbitrary files example:
metadata_startup_script = <<-EOF
#cloud-config
${yamlencode({
write_files = [
{
encoding = "b64"
content = filebase64("${path.module}/scripts/simple_bash.sh")
path = "/etc/myservice/serv.conf"
owner = "root:root"
permissions = "0644"
},
]
}}
EOF
Cloud-init evaluates its modules at various points in the startup lifecycle. The Write Files module used here is specified to run once on the first boot of an instance, which matches how Cloud-init would typically treat a naked shell script too.
I do not think your issue is related to TF interpolation. I think you have problems because of normal bash interpolation, as its bash which is going to try to resolve $ in your /etc/myservice/serv.conf while writing its content.
The regular solution is to use 'EOI', not EOI:
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<'EOI'
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
I have the following bash script:
$ echo $(dotnet run --project Updater)
UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a"
$ export UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a"
$ echo $UPDATE_NEEDED
0
$ export $(dotnet run --project Updater)
$ echo $UPDATE_NEEDED
'0'
Why is it $UPDATE_NEEDED is 0 on the 3rd command, but '0' on the 5th command?
What would I need to do to get it to simply set 0? Using UPDATE_NEEDED=0 instead is not an option, as some of the other variables may contain a space (And I'd like to optimistically quote them to have it properly parse spaces).
Also, this is a bit of a XY problem. If anyone knows an easier way to export multiple variables from an executable that can be used later on in the bash script, that could also be useful.
To expand on the answer by Glenn:
When you write something like export UPDATE_NEEDED='0' in Bash code, this is 100% identical to export UPDATE_NEEDED=0. The quotes are used by Bash to parse the command expression, but they are then discarded immediately. Their only purpose is to prevent word splitting and to avoid having to escape special characters. In the same vein, the code fragment 'foo bar' is exactly identical to foo\ bar as far as Bash is concerned: both lead to space being treated as literal rather than as a word splitter.
Conversely, parameter expansion and command substitution follows different rules, and preserves literal quotes.
When you use eval, the command line arguments passed to eval are treated as if they were Bash code, and thus follow the same rules of expansion as regular Bash code, which leads to the same result as (1).
Apparently that Updater project is doing the equivalent of
echo "UPDATE_NEEDED=\'0\' MD5_SUM=\"7e3ad68397421276a205ac5810063e0a\""
It's explicitly outputting the quotes.
When you do export UPDATE_NEEDED='0' MD5_SUM="7e3ad68397421276a205ac5810063e0a",
bash will eventually remove the quotes before actually setting the variables.
I agree with #pynexj, eval is warranted here, although additional quoting is recommended:
eval export "$(dotnet ...)"
I am trying to give an argument to my python program through the terminal.
For this I am using the lines:
import sys
something = sys.argv[1]
I now try to put in a string like this through the bash terminal:
python my_script.py 2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
This returns a bash error because some of the characters in the string are bash special characters.
How can I use the string exactly as it is?
You can put the raw string into a file, for example like this, with cat and a here document.
cat <<'EOF' > file.txt
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
EOF
and then run
python my_script.py "$(< file.txt)"
You can also use the text editor of your choice for the first step if you prefer that.
If this is a reoccurring task, which you have to perform from time to time, you can make your life easier with a little alias in your shell:
alias escape='read -r string ; printf "Copy this:\n%q\n" "${string}"'
It is using printf "%q" to escape your input string.
Run it like this:
escape
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
Copy this:
2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
You can use the escaped string directly in your shell, without additional quotes, like this:
python my_script.py 2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
In order to make life easier, shells like bash do a little bit of extra work to help users pass the correct arguments to the programs they instruct it to execute. This extra work usually results in predictable argument arrays getting passed to programs.
Oftentimes, though, this extra help results in unexpected arguments getting passed to programs; and sometimes results in the execution of undesired additional commands. In this case, though, it ended up causing Bash to emit an error.
In order to turn off this extra work, Bash allows users to indicate where arguments should begin and end by surrounding them by quotation marks. Bash supports both single quotes (') and double quotes (") to delimit arguments. As a last resort, if a string may contain single and double quotes (or double quotes are required but aren't aggressive enough), Bash allows you to indicate that a special- or whitespace-character should be part of the adjacent argument by preceding it with a backslash (\\).
If this method of escaping arguments is too cumbersome, it may be worth simplifying your program's interface by having it consume this data from a file instead of a command line argument. Another option is to create a program that loads the arguments from a more controlled location (like a file) and directly execs the target program with the desired argument array.
I'm trying to create a Here Document which is a shell script that includes the cat command. Of course, it fails when encountering the 2nd cat. I'm performing a lot of substitutions as well, so can't use the "DOC" escape trick.
myfile="/tmp/myipaddr"
cat >/usr/bin/setIPaddress <<_DOC_
...
OUT=`cat $myfile`
...
_DOC_
I supposed I could echo into a file, but that seems kludgy and I have a lot of quotes and backticks I'd need to escape?!? Any other thoughts?
Suppose the file contains
hello world
As written, the script you generate will contain the line
OUT=hello world
because the command substitution is performed immediately.
At the very least, you need to quote the line in the here document as
OUT="`cat $myfile`"
I suspect what you want is to include the literal command substitution in the resulting shell script. To do that, you would want to quote the backticks to prevent them from being evaluated immediately. Better still, use the recommended form of command substitution, $(...), and quote the dollar sign.
cat >/usr/bin/setIPaddress <<_DOC_
...
OUT=\$(cat $myfile)
...
_DOC_
/usr/bin/setIPaddress will then include the line
OUT=$(cat /tmp/myipaddr)
Autoconf scripts have trouble with a filename or pathname with spaces. For example,
./configure CPPFLAGS="-I\"/path with space\""
results in (config.log):
configure:3012: gcc -I"/path with space" conftest.c >&5
gcc: with: No such file or directory
gcc: space": No such file or directory
The compile command from ./configure is ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5' and I am not able to modify this (I could perhaps, but working around autoconf in this way is not a general solution).
I think it comes down to getting a shell variable that contains spaces to be parsed as a single command line variable rather than split at spaces. The simplest shell example I can come up with is to create a file with spaces and attempt to list is with ls with a shell variable as the argument to ls:
$ touch "a b"
$ file="a b"
$ ls $file
ls: a: No such file or directory
ls: b: No such file or directory
This works, but is illegal since in autoconf I can't modify the shell code:
$ ls "$file"
a b
None of the following attempts at quoting things work:
$ file="\"a \"b"; ls $file
ls: "a: No such file or directory
ls: b": No such file or directory
$ file="a\ b"
$ file="a\\ b"
$ file="`echo \\"a b\\"`"
and so on.
Is this impossible to accomplish in shell scripts? Is there a magical quoting that will expand a shell variable with spaces into a single command line argument?
You should try to set the $IFS environment variable.
from man bash(1):
IFS - The Internal Field Separator that is used for word splitting
after expansion and to split lines into words with the read builtin
command. The default value is ''space tab newline''.
For example
IFS=<C-v C-m> # newline
file="a b"
touch $file
ls $file
Don't forget to set $IFS back or strange things will happen.
if you give command
gcc -I"x y z"
in a shell then certainly the single command line parameter "-Ix y z" will be passed to gcc. There is no question to that. That's the whole meaning of double quotes: things inside double quotes are NOT subject to field splitting, and so not subject to $IFS either, for instance.
But you need to be careful about the number of quotes you need. For instance, if you say
file="a b" # 1
and then you say
ls $file # 2
what happens is that the file variable's contents are 'a b', not '"a b"', because the double quotes were "eaten" when line 1 was parsed. The replaced value is then field-separated and you get ls on two files 'a' and 'b'. The correct way to get what you want is
file="a b"; ls "$file"
Now the problem in your original case is that when you set a variable to a string that CONTAINS double quotes, the double quotes are later not interpreted as shell quote symbols but just as normal letters. Which is why when you do something like
file="\"a b\""; ls $file
actually the shell tokenizes the contents of the file variable into '"a' and 'b"' when the ls command is analyzed; the double quote is no longer a shell quote character but just part of the variable's contents. It's analogous to that if you set
file="\$HOME"; ls $file
you get an error that '$HOME' directory does not exist---no environment variable lookup takes place.
So your best options are
Hack autoconf
Do not use path names with spaces (best solution)
Using space in directory names in the Unix world is simply asking for trouble. It's not just the problem of quoting in shell scripts (which needs to be done right anyway): some tools simply cannot cope with spaces in filenames. For instance, you can't (portably) write a Makefile rule that says build baz.o from foo bar/baz.c.
In the case of CPPFLAGS above, I would try one of the following (in order of preference):
Fix the system not use use any space in directory names.
Write a small wrapper around the compiler and call ./configure CC=mygcc. In that case mygcc might be:
#!/bin/sh
gcc "-I/foo bar/include" "$#"
Create a symbolic link (e.g., /tmp/mypath) to the dreaded path and use CPPFLAGS=-I/tmp/mypath.
You want to quote the entire argument, in either of these ways:
./configure "CPPFLAGS=-I/path with space"
./configure CPPFLAGS="-I/path with space"
The ./configure command then sees a single argument
"CPPFLAGS=-I/path with space"
which is parsed as a parameter named«CPPFLAGS» having the value«-I/path with space» (brackets added for clarity).
Using quotes is interesting. From (lightly) reading the bash man page I thought you had to escape the space with \, thus "/path with space" becomes /path\ with\ space I've never tried the quotes, but it seems that it doesn't work generally (your ls example). Escaping works with ls without quoting and without changing IFS.
What happens if you use the "escaping spaces" format of the command?
$ file="\"a b\""
$ eval ls $file
Everything depends on how the variable is used. First, note that if you are using Autoconf, this probably means that make will be used eventually, so that the rules are dictated by make, and in particular, the default make rules. Even though you may want to use your own rules exclusively, things must remain consistent between tools, and some variables have standard meanings, so that you do not want to deviate from them. This is not the case of CPPFLAGS, but this should remain similar to CFLAGS, which is standard. See the POSIX make utility, where variables are simply expanded with standard sh word splitting, which does not provide any quoting mechanism (the field separator is controlled by $IFS, but do not change the IFS variable to accept spaces as normal characters since this will break other things, like being able to provide several -I and/or -L options in such variables with the standard way).
Since there is such a limitation with make, I suppose that it would be useless to try to avoid this limitation in Autoconf.
Now, since a space is necessarily a field separator, the only possibility is to provide pathnames without space characters. If spaces in pathnames were to be supported in the future, this would probably be done via pathname encoding, with decoding at the high-level UI (a bit like with URL's). Alternatively, if you have the choice and really want to use spaces in pathnames, you may use some non-ASCII space (BTW, this is how RISC OS supports space in pathnames, by forcing it to be the no-break space).