Is it possible to echo a text and keep all quotes and double quotes in place?
I want to write a function to copy the text currently written in the terminal (completely with quotes).
Since I am on OSX I have to use pbcopy:
pb(){echo "$#" | pbcopy}
But pb osascript -e 'tell Application "iTerm" to display dialog "Job finished"' does return
osascript -e tell Application "iTerm" to display dialog "Job finished" but not
osascript -e tell 'Application "iTerm" to display dialog "Job finished"'.
The shell is removing the outer single quotes before pb ever sees the argument(s). Pass a single argument
pb "osascript -e 'tell Application \"iTerm\" to display dialog \"Job finished\"'"
to pb, and define it as
pb () {
printf '%s\n' "$1" | pbcopy
}
It would probably be just as easy to use a here document, though, rather than define a function that feeds its argument to pbcopy:
$ pbcopy <<'EOF'
osascript -e 'tell Application "iTerm" to display dialog "Job finished"'
EOF
Slightly more typing, but no need to nest so many quotes.
Is it possible to echo a text and keep all quotes and double quotes in place? I want to write a function to copy the text currently written in the terminal (completely with quotes).
Let's explore what you mean by "currently written in the terminal". If I understand correctly, you want to provide arbitrary input to a shell command at invocation time. In other words, you have a bit of text that you want to add into your copy buffer, and you want to send it to the stdin of pbcopy to do so.
As a solution to this particular problem, a shell function is a poor fit. That's because a shell function needs to be invoked with arguments that are subject to shell interpretation, and so you'll have to escape them carefully both when invoking pb and when defining it. These strings can be escaped. But it's inconvenient, for one thing because there are several special characters that need to be escaped in a double quoted string, but ' can't itself be escaped in a single quoted string.
Let's explore some other options.
$ pbcopy <<< "this is a simple one-line string directly from the command line. Since it's an argument to pbcopy it needs to be escaped."
$ pbpaste
this is a simple one-line string directly from the command line. Since it's an argument to pbcopy it needs to be escaped.
Here we tell the shell to provide the text to pbpaste's standard input. We still have to escape the string. But we don't have to pass it anywhere or correctly enquote it again to make it a valid shell argument.
Or we can provide multi-line string data to pbcopy without having to enquote it with this special here-doc syntax:
$ pbcopy <<-'-my-chosen-delimiter'
> Since this string's delimiter is single quoted,
> no interpolation will occur. That means " double quotes
> have no meaning, nor does ' single quotes, $dollar signs
> or other such meaningful bash syntaxes.
> -my-chosen-delimiter
$ pbpaste
Since this string's delimiter is single quoted,
no interpolation will occur. That means " double quotes
have no meaning, nor does ' single quotes, $dollar signs
or other such meaningful bash syntaxes.
I have thought that bash would be more powerful than that.
Well, on one hand, I think this is a very good opportunity to compare and contrast command line arguments (which are inherently positional by design and thus must be parsed and split by, usually, whitespace between arguments) to input and output streams expressed with | pipelines. I/O streams are designed to hold arbitrary data; it's not bash's fault that you wanted to make one into a shell-parsed variable list. It's not the considerable power of bash you're observing here, it's the functional limit of your bash knowledge.
But on the other hand, you're kind of right. The concessions to the user-interactive command line interface, substantial historical constraints to achieve backwards compatibility, and many valid design considerations made bash what it is. I for one find it and its ilk to be by far the most powerful user interface to a computer. But I wouldn't use it to assemble a complex application because, frankly, it's syntactically difficult. So don't expect bash to be something it's not. If you don't want to understand it's quirky and esoteric irregularities, stick with something more recent and more pedantic like python , go , ruby, node, or whatever non unix centric people run these days:P
Related
I am trying to give an argument to my python program through the terminal.
For this I am using the lines:
import sys
something = sys.argv[1]
I now try to put in a string like this through the bash terminal:
python my_script.py 2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
This returns a bash error because some of the characters in the string are bash special characters.
How can I use the string exactly as it is?
You can put the raw string into a file, for example like this, with cat and a here document.
cat <<'EOF' > file.txt
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
EOF
and then run
python my_script.py "$(< file.txt)"
You can also use the text editor of your choice for the first step if you prefer that.
If this is a reoccurring task, which you have to perform from time to time, you can make your life easier with a little alias in your shell:
alias escape='read -r string ; printf "Copy this:\n%q\n" "${string}"'
It is using printf "%q" to escape your input string.
Run it like this:
escape
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
Copy this:
2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
You can use the escaped string directly in your shell, without additional quotes, like this:
python my_script.py 2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
In order to make life easier, shells like bash do a little bit of extra work to help users pass the correct arguments to the programs they instruct it to execute. This extra work usually results in predictable argument arrays getting passed to programs.
Oftentimes, though, this extra help results in unexpected arguments getting passed to programs; and sometimes results in the execution of undesired additional commands. In this case, though, it ended up causing Bash to emit an error.
In order to turn off this extra work, Bash allows users to indicate where arguments should begin and end by surrounding them by quotation marks. Bash supports both single quotes (') and double quotes (") to delimit arguments. As a last resort, if a string may contain single and double quotes (or double quotes are required but aren't aggressive enough), Bash allows you to indicate that a special- or whitespace-character should be part of the adjacent argument by preceding it with a backslash (\\).
If this method of escaping arguments is too cumbersome, it may be worth simplifying your program's interface by having it consume this data from a file instead of a command line argument. Another option is to create a program that loads the arguments from a more controlled location (like a file) and directly execs the target program with the desired argument array.
I execute some code in shell using
subprocess.Popen('echo '+user_string+' | pipe to some string manipulation tools',
shell=True)
where user_string is from an untrusted source.
Is it safe enough to use shlex.quote() for escaping the input?
I'm necromancing this because it's the top Google hit for "is shlex.quote() safe", and while the accepted answer seems correct, there's a lot of pitfalls to point out.
shlex.quote() escapes the shell's parsing, but it does not escape the argument parser of the command you're calling, and some additional tool-specific escaping needs to be done manually, especially if your string starts with a dash (-).
Most (but not all) tools accept -- as an argument, and anything afterward is interpreted as verbatim. You can prepend "-- " if the string starts with "-". Example: rm -- --help removes the file called --help.
When dealing with file names, you can prepend "./" if the string starts with "-": rm ./--help.
In the case of your example with echo, neither escape is sufficient: When attempting to echo the string -e, echo -- -e gives the wrong result, you'll need something like echo -e \x2de. This demonstrates that there's no universal bulletproof way to escape program arguments.
The safest route is to bypass the shell by avoiding shell=True or os.system() if the string involves any user-supplied data.
In your case, set stdin=subprocess.PIPE and pass the user_string as an argument to communicate(). Then you can even leave your original invocation as-is with shell=True:
subprocess.Popen(
'pipe to some string manipulation tools',
shell=True,
stdin=subprocess.PIPE
).communicate(user_input_string.encode())
According to the official pyton documentation for shlex.quote the answer is yes.
Of course that also depends on what you mean by "safe enough". Under the assumption that you mean "Will using shlex.quote on user_string guard me against a typical scenario of malicious shell code passed as string input to my script?" the answer is yes.
This seems like it should be simple, but I'm pulling out my remaining hair trying to get it to work. In a shell script I want to run some Applescript code that defines a string, then pass that string (containing a single quote) to a shell command that calls PHP's addslashes function, to return a string with that single quote escaped properly.
Here's the code I have so far - it's returning a syntax error.
STRING=$(osascript -- - <<'EOF'
set s to "It's me"
return "['test'=>'" & (do shell script "php -r 'echo addslashes(\"" & s & "\");") & "']"
EOF)
echo -e $STRING
It's supposed to return this:
['test'=>'It\'s me']
First, when asking a question like this, please include what's happening, not just what you're trying to do. When I try this, I get:
42:99: execution error: sh: -c: line 0: unexpected EOF while looking for matchin
sh: -c: line 1: syntax error: unexpected end of file (2)
(which is actually two error messages, with one partly overwriting the other.) Is that what you're getting?
If it is, the problem is that the inner shell command you're creating has quoting issues. Take a look at the AppleScript snippet that tries to run a shell command:
do shell script "php -r 'echo addslashes(\"" & s & "\");"
Since s is set to It's me, this runs the shell command:
php -r 'echo addslashes("It's me");
Which has the problem that the apostrophe in It's me is acting as a close-quote for the string that starts 'echo .... After that, the double-quote in me"); is seen as opening a new quoted string, which doesn't get closed before the end of the "file", causing the unexpected EOF problem.
The underlying problem is that you're trying to pass a string from AppleScript to shell to php... but each of those has its own rules for parsing strings (with different ideas about how quoting and escaping work). Worse, it looks like you're doing this so you can get an escaped string (following which set of escaping rules?) to pass to something else... This way lies madness.
I'm not sure what the real goal is here, but there has to be a better way; something that doesn't involve a game of telephone with players that all speak different languages. If not, you're pretty much doomed.
BTW, there are a few other dubious shell-scripting practices in the script:
Don't use all-caps variable named in shell scripts. There are a bunch of all-caps variables that have special meanings, and if you accidentally use one of those for something else, weird results can happen.
Put double-quotes around all variable references in scripts, to avoid them getting split into multiple "words" and/or expanded as shell wildcards. For example, if the variable string was set to "['test'=>'It\'s-me']", and you happened to have files named "t" and "m" in the current directory, echo -e $string will print "m t" because those are the files that match the [] pattern.
Don't use echo with options and/or to print strings that might contain escapes, since different versions treat these things differently. Some versions, for example, will print the "-e" as part of the output string. Use printf instead. The first argument to printf is a format string that tells it how to format all of the rest of the arguments. To emulate echo -e "$string" in a more reliable form, use printf '%b\n' "$string".
To complement Gordon Davisson's helpful answer with a pragmatic solution:
Shell strings cannot contain \0 (NUL) characters, but the following sed command emulates all other escaping that PHP's (oddly named) addslashes PHP function performs (\-escaping ', " and \ instances):
string=$(osascript <<'EOF'
set s to "It's me\\you and we got 3\" of rain."
return "['test'=>'" & (do shell script "sed 's/[\"\\\\'\\'']/\\\\&/g' <<<" & quoted form of s) & "']"
EOF
)
printf '%s\n' "$string"
yields
['test'=>'It\'s me\\you and we got 3\" of rain.']
Note the use of quoted form of, which is crucial for passing a string from AppleScript to a do shell script shell command with proper quoting.
Also note how the closing here-doc delimiter, EOF, is on its own line to ensure that it is properly recognized (in Bash 3.2.57, as used on macOS 10.12, (also when called as /bin/sh, which is what do shell script does), this isn't strictly necessary, but Bash 4.x would rightfully complain about EOF) with warning: here-document at line <n> delimited by end-of-file (wanted 'EOF')
I am having issues with my known methods of generating user input prompts:
read -p "Input something: " variabile
This causes issues if attempting to use the arrow keys, it echoes the ANSI code for each arrow key stroke
read -e -p "Input something: " variable
This fixes the arrow keys issue but when reaching the width of the terminal, text input doesn't continue on a newline but on the same line, overwriting (visually) the existing input
echo -n "Input something: "; read -e variable
This apparently fixes both formerly described issues... until I found that typing something then hitting backspace overwrites the prompt and also when the input is longer, from the second newline of input the visual overwriting manifests again.
So is there a good way of producing prompts without the above issues?
UPDATE
After re-checking, I now know what's causing the input overwrite for read -e -p
I am using these variables for highlighting text for the read prompt:
highlight=$(echo -e "\e[1;97m")
clear=$(echo -e "\e[0m")
read -e -p "Input$highlight something$clear: " variable
This is the only way I could make the highlighting work inside read prompt (assigning escape sequences to the variables doesn't work, I need to echo them like I did) but they also seem to cause the input overwrite issue.
As dimo414 mentions, readline thinks the prompt is longer than it is. It counts every character in the terminal escape sequence in computing the length. You can see how long it thinks the escape sequence is as follows
echo ${#highlight}
In the bash PS1 prompt, surrounding such an escape sequence with "\[" and "\]" instructs readline to ignore everything between when calculating current line length, but these are not the right escapes for the bash read built-in.
The escapes for read are $'\001' and $'\002', as mentioned in BashFAQ, but in my experience, you need the -e option on read, as well. The brute force way to do what you want would be:
read -e -p "Input "$'\001'"${highlight}"$'\002'something$'\001'"${clear}"$'\002'": "
You should use tput rather than hard-coded escape sequences, for the sake of terminal independence. Read man 5 termcap.
See my dotfiles for elegant bash functions to do the begin/end quoting above for you.
The shell keeps track of how long it thinks the prompt is, in order to know where the user's input starts and stops. Unfortunately when you print color escape codes in a prompt you throw of Bash's counting, since it expects the escape characters to take up space in the terminal.
To avoid that, you just need to wrap all color sequences in \[ and \], which tells your shell the enclosed characters are non-printing, and should not be counted.
For example, your highlight variable should be:
highlight=$(echo -e "\[\e[1;97m\]")
Personally, I use the color and pcolor functions from my Prompt.gem project, which handles the proper escaping and would make your command much easier to read:
read -e -p "Input $(pcolor DEFAULT BOLD)something$(pcolor): " variable
I'm looking for a way (other than ".", '.', \.) to use bash (or any other linux shell) while preventing it from parsing parts of command line. The problem seems to be unsolvable
How to interpret special characters in command line argument in C?
In theory, a simple switch would suffice (e.g. -x ... telling that the
string ... won't be interpreted) but it apparently doesn't exist. I wonder whether there is a workaround, hack or idea for solving this problem. The original problem is a script|alias for a program taking youtube URLs (which may contain special characters (&, etc.)) as arguments. This problem is even more difficult: expanding "$1" while preventing shell from interpreting the expanded string -- essentially, expanding "$1" without interpreting its result
Use a here-document:
myprogramm <<'EOF'
https://www.youtube.com/watch?v=oT3mCybbhf0
EOF
If you wrap the starting EOF in single quotes, bash won't interpret any special chars in the here-doc.
Short answer: you can't do it, because the shell parses the command line (and interprets things like "&") before it even gets to the point of deciding your script/alias/whatever is what will be run, let alone the point where your script has any control at all. By the time your script has any influence in the process, it's far too late.
Within a script, though, it's easy to avoid most problems: wrap all variable references in double-quotes. For example, rather than curl -o $outputfile $url you should use curl -o "$outputfile" "$url". This will prevent the shell from applying any parsing to the contents of the variable(s) before they're passed to the command (/other script/whatever).
But when you run the script, you'll always have to quote or escape anything passed on the command line.
Your spec still isn't very clear. As far as I know the problem is you want to completely reinvent how the shell handles arguments. So… you'll have to write your own shell. The basics aren't even that difficult. Here's pseudo-code:
while true:
print prompt
read input
command = (first input)
args = (argparse (rest input))
child_pid = fork()
if child_pid == 0: // We are inside child process
exec(command, args) // See variety of `exec` family functions in posix
else: // We are inside parent process and child_pid is actual child pid
wait(child_pid) // See variety of `wait` family functions in posix
Your question basically boils down to how that "argparse" function is implemented. If it's just an identity function, then you get no expansion at all. Is that what you want?