Hi I need to go over characters in string in bash including spaces. How can I do it?
Bash does support substrings directly (If that's what the OP wants):
$ A='Hello World!'
$ echo "${A:3:5}"
lo Wo
$ echo "${A:5:3}"
Wo
$ echo "${A:7:3}"
orl
The expansion used is generalized as:
${PARAMETER:OFFSET:LENGTH}
PARAMETER is your variable name. OFFSET and LENGTH are numeric expressions as used by `let'. See the bash info page on shell parameter expansion for more information, since there are a few important details on this.
Therefore, if you want to e.g. print all the characters in the contents of a variable each on its own line you could do something like this:
$ for ((i=0; i<${#A}; i++)); do echo ${A:i:1}; done
The advantage of this method is that you don't have to store the string elsewhere, mangle its contents or use external utilities with process substitution.
Not sure what you really mean, but in almost all cases, problems with strings including spaces can be solved by quoting them.
So, if you've got a nice day, try "a nice day" or 'a nice day'.
You use some external tool for it. The bash shell is really meant to be used to glue other programs together in usually simple combinations.
Depending on what you need, you might use cut, awk, sed or even perl.
Try this
#/bin/bash
str="so long and thanks for all the fish"
while [ -n "$str" ]
do
printf "%c\n" "$str"
str=${str#?}
done
Related
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)
I am trying to parse a series of output lines that contain a mix of values and strings.
I thought that the set command would be a straightforward way to do it.
An initial test seemed promising. Here's a sample command line and its output:
$ (set "one two" three; echo $1; echo $2; echo $3)
one two
three
Obviously I get two variables echoed and nothing for the third.
However, when I put it inside my script, where I'm using read to capture the output lines, I get a different kind of parsing:
echo \"one two\" three |
while read Line
do
echo $Line
set $Line
echo $1
echo $2
echo $3
done
Here's the output:
"one two" three
"one
two"
three
The echo $Line command shows that the quotes are there but the set command does not use them to delimit a parameter. Why not?
In researching the use of read and while read I came across the while IFS= read idiom, so I tried that, but it made no difference at all.
I've read through dozens of questions about quoting, but haven't found anything that clarifies this for me. Obviously I've got my levels of quoting confused, but where? And what might I do to get what I want, which is to get the same kind of parsing in my script as I got from the command line?
Thanks.
read does not interpret the quotes, it just reads "one as one token, and two" as another. (Think of all the ways in which things could go wrong if the shell would evaluate input from random places. The lessons from Python 2 and its flawed input() are also an excellent illustration.)
If you really want to evaluate things, eval does that; but it comes with a boatload of caveats, and too often leads to security problems if done carelessly.
Depending on what you want to accomplish, maybe provide the inputs on separate lines? Or if these are user-supplied arguments, just keep them in "$#". Notice also how you can pass a subset of them into a function, which gets its own local "$#" if you want to mess with it.
(Tangentially, you are confusing yourself by not quoting the argument to echo. See When to wrap quotes around a shell variable.)
Why not?
read splits the input on each character that's in IFS. With unset or default IFS, that's space or tab or newline. Any other characters are not special in any way and quotes are not anyhow special.
Obviously I've got my levels of quoting confused, but where?
You wrongly assumed read is smart enough to interpret quotes. It isn't. Moreover, read ignores \ sequences. Read how to read a stream line by line and bash manual word splitting.
what might I do to get what I want, which is to get the same kind of parsing in my script as I got from the command line?
To get the same parsing as you got from the command line you may use eval. eval is evil. Eval command and security issues.
echo \"one two\" three |
while IFS= read -r line; do
eval "set $line" # SUPER VERY UNSAFE DO NOT USE
printf "%s\n" "$#"
done
When using eval a malicious user may echo '"$(rm -rf *)"' | ... remove your files in an instant. The simplest solution in the shell is to use xargs, which (mostly confusingly) includes quotes parsing when parsing input.
echo \"one two\" three | xargs -n1 echo
Being new to shell scripting, I am not clear about the Quoting and splitting concepts in bash. In the below snippet:
array1=("france-country":"Italy-country":"singapore-country")
echo ${#array1[#]}
IFS=":-"
for i in ${array1[#]}
do
echo "$i"
done
unset IFS
with IFS being :-, I thought the result would be:
france-country
Italy-country
belgium-country
as I had quoted them("france-country"). I think it should not get split on "-". But the results were:
france
country
Italy
country
belgium
country
It would be great if someone can point me out my mistake in understanding.
For your problem you can simply change the field separator as : i.e. IFS=: because each country name is separated by : not :- in your example
array1=("france-country":"Italy-country":"singapore-country")
echo ${#array1[#]}
IFS=":"
for i in ${array1[#]}
do
echo "$i"
done
unset IFS
FYI, array elements in bash are separated by space so the whole string "france-country":"Italy-country":"singapore-country" is a single element of the array therefore echo ${#array1[#]} will always be 1 . So I do not see any use of an array in this example. Simple variable would have been suffice.
This script shows how to split a colon-separated string into a Bash array.
#!/usr/bin/env bash
words="france-country:Italy-country:singapore-country:Australia-country"
IFS=':' array1=($words)
numwords="${#array1[#]}"
for((i=0; i<numwords; i++));do
echo "$i: ${array1[i]}"
done
output
0: france-country
1: Italy-country
2: singapore-country
3: Australia-country
Note that in
array1=($words)
we don't put quotes around $words, as that would prevent word splitting.
We don't actually need the quotes in
words="france-country:Italy-country:singapore-country:Australia-country"
but quotes (either single or double) would be needed if there were any spaces in that string. Similarly, the quotes are superfluous in numwords="${#array1[#]}", and many Bash programmers would omit them because we know the result of ${#array1[#]} will never have spaces in it.
It's a Good Idea to always use quotes in Bash unless you're sure you don't want them, eg in the case of array1=($words) above, or when performing tests with the [[ ... ]] syntax.
I suggest you bookmark BashGuide and browse through their FAQ. But the only way to really learn the arcane ways of quoting and splitting in Bash is to to write lots of scripts. :)
You may also find ShellCheck helpful.
I would like my bash script to check the name of the directory where it is run. Something like:
#!/bin/bash
path=eval 'pwd'
dirname=eval 'basename $path'
But it doesn't work: I get
./foo.sh: line 5: basename $path: command not found
How can I fix it? Also, once I get dirname to contain the correct dirname, I'd like to convert it to lowercase, to test it. I'm able to do this on the command line with awk:
echo $dirname | awk '{print tolower($0)}'
but how do I capture the return value into a variable?
Why not use:
#!/bin/bash
path=`pwd`
dirname=`basename $path | awk '{print tolower($0)}'`
Or if you want to do it as a one liner:
dirname=`pwd | xargs basename | awk '{print tolower($0)}'`
You can rewrite it to
dirname=eval "basename $path"
With single-quotes, you don't get shell expansion, but you want $path getting expanded.
BTW: I'd suggesst using
path=$(basename $path)
It's way more generic and better readable if you do something like
path=$(basename $(pwd))
or to get the lowercase result
path=$(basename $(pwd) | awk '{print tolower($0)}')
or
path=$(basename $(pwd) | tr 'A-Z' 'a-z' )
The form
x=y cmd
means to temporarily set environment variable x to value y and then run cmd, which is how these lines are interpreted:
path=eval 'pwd'
dirname=eval 'basename $path'
That is, they aren't doing what you seem to expect at all, instead setting an environment variable to the literal value eval and then running (or failing to find) a command. As others have said, the way to interpolate the results of a command into a string is to put it inside $(...) (preferred) or `...` (legacy). And, as a general rule, it's safer to wrap those in double quotes (as it is safer to wrap any interpolated reference in quotes).
path="$(pwd)"
dirname="$(basename "$path")"
(Technically, in this case the outer quotes aren't strictly necessary. However, I'd say it's still a good habit to have.)
B=$(echo "Some text that has CAPITAL letters " | awk '{print tolower($0)}')
eval executes command passed to it, but it returns only command exit status code, so you cannot really use it in set operator. The way to go to embed command into set operator either to use right single quotes or $()
So the script will look like this:
#!/bin/bash
curr_path=$(pwd)
echo $curr_path
curr_dir=$(basename $curr_path)
echo $curr_dir
echo $curr_dir | awk '{print tolower($0)}'
Your code doesn't work because you use single quotes rather than double quotes. Single quotes prevent variable expansion, thus $path is not expanded into the path you want to use and is taken as it is, as it if were a string.
Your awk invocation would not work for the same reason as well.
Although you could solve the problem replacing single quotes with double quotes, like this:
#!/bin/bash
path=eval "pwd"
dirname=eval "basename $path"
I would suggest using grave accents instead (). There's no reason to useeval` in this case. Plus, you can also use it to collect the return value you are interested in:
#!/bin/bash
path=`pwd`
dirname=`basename $path`
variable=`echo $dirname | awk "{print tolower($0)}"`
Here's an excerpt from my answer to What platform independent way to find directory of shell executable in shell script? which, in itself, fully answers your question aside from the lowercase part, which, in my opinion, has been duly addressed many times in other answers here.
What's unique about my answer is that when I was attempting to write it for the other question I encountered your exact problem - how do I store the function's results in a variable? Well, as you can see, with some help, I hit upon a pretty simple and very powerful solution:
I can pass the function a sort of messenger variable and dereference any explicit use of the resulting function's argument's $1 name with eval as necessary, and, upon the function routine's completion, I use eval and a backslashed quoting trick to assign my messenger variable the value I desire without ever having to know its name.
In full disclosure, though this was the solution to my problem, it was not by any means my solution. I've had several occasions to visit there before, but some of his descriptions, though probably brilliant, are a little out of my league, and so I thought others might benefit if include my own version of how this works in the previous paragraph. Though of course it was very simple to understand once I did, for this one especially, I had to think long and hard to figure out how it might work. Anyway, you can find that and more at Rich's sh tricks and I have also excerpted the relevant portion of his page below my own answer's excerpt.
...
EXCERPT:
...
Though not strictly POSIX yet, realpath is a GNU core app since 2012. Full disclosure: never heard of it before I noticed it in the info coreutils TOC and immediately thought of [the linked] question, but using the following function as demonstrated should reliably, (soon POSIXLY?), and, I hope, efficiently
provide its caller with an absolutely sourced $0:
% _abs_0() {
> o1="${1%%/*}"; ${o1:="${1}"}; ${o1:=`realpath "${1}"`}; eval "$1=\${o1}";
> }
% _abs_0 ${abs0:="${0}"} ; printf %s\\n "${abs0}"
/no/more/dots/in/your/path2.sh
EDIT: It may be worth highlighting that this solution uses POSIX parameter expansion to first check if the path actually needs expanding and resolving at all before attempting to do so. This should return an absolutely sourced $0via a messenger variable (with the notable exception that it will preserve symlinks) as efficiently as I could imagine it could be done whether or not the path is already absolute.
...
(minor edit: before finding realpath in the docs, I had at least pared down my version of [the version below] not to depend on the time field [as it does in the first ps command], but, fair warning, after testing some I'm less convinced ps is fully reliable in its command path expansion capacity)
On the other hand, you could do this:
ps ww -fp $$ | grep -Eo '/[^:]*'"${0#*/}"
eval "abs0=${`ps ww -fp $$ | grep -Eo ' /'`#?}"
...
And from Rich's sh tricks:
...
Returning strings from a shell function
As can be seen from the above pitfall of command substitution, stdout is not a good avenue for shell functions to return strings to their caller, unless the output is in a format where trailing newlines are insignificant. Certainly such practice is not acceptable for functions meant to deal with arbitrary strings. So, what can be done?
Try this:
func () {
body here
eval "$1=\${foo}"
}
Of course ${foo} could be replaced by any sort of substitution. The key trick here is the eval line and the use of escaping. The “$1” is expanded when the argument to eval is constructed by the main command parser. But the “${foo}” is not expanded at this stage, because the “$” has been quoted. Instead, it’s expanded when eval evaluates its argument. If it’s not clear why this is important, consider how the following would be bad:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=$foo"
But of course the following version is perfectly safe:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=\$foo"
Note that in the original example, “$1” was used to allow the caller to pass the destination variable name as an argument the function. If your function needs to use the shift command, for instance to handle the remaining arguments as “$#”, then it may be useful to save the value of “$1” in a temporary variable at the beginning of the function.
I have done a script on a machine with this command on it
printf -v $1 %s $2
It's working fine on the server that I am using. But when I copied the scripts to another server, I get this error. What am I missing here?
EDIT: What the code does to my script is it declares variables, but is not localized on a function. Here's the complete function
#Declare each property=value as regular bash variable=value
function getProperty {
for x in $(echo ${1} | tr ":" "\n")
do
set -- $(echo ${x} | tr "=" "\n")
printf -v $1 %s $2
#I tried using declare, but the variables become localized to this function only
done
}
Will export work?
export "${1}=$(printf %s "$2")"
I might be barking up the wrong tree here...
Looks like you're trying to take some input in the form of:
var=value:var=value
This will fail if any part of the input contains whitespace, newlines, or glob characters. See: don't read lines with for. It will also fail if the first parameter is ever empty. Because your expansions are unquoted you're dealing with multiple passes of globbing and word-splitting which will probably lead to problems.
Unfortunately, indirect assignment is one of the biggest failures of Bash. If you do this a lot you should seriously consider switching the entire script to ksh where you can use nameref variables. It's very worth it just for that one feature. (The next version of Bash should solve this).
You should use arrays to deal with collections. If you need to deal with key-value pairs like this then you should use Bash 4 and associative arrays. printf -v is the best way to do indirect assignment to an outer scope in a recent enough Bash. If you have Bash 4.2 then you can use declare -g to assign to a global. If your bash is too old to do any of this, upgrade.
Some techniques plus how to use associative arrays are explained here: http://mywiki.wooledge.org/BashFAQ/006
Also read up on how to quote.