test#hwy:~$ echo "//" | xargs -0
//
test#hwy:~$
Why there is a newline here?Why the result is not as below?
test#hwy:~$ echo "//" | xargs -0
//
test#hwy:~$
Man xargs can't tell me reason.
-0 Input items are terminated by a null character instead of by
whitespace, and the quotes and backslash are not special (every
character is taken literally). Disables the end of file string,
which is treated like any other argument.
echo prints a newline and xargs is told to not consider a special character.
You may get the wanted effect by using the -n parameter to echo.
Related
I'm trying to pipe a list of values through xargs. Here's a simple example:
echo "Hello Hola Bonjour" | xargs -I _ echo _ Landon
I would expect this to output the following:
Hello Landon
Hola Landon
Bonjour Landon
Instead, the command outputs this:
Hello Hola Bonjour Landon
What am I missing?
Under -I, man xargs says
unquoted blanks do not terminate input items; instead the
separator is the newline character
You can specify a different delimiter (at least in GNU xargs):
printf 'Hello Hola Bonjour' | xargs -d' ' -I _ echo _ Landon
More portably, use \0 as the delimiter and -0 to use it:
printf '%s\0' Hello Hola Bonjour | xargs -0 -I _ echo _ Landon
The delimiter needs to be changed, and you also have to specify the -L option. So either change the delimiter via -d like the other answer suggested, or pipe to sed and replace space with linefeed
echo "Hello Hola Bonjour" | sed -e 's/ /\n/g' | xargs -L 1 -I _ echo _ Landon
Results in
Hello Landon
Hola Landon
Bonjour Landon
Sometimes changing the delimiters is not enough. xargs will sometimes take all the input arguments and pass it all at once. There is no splitting of the arguments.
e.g.
seq 1 7 | xargs echo
results in
1
2
3
4
5
6
7
being passed to xargs, so the output would be
1 2 3 4 5 6 7
If you add a -L 1 shown in the xargs man page
-L max-lines
Use at most max-lines nonblank input lines per command line. Trailing blanks cause an input line to be
logically continued on the next input line. Implies -x.
seq 1 7 | xargs -L 1 echo
then you will see
1
2
3
4
5
6
7
You can also covert it manually to a for loop which lets you setup multi line statements more easily.
# be absolutely sure the values the for loop iterates over is well sanitized to avoid glob expansion of the *.
for i in Hello Hola Bonjour
do
if [ "$i" = "Hello" ]
then
echo "$i Landon, language detected as English!"
else
echo "$i Landon, language detected as non English."
fi
done
The -I flag changes the delimiter to newline.
Unquoted blanks do not terminate input items; instead the separator is the newline character.
You can read about it here.
You have to manually specify the delimiter to be a space. Echo also inserts a newline by default, which messes up xargs. Use the -n flag to remove the newline.
Here is the fixed command:
echo -n "Hello Hola Bonjour" | xargs -d' ' -I _ echo _ Landon
I am trying to only change the first letter of a string to lowercase using a Shell script. Ideally a simple way to go from CamelCase to lowerCamelCase.
GOAL:
$DIR="SomeString"
# missing step
$echo $DIR
someString
I have found some great resources for doing this to the entire string but not just altering the first letter and leaving the remaining string untouched.
If your shell is recent enough, you can use the following parameter expansion:
DIR="SomeString" # Note the missing dollar sign.
echo ${DIR,}
Alternative solution (will work on old bash too)
DIR="SomeString"
echo $(echo ${DIR:0:1} | tr "[A-Z]" "[a-z]")${DIR:1}
prints
someString
for assing to variable
DIR2="$(echo ${DIR:0:1} | tr "[A-Z]" "[a-z]")${DIR:1}"
echo $DIR2
prints
someString
alternative perl
DIR3=$(echo SomeString | perl -ple 's/(.)/\l$1/')
DIR3=$(echo SomeString | perl -nle 'print lcfirst')
DIR3=$(echo "$DIR" | perl -ple 's/.*/lcfirst/e'
some terrible solutions;
DIR4=$(echo "$DIR" | sed 's/^\(.\).*/\1/' | tr "[A-Z]" "[a-z]")$(echo "$DIR" | sed 's/^.//')
DIR5=$(echo "$DIR" | cut -c1 | tr '[[:upper:]]' '[[:lower:]]')$(echo "$DIR" | cut -c2-)
All the above is tested with OSX's /bin/bash.
With sed:
var="SomeString"
echo $var | sed 's/^./\L&/'
^ means the start of the line
\L is the command to make the match in lowercase
& is the whole match
Perl solution:
DIR=SomeString
perl -le 'print lcfirst shift' "$DIR"
Since awk hasn't yet been mentioned, here's another way you could do it (requires GNU awk):
dir="SomeString"
new_dir=$(awk 'BEGIN{FS=OFS=""}{$1=tolower($1)}1' <<<"$dir")
This sets the input and output field separators to an empty string, so each character is a field. The tolower function does what you think it does. 1 at the end prints the line. If your shell doesn't support <<< you can do echo "$dir" | awk ... instead.
If you are looking for a POSIX compliant solution then have a look at typeset.
var='SomeString'
typeset -lL1 b="$var"
echo "${b}${var#?}"
Output:
someString
The typeset command creates a special variable that is lowercase, left aligned and one char long. ${var#?} trims the first occurrence of pattern from the start of $var and ? matches a single
character.
I am new to shell scripts. I want to read a file line by line, which contains arguments and if the arguments contains any spaces in it, I want to replace it by enclosing with quotes.
For example if the file (test.dat) contains:
-DtestArgument1=/path/to a/text file
-DtestArgument2=/path/to a/text file
After parsing the above file, shell script should prepare the string with following:
-DtestArgument1="/path/to a/text file" -DtestArgument2="/path/to a/text file"
Here is my shell script:
while read ARGUMENT; do
ARGUMENT=`echo ${ARGUMENT} | tr "\n" " "`
if [[ "${ARGUMENT}" =~ " " ]]; then
ARGUMENT=`echo $ARGUMENT | sed 's/\^(-D.*\)=(.*)/\1=\"\2\"/g'`
NEW_ARGUMENT="${NEW_ARGUMENT} ${ARGUMENT}"
else
echo "doesn't contains spaces"
NEW_ARGUMENT="${NEW_ARGUMENT} ${ARGUMENT}"
fi
done < test.dat
But it's throwing the following error:
sed: -e expression #1, char 28: Unmatched ) or \)
The code should be compatible with all shells.
I think you should simplify the problem. Rather than worrying about spaces, just quote the argument after the =. Something like:
sed -e 's/=/="/' -e 's/$/"/' test.dat | paste -s -d\ -
Should be sufficient. If you really care about spaces, you could try something like:
sed -e '/=.* /{ s/=/="/; s/$/"/; }' test.dat | paste -s -d\ -
That will only notice spaces after the =. Just use / / if you really want to change any line that has a space anywhere.
There's no need to use a while/read loop: just let sed read the file directly.
The sed parentheses should be escaped:
ARGUMENT=`echo $ARGUMENT | sed "s/\^\(-D.*\)=\(.*\)/\1=\"\2\"/g"`
One place you did, in 3 places you forgot... BTW, I generally use " quotation.
If you prefer '-style, do like this:
ARGUMENT=`echo $ARGUMENT | sed 's/\^(-D.*)=(.*)/\1="\2"/g'`
How to remove extra spaces in variable HEAD?
HEAD=" how to remove extra spaces "
Result:
how to remove extra spaces
Try this:
echo "$HEAD" | tr -s " "
or maybe you want to save it in a variable:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
Update
To remove leading and trailing whitespaces, do this:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
NEWHEAD=${NEWHEAD%% }
NEWHEAD=${NEWHEAD## }
Using awk:
$ echo "$HEAD" | awk '$1=$1'
how to remove extra spaces
Take advantage of the word-splitting effects of not quoting your variable
$ HEAD=" how to remove extra spaces "
$ set -- $HEAD
$ HEAD=$*
$ echo ">>>$HEAD<<<"
>>>how to remove extra spaces<<<
If you don't want to use the positional paramaters, use an array
ary=($HEAD)
HEAD=${ary[#]}
echo "$HEAD"
One dangerous side-effect of not quoting is that filename expansion will be in play. So turn it off first, and re-enable it after:
$ set -f
$ set -- $HEAD
$ set +f
This horse isn't quite dead yet: Let's keep beating it!*
Read into array
Other people have mentioned read, but since using unquoted expansion may cause undesirable expansions all answers using it can be regarded as more or less the same. You could do
set -f
read HEAD <<< $HEAD
set +f
or you could do
read -rd '' -a HEAD <<< "$HEAD" # Assuming the default IFS
HEAD="${HEAD[*]}"
Extended Globbing with Parameter Expansion
$ shopt -s extglob
$ HEAD="${HEAD//+( )/ }" HEAD="${HEAD# }" HEAD="${HEAD% }"
$ printf '"%s"\n' "$HEAD"
"how to remove extra spaces"
*No horses were actually harmed – this was merely a metaphor for getting six+ diverse answers to a simple question.
Here's how I would do it with sed:
string=' how to remove extra spaces '
echo "$string" | sed -e 's/ */ /g' -e 's/^ *\(.*\) *$/\1/'
=> how to remove extra spaces # (no spaces at beginning or end)
The first sed expression replaces any groups of more than 1 space with a single space, and the second expression removes any trailing or leading spaces.
echo -e " abc \t def "|column -t|tr -s " "
column -t will:
remove the spaces at the beginning and at the end of the line
convert tabs to spaces
tr -s " " will squeeze multiple spaces to single space
BTW, to see the whole output you can use cat - -A: shows you all spacial characters including tabs and EOL:
echo -e " abc \t def "|cat - -A
output: abc ^I def $
echo -e " abc \t def "|column -t|tr -s " "|cat - -A
output:
abc def$
Whitespace can take the form of both spaces and tabs. Although they are non-printing characters and unseen to us, sed and other tools see them as different forms of whitespace and only operate on what you ask for. ie, if you tell sed to delete x number of spaces, it will do this, but the expression will not match tabs. The inverse is true- supply a tab to sed and it will not match spaces, even if the number of them is equal to those in a tab.
A more extensible solution that will work for removing either/both additional space in the form of spaces and tabs (I've tested mixing both in your specimen variable) is:
echo $HEAD | sed 's/^[[:blank:]]*//g'
or we can tighten-up #Frontear 's excellent suggestion of using xargs without the tr:
echo $HEAD | xargs
However, note that xargs would also remove newlines. So if you were to cat a file and pipe it to xargs, all the extra space- including newlines- are removed and everything put on the same line ;-).
Both of the foregoing achieved your desired result in my testing.
Try this one:
echo ' how to remove extra spaces ' | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g'
or
HEAD=" how to remove extra spaces "
HEAD=$(echo "$HEAD" | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g')
I would make use of tr to remove the extra spaces, and xargs to trim the back and front.
TEXT=" This is some text "
echo $(echo $TEXT | tr -s " " | xargs)
# [...]$ This is some text
echo variable without quotes does what you want:
HEAD=" how to remove extra spaces "
echo $HEAD
# or assign to new variable
NEW_HEAD=$(echo $HEAD)
echo $NEW_HEAD
output: how to remove extra spaces
I'm using
sed -e "s/\*DIVIDER\*/$DIVIDER/g" to replace *DIVIDER* with a user-specified string, which is stored in $DIVIDER. The problem is that I want them to be able to specify escape characters as their divider, like \n or \t. When I try this, I just end up with the letter n or t, or so on.
Does anyone have any ideas on how to do this? It will be greatly appreciated!
EDIT: Here's the meat of the script, I must be missing something.
curl --silent "$URL" > tweets.txt
if [[ `cat tweets.txt` == *\<error\>* ]]; then
grep -E '(error>)' tweets.txt | \
sed -e 's/<error>//' -e 's/<\/error>//' |
sed -e 's/<[^>]*>//g' |
head $headarg | sed G | fmt
else
echo $REPLACE | awk '{gsub(".", "\\\\&");print}'
grep -E '(description>)' tweets.txt | \
sed -n '2,$p' | \
sed -e 's/<description>//' -e 's/<\/description>//' |
sed -e 's/<[^>]*>//g' |
sed -e 's/\&\;/\&/g' |
sed -e 's/\<\;/\</g' |
sed -e 's/\>\;/\>/g' |
sed -e 's/\"\;/\"/g' |
sed -e 's/\&....\;/\?/g' |
sed -e 's/\&.....\;/\?/g' |
sed -e 's/^ *//g' |
sed -e :a -e '$!N;s/\n/\*DIVIDER\*/;ta' | # Replace newlines with *divider*.
sed -e "s/\*DIVIDER\*/${DIVIDER//\\/\\\\}/g" | # Replace *DIVIDER* with the actual divider.
head $headarg | sed G
fi
The long list of sed lines are replacing characters from an XML source, and the last two are the ones that are supposed to replace the newlines with the specified character. I know it seems redundant to replace a newline with another newline, but it was the easiest way I could come up with to let them pick their own divider. The divider replacement works great with normal characters.
You can use bash to escape the backslash like this:
sed -e "s/\*DIVIDER\*/${DIVIDER//\\/\\\\}/g"
The syntax is ${name/pattern/string}. If pattern begins with /, every occurence of pattern in name is replaced by string. Otherwise only the first occurence is replaced.
Maybe:
case "$DIVIDER" in
(*\\*) DIVIDER=$(echo "$DIVIDER" | sed 's/\\/\\\\/g');;
esac
I played with this script:
for DIVIDER in 'xx\n' 'xxx\\ddd' "xxx"
do
echo "In: <<$DIVIDER>>"
case "$DIVIDER" in (*\\*) DIVIDER=$(echo "$DIVIDER" | sed 's/\\/\\\\/g');;
esac
echo "Out: <<$DIVIDER>>"
done
Run with 'ksh' or 'bash' (but not 'sh') on MacOS X:
In: <<xx\n>>
Out: <<xx\\n>>
In: <<xxx\\ddd>>
Out: <<xxx\\\\ddd>>
In: <<xxx>>
Out: <<xxx>>
It seems to be a simple substitution:
$ d='\n'
$ echo "a*DIVIDER*b" | sed "s/\*DIVIDER\*/$d/"
a
b
Maybe I don't understand what you're trying to accomplish.
Then maybe this step could take the place of the last two of yours:
sed -n ":a;$ {s/\n/$DIVIDER/g;p;b};N;ba"
Note the space after the dollar sign. It prevents the shell from interpreting "${s..." as a variable name.
And as ghostdog74 suggested, you have way too many calls to sed. You may be able to change a lot of the pipe characters to backslashes (line continuation) and delete "sed" from all but the first one (leave the "-e" everywhere). (untested)
You just need to escape the escape char.
\n will match \n
\ will match \
\\ will match \
Using FreeBSD sed (e.g. on Mac OS X) you have to preprocess the $DIVIDER user input:
d='\n'
d='\t'
NL=$'\\\n'
TAB=$'\\\t'
d="${d/\\n/${NL}}"
d="${d/\\t/${TAB}}"
echo "a*DIVIDER*b" | sed -E -e "s/\*DIVIDER\*/${d}/"