Bash backslash escaping issue when using read [duplicate] - bash

This question already has answers here:
How to keep backslash when reading from a file?
(4 answers)
Closed 8 years ago.
I have some XSL spitting out samba paths on stdout. I am iterating these paths to locate them on their mountpoints on disk, so have something along the lines of:
while read src dst ; do
...
done < <(xsltproc - file.xml <<XSL
...
XSL
)
Now, I can trivially solve the problem by performing the path escaping either in the XSL stylesheet or by using sed. However, I am curious from a bash perspective, how to solve the problem. Here is a working example of the problem:
a='\\a\b\c\d\e'
ecyo $a
\\a\b\c\d\e
echo ${a//\\//}
//a/b/c/d/e
b=$a
echo $b
\\a\b\c\d\e
b=$(echo $a)
echo $b
\\a\b\c\d\e
That's all fine, does exactly what I expect it to do. This is where bash gets a bit funny:
read b < <(echo $a)
echo $b
\abcde
echo ${b//\\//}
/abcde
As you can see, read has stripped all of the unescaped backslashes when it read them in, so the directory information gets lost.

Reading the bash manual, it seems this works just fine:
read -r b < <(echo $a)
The -r flag tells read not to treat backslashes as escape characters.

Related

Using wildcard in bash-variable for echo and grep [duplicate]

This question already has answers here:
How to grep asterisk without escaping?
(2 answers)
When to wrap quotes around a shell variable?
(5 answers)
Closed 3 years ago.
I am trying to come up with a function that searches a given file for a given pattern. If the pattern is not found it shall be appended to the file.
This works fine for some cases, but when my pattern includes special characters, like wildcards (*) this method fails.
Patterns that work:
pattern="some normal string"
Patterns that don't work:
pattern='#include "/path/to/dir/\*.conf"'
This is my function:
check_pattern() {
if ! grep -q "$1" $2
then
echo $1 >> $2
fi
}
I' calling my function like this:
check_pattern $pattern $target_file
When escaping the wildcard in my pattern variable to get grep running correctly echo interprets the \ as character.
So when I run my script a second time my grep does not find the pattern and appends it again.
To put it simple:
Stuff that gets appended:
#include "/path/to/dir/\*.conf"
Stuff that i want to have appended:
#include "/path/to/dir/*.conf"
Is there some way to get my expected result without storing my echo-pattern in a second variable?
Use
grep -f
and
check_pattern "$pattern" "$target_file"
Thanks all, I got it now.
Using grep -F as pointed out by Gem Taylor in combination with calling my function like this check_pattern "$pattern" "$target_file" did the tricks.

Cannot echo bash variable from single line command [duplicate]

This question already has answers here:
Rename multiple files, but only rename part of the filename in Bash
(6 answers)
Closed 4 years ago.
I have the following file names where I am trying to relabel v5.4b to v5.7:
v5.4b_lvl-1.e8974326
v5.4b_lvl-1.o8974326
v5.4b_lvl-1.pe8974326
v5.4b_lvl-1.po8974326
v5.4b_lvl-2.1.e8974303
v5.4b_lvl-2.1.o8974303
v5.4b_lvl-2.1.pe8974303
v5.4b_lvl-2.1.po8974303
v5.4b_lvl-2.2.e8974304
v5.4b_lvl-2.2.o8974304
v5.4b_lvl-2.2.pe8974304
v5.4b_lvl-2.2.po8974304
v5.4b_lvl-3.1.e8974305
v5.4b_lvl-3.1.o8974305
v5.4b_lvl-3.1.pe8974305
v5.4b_lvl-3.1.po8974305
v5.4b_lvl-4.1.e8974327
v5.4b_lvl-4.1.o8974327
v5.4b_lvl-4.1.pe8974327
v5.4b_lvl-4.1.po8974327
I can't do mv v5.4b_* v5.7_* because it thinks v5.7_* is a directory so I am trying a for-loop but I can't get it to work
I am trying the recommended answer from this SO post How to set a variable to the output of a command in Bash? but getting a bunch of empty lines.
What am I doing incorrectly? How can I save the output of cut to SUFFIX so I can mv $i v5.7_$SUFFIX?
-bash-4.1$ for i in v5.4b*; do echo $i | SUFFIX=`cut -f2 -d'_'`; echo ${SUFFIX}; done
You've got echo $i in the wrong place. The output of that command needs to be piped to cut for it to read anything, then the result is assigned to SUFFIX:
for i in v5.4b*
do
SUFFIX=`echo $i | cut -f2 -d'_'`
echo ${SUFFIX}
done
If you rename utility then just do:
rename -n 's/v5\4.b/v5.7/' v5.4b*
PS: -n is for dry-run. You may remove it later for real renaming.
If rename is not available then use:
for i in v5.4b*; do
echo mv "$i" "${i/v5.4b/v5.7}"
done
Remove 'echo` if you're satisfied with the output.

Bash/Shell | How to prioritize quote from IFS in read [duplicate]

This question already has answers here:
IFS separate a string like "Hello","World","this","is, a boring", "line"
(3 answers)
Closed 6 years ago.
I'm working with a hand fill file and I am having issue to parse it.
My file input file cannot be altered, and the language of my code can't change from bash script.
I made a simple example to make it easy for you ^^
var="hey","i'm","happy, like","you"
IFS="," read -r one two tree for five <<<"$var"
echo $one:$two:$tree:$for:$five
Now I think you already saw the problem here. I would like to get
hey:i'm:happy, like:you:
but I get
hey:i'm:happy: like:you
I need a way to tell the read that the " " are more important than the IFS. I have read about the eval command but I can't take that risk.
To end this is a directory file and the troublesome field is the description one, so it could have basically anything in it.
original file looking like that
"type","cn","uid","gid","gecos","description","timestamp","disabled"
"type","cn","uid","gid","gecos","description","timestamp","disabled"
"type","cn","uid","gid","gecos","description","timestamp","disabled"
Edit #1
I will give a better exemple; the one I use above is too simple and #StefanHegny found it cause another error.
while read -r ldapLine
do
IFS=',' read -r objectClass dumy1 uidNumber gidNumber username description modifyTimestamp nsAccountLock gecos homeDirectory loginShell createTimestamp dumy2 <<<"$ldapLine"
isANetuser=0
while IFS=":" read -r -a class
do
for i in "${class[#]}"
do
if [ "$i" == "account" ]
then
isANetuser=1
break
fi
done
done <<< $objectClass
if [ $isANetuser == 0 ]
then
continue
fi
#MORE STUFF APPEND#
done < file.csv
So this is a small part of the code but it should explain what I do. The file.csv is a lot of lines like this:
"top:shadowAccount:account:posixAccount","Jdupon","12345","6789","Jdupon","Jean Mark, Dupon","20140511083750Z","","Jean Mark, Dupon","/home/user/Jdupon","/bin/ksh","20120512083750Z","",""
If the various bash versions you will use are all more recent than v3.0, when regexes and BASH_REMATCH were introduced, you could use something like the following function: [Note 1]
each_field () {
local v=,$1;
while [[ $v =~ ^,(([^\",]*)|\"[^\"]*\") ]]; do
printf "%s\n" "${BASH_REMATCH[2]:-${BASH_REMATCH[1]:1:-1}}";
v=${v:${#BASH_REMATCH[0]}};
done
}
It's argument is a single line (remember to quote it!) and it prints each comma-separated field on a separate line. As written, it assumes that no field has an enclosed newline; that's legal in CSV, but it makes dividing the file into lines a lot more complicated. If you actually needed to deal with that scenario, you could change the \n in the printf statement to a \0 and then use something like xargs -0 to process the output. (Or you could insert whatever processing you need to do to the field in place of the printf statement.)
It goes to some trouble to dequote quoted fields without modifying unquoted fields. However, it will fail on fields with embedded double quotes. That's fixable, if necessary. [Note 2]
Here's a sample, in case that wasn't obvious:
while IFS= read -r line; do
each_field "$line"
printf "%s\n" "-----"
done <<EOF
type,cn,uid,gid,gecos,"description",timestamp,disabled
"top:shadowAccount:account:posixAccount","Jdupon","12345","6789","Jdupon","Jean Mark, Dupon","20140511083750Z","","Jean Mark, Dupon","/home/user/Jdupon","/bin/ksh","20120512083750Z","",""
EOF
Output:
type
cn
uid
gid
gecos
description
timestamp
disabled
-----
top:shadowAccount:account:posixAccount
Jdupon
12345
6789
Jdupon
Jean Mark, Dupon
20140511083750Z
Jean Mark, Dupon
/home/user/Jdupon
/bin/ksh
20120512083750Z
-----
Notes:
I'm not saying you should use this function. You should use a CSV parser, or a language which includes a good CSV parsing library, like python. But I believe this bash function will work, albeit slowly, on correctly-formatted CSV files of a certain common CSV dialect.
Here's a version which handles doubled quotes inside quoted fields, which is the classic CSV syntax for interior quotes:
each_field () {
local v=,$1;
while [[ $v =~ ^,(([^\",]*)|\"(([^\"]|\"\")*)\") ]]; do
echo "${BASH_REMATCH[2]:-${BASH_REMATCH[3]//\"\"/\"}}";
v=${v:${#BASH_REMATCH[0]}};
done
}
My suggestion, as in some previous answers (see below), is to switch the separator to | (and use IFS="|" instead):
sed -r 's/,([^,"]*|"[^"]*")/|\1/g'
This requires a sed that has extended regular expressions (-r) however.
Should I use AWK or SED to remove commas between quotation marks from a CSV file? (BASH)
Is it possible to write a regular expression that matches a particular pattern and then does a replace with a part of the pattern

Getting Bash to parse variables from file input [duplicate]

This question already has answers here:
Forcing bash to expand variables in a string loaded from a file
(13 answers)
Closed 7 years ago.
Let's say I have a file called path.txt containing the text $HOME/filedump/ on a single line. How can I then read the contents of path.txt into a variable, while having Bash parse said content?
Here's an example of what I'm trying to do:
#!/bin/bash
targetfile="path.txt"
target=$( [[ -f $targetfile ]] && echo $( < $targetfile ) || echo "Not set" )
echo $target
Desired output: /home/joe/filedump/
Actual output: $HOME/filedump/
I've tried using cat in place of <, wrapping it in quotes, and more. Nothing seems to get me anywhere.
I'm sure I'm missing something obvious, and there's probably a simple builtin command. All I can find on Google is pages about reading variables from ini/config files or splitting one string into multiple variables.
If you want to evaluate the contents of path.txt and assign that to target, then use:
target=$(eval echo $(<path.txt))
for example:
$ target=$(eval echo $(<path.txt)); echo "$target"
/home/david/filedump/
This might not necessarily suit your needs (depending on the context of the code you provided), but the following worked for me:
targetfile="path.txt"
target=$(cat $targetfile)
echo $target
Here's a safer alternative than eval. In general, you should not be using configuration files that require bash to evaluate their contents; that just opens a security risk in your script. Instead, detect if there is something that requires evaluation, and handle it explicitly. For example,
IFS= read -r path < path.txt
if [[ $path =~ '$HOME' ]]; then
target=$HOME/${path#\$HOME}
# more generally, target=${path/\$HOME/$HOME}, but
# when does $HOME ever appear in the *middle* of a path?
else
target=$path
fi
This requires you to know ahead of time what variables might appear in path.txt, but that's a good thing. You should not be evaluating unknown code.
Note that you can use any placeholder instead of a variable in this case; %h/filedump can be detected and processed just as easily as $HOME/filedump, without the presumption that the contents can or should be evaluated as shell code.

bash script error possibly related to length of filename (actually I don't know what's wrong) [duplicate]

This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 5 years ago.
Here's an example of my problematic code:
#!/bin/bash
fileList='fileList.txt'
#IFS=$'\n'
while read filename
do
echo listing "$filename"
ls -ligG "$filename"
done < "$fileList"
echo "done."
#unset IFS
exit 0
The output is:
listing /some/long/path/README.TXT
ls: cannot access /some/long/pa
: No such file or directoryDME.TXT
Notice that ls cuts off the path. Also notice that the end of the path/filename is appended to the error message (after "No such file or directory").
I just tested it with a path exactly this long and it still gives the error:
/this/is/an/example/of/shorter/name.txt
Anyone know what's going on? I've been messing with this for hours already :-/
In response to torek's answer, here is more info:
First, here's the modified script based on torek's suggestions:
#!/bin/bash
fileList=/settings/Scripts/fileList.txt
while IFS=$'\n' read -r filename
do
printf 'listing %q\n' "$filename"
ls -ligG $filename
done < "$fileList"
echo "done."
exit 0
Here's the output of that:
# ./test.sh
listing $'/example/pathname/myfile.txt\r'
: No such file or directorypathname/myfile.txt
done.
Notice there is some craziness going on still.
Here's the file. It does exist.
ls -ligG /example/pathname/myfile.txt
106828 -rwxrwx--- 1 34 Mar 28 00:55 /example/pathname/myfile.txt
Based on the unusual behavior, I'm going to say the file has CRLF line terminators. Your file names actually have an invisible carriage return appended to the name. In echo, this doesn't show up, since it just jumps to the first column then prints a newline. However, ls tries to access the file including the hidden carriage return, and in its error message, the carriage return causes the error message partially overwrite your path.
To trim these chars away, you can use tr:
tr -d '\r' < fileList.txt > fileListTrimmed.txt
and try using that file instead.
That embedded newline is a clue: the error message should read ls: cannot access /some/long/path/README.TXT: No such file or directory (no newline after the "a" in "path"). Even if there were some mysterious truncation happening, the colon should happen right after the "a" in "path". It doesn't, so, the string is not what it seems to be.
Try:
printf 'listing %q\n' "$filename"
for printing the file name before invoking ls. Bash's built-in printf has a %q format that will quote funny characters.
I'm not sure what the intent of the commented-out IFS-setting is. Perhaps you want to prevent read from splitting at whitespace? You can put the IFS= in front of the read, and you might want to use read -r as well:
while IFS=$'\n' read -r filename; do ...; done < "$fileList"

Resources