I have a text file with the following contents,
My test
strings
that dont have
a question
mark except this line?
but not
these two
and when i try to read the file in bash using, for example,
ph_lines="/path/to/file.txt"
for l in $(cat "$ph_lines")
do
echo "$l"
done
everything prints on the output except for the string with the question mark in it.
I have tried using while read line; echo line; done < $filename and it still has the same problem
The only thing that would work to capture all of the lines is when i used sed to remove question marks.
for l in $(cat ${ph_lines} | sed $'s/\?//' )
Thank you!
To correctly read from a file, use the example from Read a file line by line assigning the value to a variable:
while IFS= read -r line; do
echo "$line"
done < my_filename.txt
The difference is not that it's a while loop (because you've tried that), but because this loop is correctly quoted. The problem you're seeing happens because you secretly enabled nullglob first, and then neglected to quote:
$ shopt -s nullglob
$ var='question?'
$ echo "$var"
question?
$ echo $var
(blank line)
Unquoted expansion causes pathname expansion, and since you enabled nullglob and have no matching files, the previous example shows nothing. If you had some matching files, you'd see those instead:
$ touch questions question2
$ echo $var
question2 questions
You can set up shellcheck in your editor to get automatic warnings about these issues.
Related
I need to read from a variable line by line, do some operations with every line and then work with the data afterwards. I have to work in sh.
I already tried this, but $VAR is empty since I assume, that I saved into it in a subshell
#!/bin/sh POSIXLY_CORRECT=yes
STRING="a\nb\nc\n"
echo $STRING | while read line; do
*some operations with line*
VAR=$(echo "$VAR$line")
done
echo $VAR
I also tried redirecting a variable...
done <<< $STRING
done < $(echo $STRING)
done < < $(echo $STRING)
...and so on, but only got No such file or Redirection unexpected
Please help.
As you've guessed, variable assignments in a subshell aren't seen by the parent, and the commands in a pipelines are run in subshells.
Having to work in plain sh is a real buzzkill. All right, all right, here are a few ideas. One is to extend the life of the subshell and do your work after the loop ends:
string="a
b
c"
echo "$string" | {
var=
while IFS= read -r line; do
*some operations with line*
var=$var$line
done
echo "$var"
}
Another is to use a heredoc (<<), since sh doesn't have herestrings (<<<).
var=
while IFS= read -r line; do
*some operations with line*
var=$var$line
done <<STR
$var
STR
Other improvements:
Escapes aren't interpreted in string literals, so if you want literal newlines put literal newlines, not \n.
Make sure to empty out $var with var=. You don't want an environment variable leaking into your script.
Quote variable expansions. Not needed in assignments, though.
$(echo foo) is an anti-pattern: you can just write foo.
It's best to keep variables lowercase. All uppercase is reserved for shell variables.
Use IFS= and -r to keep read from stripping leading whitespace or interpreting backslashes.
Supposing we have a file list.txt:
line 1
line 2
if I use this:
for line in $(cat list.txt)
do
echo $line"_"
done
Even if I do:
OLD_IFS=$IFS
$IFS='$'
I get:
line1
line2_
and not:
line1_
line2_
How can I solve the problem?
Ignoring your incorrect assignment $IFS='$', $ does not mean newline, linefeed or end-of-line. It means literal dollar sign.
To assign a line feed, use
IFS=$'\n'
However, do not attempt to use this to iterate over lines. Instead, use a while read loop, which will not expand globs or collapse empty lines:
while IFS= read -r line
do
echo "${line}_"
done < file
or with similar benefits, read the lines into an array with mapfile:
mapfile -t lines < file
for line in "${lines[#]}"
do
echo "${line}_"
done
I have code that requires a response within a for loop.
Prior to the loop I set IFS="\n"
Within the loop echo -n is ignored (except for the last line).
Note: this is just an example of the behavior of echo -n
Example:
IFS='\n'
for line in `cat file`
do
echo -n $line
done
This outputs:
this is a test
this is a test
this is a test$
with the user prompt occuring only at the end of the last line.
Why is this occuring and is there a fix?
Neither IFS="\n" nor IFS='\n' set $IFS to a newline; instead they set it to literal \ followed by literal n.
You'd have to use an ANSI C-quoted string in order to assign an actual newline: IFS=$'\n'; alternatively, you could use a normal string literal that contains an actual newline (spans 2 lines).
Assigning literal \n had the effect that the output from cat file was not split into lines, because an actual newline was not present in $IFS; potentially - though not with your sample file content - the output could have been split into fields by embedded \ and n characters.
Without either, the entire file contents were passed at once, resulting in a single iteration of your for loop.
That said, your approach to looping over lines from a file is ill-advised; try something like the following instead:
while IFS= read -r line; do
echo -n "$line"
done < file
Never use for loops when parsing files in bash. Use while loops instead. Here is a really good tutorial on that.
http://mywiki.wooledge.org/BashFAQ/001
I'm doing the following, which basically works.
The script tries to insert some lines into a file to rewrite it.
But it is stripping all blank lines and also all line padding.
The main problem is that it does not process the last line of the file.
I'm not sure why.
while read line; do
<... process some things ...>
echo ${line}>> "${ACTION_PATH_IN}.work"
done < "${ACTION_PATH_IN}"
What can be done to fix this?
while IFS= read -r line; do
## some work
printf '%s\n' "$line" >> output
done < <(printf '%s\n' "$(cat input)")
An empty IFS tells read not to strip leading and trailing whitespace.
read -r prevents backslash at EOL from creating a line continuation.
Double-quote your parameter substitution ("$line") to prevent the shell from doing word splitting and globbing on its value.
Use printf '%s\n' instead of echo because it is reliable when processing values like like -e, -n, etc.
< <(printf '%s\n' "$(cat input)") is an ugly way of LF terminating the contents of input. Other constructions are possible, depending on your requirements (pipe instead of redirect from process substitution if it is okay that your whole while runs in a subshell).
It might be better if you just ensured that it was LF-terminated before processing it.
Best yet, use a tool such as awk instead of the shell's while loop. First, awk is meant for parsing/manipulating files so for a huge file to process, awk has the advantage. Secondly, you won't have to care whether you have the last newline or not (for your case).
Hence the equivalent of your while read loop:
awk '{
# process lines
# print line > "newfile.txt"
}' file
One possible reason for not reading the last line is that the file does not end with a newline. On the whole, I'd expect it to work even so, but that could be why.
On MacOS X (10.7.1), I got this output, which is the behaviour you are seeing:
$ /bin/echo -n Hi
Hi$ /bin/echo -n Hi > x
$ while read line; do echo $line; done < x
$
The obvious fix is to ensure that the file ends with a newline.
First thing, use
echo "$line" >> ...
Note the quotes. If you don't put them, the shell itself will remove the padding.
As for the last line, it is strange. It may have to do with whether the last line of the file is terminated by a \n or not (it is a good practice to do so, and almost any editor will do that for you).
A simple yet annoying thing:
Using a script like this:
while read x; do
echo "$x"
done<file
on a file containing whitespace:
text
will give me an output without the whitespace:
text
The problem is i need this space before text (it's one tab mostly but not always).
So the question is: how to obtain identical lines as are in input file in such a script?
Update: Ok, so I changed my while read x to while IFS= read x.
echo "$x" gives me correct answer without stripping first tab, but, eval "echo $x" strips this tab.
What should I do then?
read is stripping the whitespace. Wipe $IFS first.
while IFS= read x
do
echo "$x"
done < file
The entire contents of the read are put into a variable called REPLY. If you use REPLY instead of 'x', you won't have to worry about read's word splitting and IFS and all that.
I ran into the same trouble you are having when attempting to strip spaces off the end of filenames. REPLY came to the rescue:
find . -name '* ' -depth -print | while read; do mv -v "${REPLY}" "`echo "${REPLY}" | sed -e 's/ *$//'`"; done
I found the solution to the problem 'eval "echo $x" strips this tab.' This should fix it:
eval "echo \"$x\""
I think this causes the inner (escaped) quotes will be evaluated with the echo, whereas I think that both
eval "echo $x"
and
eval echo "$x"
cause the quotes to be evaluated before the echo, which means that the string passed to echo has no quotes, causing the white space to be lost. So the complete answer is:
while IFS= read x
do
eval "echo \"$x\""
done < file