This is my code to read a file line by line:
IFS=$'\n'
myfile="$1"
i=0
while read line; do
echo "Line # $i: '$line'"
let i++
done < "$myfile"
This is the file passed as parameter
Hello
stack
overflow
friends
I execute it like this: test.sh input.txt and I get this result:
'ine # 0: 'Hello
'ine # 1: 'stack
'ine # 2: 'overflow
'ine # 3: 'friends
As you see, The fisrt character is replaced by a quote. And the quote of the final of the line does not appear. Whats going on here? I can't see the mistake? Any idea?
Most likely you have \r before end of line in your input file.
You can test same by using:
cat -vte file
This will show ^M$ in the end of file has dos carriage return \r.
You can use this script to read your file correctly:
i=1
while IFS=$'\r' read -r line; do
echo "Line # $i: '$line'"
let i++
done < "$myfile"
OR else convert your file into unix file using:
dos2unix file
OR If you don't wish to actually save the file stripped off of \r, you can also use:
while read line; do
........# your code as-is
done < <( tr -d '\r' < "$myfile")
Related
In bash script, how can I read the file line by line and assign to the variable with delimiter?
example.txt file contents:
string1
string2
string3
string4
Expected output:
string1,string2,string3,string4
Thanks in advance
Apparently my answer below leaves a comma at the end of the line. A quick workaround is to use the following builtin in Unix:
paste -sd, example.txt
Where you use the paste program to concatenate all the lines into one and then add the string delimiter ','
Using the builtin commands in unix:
tr '\n' ',' < example.txt
This can be broken down as truncating all Newline widcards and inserting a comma delimiter instead.
Other possible ways, just for fun:
mapfile -t a < example.txt
(IFS=,; echo "${a[*]}")
mapfile -t a < example.txt
foo=$(printf '%s' "${a[#]/%/,}")
echo "${foo%,}"
foo=$(<example.txt)
echo "${foo//$'\n'/,}"
{
IFS= read -r foo
while IFS= read -r line; do
foo+=,$line
done
} < example.txt
echo "$foo"
sed ':a;N;$!ba;s/\n/,/g' example.txt
It should work:
#!/bin/bash
output=''
while IFS='' read -r line || [[ -n "$line" ]]; do
output=$output:",$line"
done < "$1"
echo $output
Give the file as argument
I would like to know if I can write a shell script that accepts two arguments simultaneously, one from a file and the another one from stdin. Could you give some example please?.
I trying
while read line
do
echo "$line"
done < "${1}" < "{/dev/stdin}"
But this does not work.
You can use cat - or cat /dev/stdin:
while read line; do
# your code
done < <(cat "$1" -)
or
while read line; do
# your code
done < <(cat "$1" /dev/stdin)
or, if you want to read from all files passed through command line as well as stdin, you could do this:
while read line; do
# your code
done < <(cat "$#" /dev/stdin)
See also:
How to read from a file or stdin in Bash?
This topic seems to be helpful here:
{ cat $1; cat; } | while read line
do
echo "$line"
done
Or just
cat $1
cat
if all you're doing is printing the content
In a bash script, I see several while statements with those redirect signs when closing the loop.
I know that if I end it with "done < file", I am redirecting the file to the stdin of the command in the while statement. But what the others means?
I would appreciate if someone could give an explanation with examples.
With the file text.txt
1aa
2bb
3cc
Redirection:
$ cat < text.txt
1aa
2bb
3cc
Here document:
$ cat << EOF
> 1AA
> 2BB
> EOF
1AA
2BB
Here string:
$ cat <<< 1aaa
1aaa
The first form, <, is an input redirection. It somewhat different than << and <<< which are two variants of a here document.
The first form, <, is primarily used to redirect the contents of a file to a command or process. It is a named FIFO, and therefor a file that is passed to a command that accepts file arguments.
cmd < file
will open the file named file and create a new file name to open and read. The difference between cmd file and cmd < file is the name passed to cmd in the second case is the name of a named pipe.
You can also do process substitution:
cmd <(process)
An example use would be comparing two directories:
diff <(ls dir1) <(ls dir2)
In this case, the command ls dir1 and ls dir2 has output redirected to a file like stream that is then read by diff as if those were two files.
You can see the name of the file device by passing to echo a process substitution:
$ echo <(ls)
/dev/fd/63
Since echo does not support opening files, it just prints the name of the FIFO.
Here documents are easier to demonstrate. The << form has a 'limit string' that is not included in the output:
$ cat <<HERE
> line 1
> line 2
> line 3
> HERE
line 1
line 2
line 3
The HERE is a unique string that must be on its own line.
The 'here string' or <<< form does not require the delimiting string of the << form and is on a single line:
$ cat <<< 'line 1'
line 1
You can also expand parameters:
$ v="some text"
$ cat <<< "$v"
some text
But not other forms of shell expansions:
Brace expansion:
$ echo a{b,c,d}e
abe ace ade
$ cat <<< a{b,c,d}e
a{b,c,d}e
Given a 'generic' Bash while loop that reads input line by line:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done
There are several ways that you can feed input into that loop.
First example, you can redirect a file. For demo, create a 6 line file:
$ seq 6 > /tmp/6.txt
Redirect the input of the file into the loop:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/6.txt
'1'
'2'
'3'
'4'
'5'
'6'
Or, second example, you can directly read from the output of seq using redirection:
$ while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done < <(seq 3)
'1'
'2'
'3'
(Please note the extra < with a space for this form)
Or, third example, you can use a 'HERE' doc separated by CR:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done <<HERE
1
2 3
4
HERE
'1 '
'2 3'
' 4'
Going back to diff which will only work on files, you can use process substitution and a HERE doc or process substitution and redirection to use diff on free text or the output of a program.
Given:
$ cat /tmp/f1.txt
line 1
line 2
line 3
Normally you would need to have a second file to compare free text with that file. You can use a HERE doc and process substitution to skip creating a separate file:
$ diff /tmp/f1.txt <(cat <<HERE
line 1
line 2
line 5
HERE
)
3c3
< line 3
---
> line 5
command < foo
Redirect the file foo to the standard input of command.
command << foo
blah 1
blah 2
foo
Here document: send the following lines up to foo to the standard input of command.
command <<< foo
Here-string. The string foo is sent to the standard input of command.
I have the script
#!/bin/bash
set i=0;
while read line
do
echo "$line";
$i < cat "my.log" | grep -w "$line" | wc -l;
echo "$i";
if [ "$i" == 0 ]; then
cat $line << "notfound.txt"
fi
i=0;
done < "test.txt"
which is giving the error
./test.sh: line 13: warning: here-document at line 10 delimited by end-of-file (wanted `notfound.txt')
./test.sh: line 14: syntax error: unexpected end of file
My goal is to test the value of the variable i. If it is 0 then I would like to redirect the value stored in the variable $line to a file "notfound.txt"
Instead of
cat $line << "notfound.txt"
say:
echo $line > "notfound.txt"
You don't cat variables, you echo those instead. command > file would redirect the output of the command to the file, overwriting it. If you want to append the file, use >> instead.
You can learn more about redirection here.
I am new to shell scripting. I need to read a file that works in all shells that has variables defined in it. Something like:
variable1=test1
variable2=test2
....
I have to read this file line by line and prepare new string separated by spaces, like:
variable=variable1=test1 variable2=test2 ....
I tried with the below code:
while read LINE
do
$VAR="$VAR $LINE"
done < test.dat
but it's throwing me this error:
command not found Test.sh: line 3: = variable1=test1
The problem with your script is the leading $ before var is initialized, try:
#/bin/bash
while read line; do
var="$var $line"
done < file
echo "$var"
However you can do this with the tr command by substituting the newline character with a space.
$ tr '\n' ' ' < file
variable1=test1 variable2=test2
$ var="$(tr '\n' ' ' < file)"
$ echo "$var"
variable1=test1 variable2=test2
When defining a shell variable you must omit the $. So VAR="bla" is right, $VAR="bla" is wrong. The $ is only necessary for using the variable, as in echo $VAR;
while read LINE
do
VAR="$VAR $LINE"
done < test.dat