I have read the config file which has the below variable:
export BASE_DIR="\usr\usr1"
In the same script I read a file line by line and I wanted to substitute the ${BASE_DIR} with \usr\usr1.
In the script:
while read line; do
echo $line
done <file.txt
${BASE_DIR}\path1 should be printed as \usr\usr1\path1
Tried eval echo and $(( )).
Can use sed, This command will search and replace a value. The dollar sign is the separator.
sed -ie 's$\${BASE_DIR}$\\usr\\usr1$1' hello.txt
You need to set the variable when you read the line that contains the assignment. Then you can replace it later.
#!/bin/bash
while read line; do
if [[ $line =~ ^BASE_DIR= ]]
then basedir=${line#BASE_DIR=}
fi
line=${line/'${BASE_DIR}'/$basedir}
printf "%s\n" "$line"
done < file.txt > newfile.txt
Related
In bash script, how can I read the file line by line and assign to the variable with delimiter?
example.txt file contents:
string1
string2
string3
string4
Expected output:
string1,string2,string3,string4
Thanks in advance
Apparently my answer below leaves a comma at the end of the line. A quick workaround is to use the following builtin in Unix:
paste -sd, example.txt
Where you use the paste program to concatenate all the lines into one and then add the string delimiter ','
Using the builtin commands in unix:
tr '\n' ',' < example.txt
This can be broken down as truncating all Newline widcards and inserting a comma delimiter instead.
Other possible ways, just for fun:
mapfile -t a < example.txt
(IFS=,; echo "${a[*]}")
mapfile -t a < example.txt
foo=$(printf '%s' "${a[#]/%/,}")
echo "${foo%,}"
foo=$(<example.txt)
echo "${foo//$'\n'/,}"
{
IFS= read -r foo
while IFS= read -r line; do
foo+=,$line
done
} < example.txt
echo "$foo"
sed ':a;N;$!ba;s/\n/,/g' example.txt
It should work:
#!/bin/bash
output=''
while IFS='' read -r line || [[ -n "$line" ]]; do
output=$output:",$line"
done < "$1"
echo $output
Give the file as argument
I am trying to parse a huge text file, say 200mb.
the text file contains some strings
123
1234
12345
12345
so my script looked like
while read line ; do
echo "$line"
done <textfile
however using this above method, my string " 12345" gets truncated to "12345"
I tried using
sed -n "$i"p textfile
but the the throughput is reduced from 27 to 0.2 lines per second, which is inacceptable ;-)
any Idea howto solve this?
You want to echo the lines without a fieldsep:
while IFS="" read line; do
echo "$line"
done <<< " 12345"
When you also want to skip interpretation of special characters, use
while IFS="" read -r line; do
echo "$line"
done <<< " 12345"
You can write the IFS without double quotes:
while IFS= read -r line; do
echo "$line"
done <<< " 12345"
This seems to be what you're looking for:
while IFS= read line; do
echo "$line"
done < textfile
The safest method is to use read -r in comparison to just read which will skip interpretation of special characters (thanks Walter A):
while IFS= read -r line; do
echo "$line"
done < textfile
OPTION 1:
#!/bin/bash
# read whole file into array
readarray -t aMyArray < <(cat textfile)
# echo each line of the array
# this will preserve spaces
for i in "${aMyArray[#]}"; do echo "$i"; done
readarray -- read lines from standard input
-t -- omit trailing newline character
aMyArray -- name of array to store file in
< <() -- execute command; redirect stdout into array
cat textfile -- file you want to store in variable
for i in "${aMyArray[#]}" -- for every element in aMyArray
"" -- needed to maintain spaces in elements
${ [#]} -- reference all elements in array
do echo "$i"; -- for every iteration of "$i" echo it
"" -- to maintain variable spaces
$i -- equals each element of the array aMyArray as it cycles through
done -- close for loop
OPTION 2:
In order to accommodate your larger file you could do this to help alleviate the work and speed up the processing.
#!/bin/bash
sSearchFile=textfile
sSearchStrings="1|2|3|space"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")
This will grep the file (faster) before it cycles it through the while command. Let me know how this works for you. Notice you can add multiple search strings to the $sSearchStrings variable.
OPTION 3:
and an all in one solution to have a text file with your search criteria and everything else combined...
#!/bin/bash
# identify file containing search strings
sSearchStrings="searchstrings.file"
while IFS= read -r string; do
# if $sSearchStrings empty read in strings
[[ -z $sSearchStrings ]] && sSearchStrings="${string}"
# if $sSearchStrings not empty read in $sSearchStrings "|" $string
[[ ! -z $sSearchStrings ]] && sSearchStrings="${sSearchStrings}|${string}"
# read search criteria in from file
done <"${sSearchStrings}"
# identify file to be searched
sSearchFile="text.file"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")
I am trying to remove newlines from a file. My file is like this (it contains backward slashes):
line1\|
line2\|
I am using the following script to remove newlines:
#!/bin/bash
INPUT="file1"
while read line
do
: echo -n $line
done < $INPUT
I get the following output:
line1|line2|
It removes the backslashes. How can I retain those backslashes?
The -r option to read prevents backslash processing of the input.
while read -r line
do
echo -n "$line"
done < $INPUT
But if you just want to remove all newlines from the input, the tr command would be better:
tr -d '\n' < $INPUT
Try sed 's/\n//' /path/to/file
I don't have a code example here since I'm not sure how to do this at all, but I have a file. A legal empty line is one that only contains the new-line tab. Spaces or tabs are illegal.
How do I check if a line is "legally empty"?
If it doesn't have any words (I can check this with wc -w), how do I check if it has no spaces or tabs either, just new-line?
So I've tried something like this:
while read line; do
if [[ "$line" =~ ^$ ]]; then
echo empty line
continue
fi
done < $1
But it's not working. If I put a " " in an otherwise empty line, it still considers it empty.
If you want the line numbers of those empty lines:
perl -lne 'print $. if(/^$/)' your_file
If you want to delete those lines without Perl:
grep . your_file >new_file
If you want to delete those empty line in place using Perl:
perl -i -lne 'print if(/./)' your_file
Terminology: a line that contains only white space is a blank line. A line that contains nothing (except for the newline terminator) is an empty line.
The read builtin strips off leading and trailing whitespace. So if it encounters a blank line, it sets its argument to an empty string, regardless of the amount of whitespace. To avoid this behavior and return the input line unmodified, set the field separator characters to nothing (by default, they are space, tab and newline): set the IFS variable to the empty string. See Why is while IFS= read used so often, instead of IFS=; while read..? for a more detailed explanation. While you're at it, pass the -r option to read, unless you want backslash-newline sequences to be a line continuation.
while IFS= read -r line; do
if [ -z "$line" ]; then
echo empty line
fi
done <"$1"
If you want to tell whether a line is blank:
while IFS= read -r line; do
case "$line" in
'') echo "empty line";;
*[![:space:]]*) echo "non-blank line";;
*) echo "non-empty blank line";;
esac
done <"$1"
You can use Bash regular expression matching if you prefer:
while IFS= read -r line; do
if [[ "$line" =~ ^$ ]]; then
echo "empty line"
elif [[ "$line" =~ ^[[:space:]]+$ ]]; then
echo "non-empty blank line"
else
echo "non-blank line"
fi
done <"$1"
These can be done with pattern matching too (using shell wildcards, which have a different syntax from common regular expressions):
while IFS= read -r line; do
if [[ "$line" == "" ]]; then
echo "empty line"
elif [[ "$line" != *[![:space:]]* ]]; then
echo "non-empty blank line"
else
echo "non-blank line"
fi
done <"$1"
If you merely want to look for empty lines in the file and aren't processing the lines in any other way, you can use grep:
if grep -qxF '' <"$1"; then
echo "$1 contains an empty line"
fi
If you're looking for blank lines that are not empty:
if grep -Ex '[[:space:]]+' <"$1"; then
echo "$1 contains a non-empty blank line"
fi
You can check for an empty line with the regex
^$
^ is the beginning of a line, $ is the end of a line, the above regex matches if there are no other characters.
You can now use that in e.g. sed
sed '/^$/d' input.txt
This would delete all empty lines from your input file.
This would remove empty lines from the file and display the file content on console. The file still remains unchanged.
If you want to remove the empty lines from the file (meaning, changing the file content), then run:
sed -i '/^$/d' input.txt
I am trying to write a script which reads a text file and saves each line to a string. I would also like the script to skip any lines which start with a hash symbol. Any suggestions?
You should not leave skipping lines to ksh. E.g. do this:
grep -v '^#' INPUTFILE | while IFS="" read line ; do echo $line ; done
And instead of the echo part do whatever you want.
Or if ksh does not support this syntax:
grep -v '^#' INPUTFILE > tmpfile
while IFS="" read line ; do echo $line ; done < tmpfile
rm tmpfile
while read -r line; do
[[ "$line" = *( )#* ]] && continue
# do something with "$line"
done < filename
look for "File Name Patterns" or "File Name Generation" in the ksh man page.