I have a properties file and wanted to read line by line and split each line into a array.
I referenced Split string into an array in Bash and the awk statement looks complicated. In the same time I wanted to follow suggestion provided in the link.
When I try the command in bash shell:
export $line1="table1;/users/user_name/dir1/dir2;/users/user_name/dirA/dirB"
readarray -td; a <<<"$line1,"; unset 'a[-1]'; declare -p a;
throws error.
bash: readarray: -d: invalid option
readarray: usage: readarray [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array]
Couldn't under the below awk statements:
array=()
while read -r -d $'\0' each; do # use a NUL terminated field separator
array+=("$each")
done < <(printf "%s" "$str" | awk '{ gsub(/,[ ]+|$/,"\0"); print }')
declare -p array
to translate to my requirement.
This is how I read the file:
while read record_line; do
if [ ! -z "$record_line" -a "$record_line" != " " ]; then
readarray -td; a <<<"$record_line,"; unset 'a[-1]'; declare -p a;
fi
done<${PPROPERTIES_FILE}
Any help. Thanks.
Consider the following
#!/usr/bin/env ksh
line=0
while IFS=';' read -rA pieces; do
line=$(( line + 1 ))
unset "pieces[0]" # ignore first
echo "Found pieces on line $line:"
printf ' - %s\n' "${all_pieces[#]}"
done <in.txt
Note that unlike in bash, reading to an array in ksh is done with -A, not -a.
Related
I am in a situation similar to this one and having difficulties implementing this kind of solution for my situation.
I have file.tsv formatted as follows:
x y
dog woof
CAT meow
loud_goose honk-honk
duck quack
with a fixed number of columns (but variable rows) and I need to loop those pairs of values, all but the first one, in a script like the following (pseudocode)
for elements in list; do
./script1 elements[1] elements[2]
./script2 elements[1] elements[2]
done
so that script* can take the arguments from the pair and run with it.
Is there a way to do it in Bash?
I was thinking I could do something like this:
list1={`awk 'NR > 1{print $1}' file.tsv`}
list2={`awk 'NR > 1{print $2}' file.tsv`}
and then to call them in the loop based on their position, but I am not sure on how.
Thanks!
Shell tables are not multi-dimensional so table element cannot store two arguments for your scripts. However since you are processing lines from file.tsv, you can iterate on each line, reading both elements at once like this:
#!/usr/bin/env sh
# Populate tab with a tab character
tab="$(printf '\t')"
# Since printf's sub-shell added a trailing newline, remove it
tab="${tab%?}"
{
# Read first line in dummy variable _ to skip header
read -r _
# Iterate reading tab delimited x and y from each line
while IFS="$tab" read -r x y || [ -n "$x" ]; do
./script1 "$x" "$y"
./script2 "$x" "$y"
done
} < file.tsv # from this file
You could try just a while + read loop with the -a flag and IFS.
#!/usr/bin/env bash
while IFS=$' \t' read -ra line; do
echo ./script1 "${line[0]}" "${line[1]}"
echo ./script2 "${line[0]}" "${line[1]}"
done < <(tail -n +2 file.tsv)
Or without the tail
#!/usr/bin/env bash
skip=0 start=-1
while IFS=$' \t' read -ra line; do
if ((start++ >= skip)); then
echo ./script1 "${line[0]}" "${line[1]}"
echo ./script2 "${line[0]}" "${line[1]}"
fi
done < file.tsv
Remove the echo's if you're satisfied with the output.
In bash script, how can I read the file line by line and assign to the variable with delimiter?
example.txt file contents:
string1
string2
string3
string4
Expected output:
string1,string2,string3,string4
Thanks in advance
Apparently my answer below leaves a comma at the end of the line. A quick workaround is to use the following builtin in Unix:
paste -sd, example.txt
Where you use the paste program to concatenate all the lines into one and then add the string delimiter ','
Using the builtin commands in unix:
tr '\n' ',' < example.txt
This can be broken down as truncating all Newline widcards and inserting a comma delimiter instead.
Other possible ways, just for fun:
mapfile -t a < example.txt
(IFS=,; echo "${a[*]}")
mapfile -t a < example.txt
foo=$(printf '%s' "${a[#]/%/,}")
echo "${foo%,}"
foo=$(<example.txt)
echo "${foo//$'\n'/,}"
{
IFS= read -r foo
while IFS= read -r line; do
foo+=,$line
done
} < example.txt
echo "$foo"
sed ':a;N;$!ba;s/\n/,/g' example.txt
It should work:
#!/bin/bash
output=''
while IFS='' read -r line || [[ -n "$line" ]]; do
output=$output:",$line"
done < "$1"
echo $output
Give the file as argument
Using bash shell:
I am trying to read a file line by line.
and every line contains two meaning full file names delimited by "``"
file:1 image_config.txt
bbbbb.mp4``thumb/hashdata.gif
bbbbb.mp4``thumb/hashdata2.gif
Shell Script
#!/bin/bash
filename="image_config.txt"
while IFS='' read -r line || [[ -n "$line" ]]; do
IFS='``' read -r -a array <<< "$line"
if [ "$line" = "" ]; then
echo lineempty
else
file=${array[0]}
hash=${array[2]}
echo $file$hash;
output=$(ffmpeg -v warning -ss 2 -t 0.8 -i $file -vf scale=200:-1 -gifflags +transdiff -y $hash);
echo $output;
# echo ${array[0]}${array[1]}${array[2]}
fi;
done < "$filename"
first time executed successfully but when loop executes second time.
variable file lost bbbbb from bbbbb.mp4
and following output comes out
Output :
user#domain [~/public_html/Videos]$ sh imager.sh
bbbbb.mp4thumb/hashdata.gif
.mp4thumb/hashdata2.gif
.mp4: No such file or directory
lineempty
Please check out Bash FAQ 89 - I'm using a loop which runs once per line of input but it only seems to run once; everything after the first line is ignored? which seems to be helpful in your case.
Aside:
There is no point in using the same character twice in IFS.
IFS=\`
Is enough.
Check out this:
var='abc``def'
IFS=\`\` read -ra arr <<< "$var"
printf '<%s>\n' "${arr[#]}"
Output:
<abc>
<>
<def>
As you can see, arr[0] is abc, arr[1] is empty and arr[2] is def, and not arr[0] is abc and arr[1] is def as one might expect.
Taken from the IFS wiki of Greycat and Lhunath Bash Guide :
The IFS variable is used in shells (Bourne, POSIX, ksh, bash) as the input field separator (or internal field separator). Essentially, it is a string of special characters which are to be treated as delimiters between words/fields when splitting a line of input.
Here is how you could do differently, avoiding a read in the read:
#!/bin/bash
filename="image_config.txt"
while IFS='' read -r line || [[ -n "$line" ]]; do
if [ "$line" = "" ]; then
echo lineempty
else
file=$( echo ${line} | awk -F \` ' { print $1 } ' )
hash=$( echo ${line} | awk -F \` ' { print $3 } ' )
echo $file$hash;
output=$(ffmpeg -v warning -ss 2 -t 0.8 -i $file -vf scale=200:-1 -gifflags +transdiff -y $hash);
echo $output;
fi;
done < "$filename"
I have a large config file that I use to define variables for a script to pull from it, each defined on a single line. It looks something like this:
var val
foo bar
foo1 bar1
foo2 bar2
I have gathered a list of out of date variables that I want to remove from the list. I could go through it manually, but I would like to do it with a script, which would be at least more stimulating. The file that contains the vlaues may contain multiple instances. The idea is to find the value, and if it's found, remove the entire line.
Does anyone know if this is possible? I know sed does this but I do not know how to make it use a file input.
#!/bin/bash
shopt -s extglob
REMOVE=(foo1 foo2)
IFS='|' eval 'PATTERN="#(${REMOVE[*]})"'
while read -r LINE; do
read A B <<< "$LINE"
[[ $A != $PATTERN ]] && echo "$LINE"
done < input_file.txt > output_file.txt
Or (Use with a copy first)
#!/bin/bash
shopt -s extglob
FILE=$1 REMOVE=("${#:2}")
IFS='|' eval 'PATTERN="#(${REMOVE[*]})"'
SAVE=()
while read -r LINE; do
read A B <<< "$LINE"
[[ $A != $PATTERN ]] && SAVE+=("$LINE")
done < "$FILE"
printf '%s\n' "${SAVE[#]}" > "$FILE"
Running with
bash script.sh your_config_file pattern1 pattern2 ...
Or
#!/bin/bash
shopt -s extglob
FILE=$1 PATTERNS_FILE=$2
readarray -t REMOVE < "$PATTERNS_FILE"
IFS='|' eval 'PATTERN="#(${REMOVE[*]})"'
SAVE=()
while read -r LINE; do
read A B <<< "$LINE"
[[ $A != $PATTERN ]] && SAVE+=("$LINE")
done < "$FILE"
printf '%s\n' "${SAVE[#]}" > "$FILE"
Running with
bash script.sh your_config_file patterns_file
Here's one with sed. Add words to the array. Then use
./script target_filename
(assuming you put the following in a file called script). (Not very efficient). I think it might be more efficient if we concat the words and put it in the regex like bbonev did
#!/bin/bash
declare -a array=("foo1" "foo2")
for i in "${array[#]}";
do
sed -i "/^${i}\s.*/d" $1
done
It's actually even simpler using file input
If you have a word file
word1
word2
word3
.....
then the following will do the job
#!/bin/bash
while read i;
do
sed -i "/^${i}\s.*/d" $2
done <$1
usage:
./script wordlist target_file
I am trying to write a bash script(script.sh) to search and replace some variables in input.sh file. But I need to modify only the variables which are present in variable_list file and leave others as it is.
variable_list
${user}
${dbname}
input.sh
username=${user}
password=${password}
dbname=${dbname}
Expected output file
username=oracle
password=${password} > This line won't be changed as this variable(${password}) is not in variable_list file
dbname=oracle
Following is the script I am trying to use but I am not able to find the correct sed expression
script.sh
export user=oracle
export password=oracle123
export dbname=oracle
variable='variable_list'
while read line ;
do
if [[ -n $line ]]
then
sed -i 's/$line/$line/g' input.sh > output.sh
fi
done < "$variable"
This could work:
#!/bin/bash
export user=oracle
export password=oracle123
export dbname=oracle
variable='variable_list'
while read line ;
do
if [[ -n $line ]]
then
exp=$(sed -e 's/\$/\\&/g' <<< "$line")
var=$(sed -e 's/\${\([^}]\+\)}/\1/' <<< "$line")
sed -i "s/$exp/${!var}/g" input.sh
fi
done < "$variable"
The first sed expression escapes the $ which is a regex metacharacter. The second extracts just the variable name, then we use indirection to get the value in our current shell and use it in the sed expression.
Edit
Rather than rewriting the file so many times, it's probably more efficient to do it like this, building the arguments list for sed:
#!/bin/bash
export user=oracle
export password=oracle123
export dbname=oracle
while read var
do
exp=$(sed -e 's/\$/\\&/g' <<< "$var")
var=$(sed -e 's/\${\([^}]\+\)}/\1/' <<< "$var")
args+=("-e s/$exp/${!var}/g")
done < "variable_list"
sed "${args[#]}" input.sh > output.sh
user=oracle
password=oracle123
dbname=oracle
variable_list=( '${user}' '${dbname}' )
while IFS="=$IFS" read variable value; do
for subst_var in "${variable_list[#]}"; do
if [[ $subst_var = $value ]]; then
eval "value=$subst_var"
break
fi
done
printf "%s=%s\n" "$variable" "$value"
done < input.sh > output.sh
Here is a script.sh that works:
#!/bin/bash
user=oracle
password=oracle123
dbname=oracle
variable='variable_list'
text=$(cat input.sh)
while read line
do
value=$(eval echo $line)
text=$(sed "s/$line/$value/g" <<< "$text")
done < "$variable"
echo "$text" > output.sh
Note that your original version contains single quotes around the sed string, which doesn't insert the value of $line. It is trying to look for the literal line after the end of the line $ (which will never find anything).
Since you are looking for the value of the variable in $line, you need to do an eval to get this.
Also, since there are multiple variables you are looping over, the intermediate text variable stores the result as it loops.
The export keyword is also unnecessary in this script, unless it is being used in some sub-process not shown.
TXR solution. Build a filter dynamically. The filter is implemented internally as a trie data structure, which gives us a lex-like state machine which matches the entire dictionary at once as the input is scanned. For simplicity, we include the ${ and } as part of the variable name.
#(bind vars (("${user}" "oracle")
("${dbname}" "oracle")
("${password}" "letme1n")))
#(next "variable_list")
#(collect)
#entries
#(end)
#(deffilter subst . #(mapcar (op list #1 (second [find vars #1 equal first]))
entries))
#(next "input.sh")
#(collect)
#line
# (output :filter subst)
#line
# (end)
#(end)
Run:
$ txr subst.txr
username=oracle
password=${password}
dbname=oracle
input.sh: (as given)
username=${user}
password=${password}
dbname=${dbname}
variable_list: (as given)
${user}
${dbname}