I am reading a file with lines like:
folder=abc
name=xyz
For some lines line I would like set a variable e.g name=xyz corresponding to the line I have read.
Cutting it down, with name=xyz and folder=abc, I have tried:
while read -r line; do
$line
echo $name
done < /etc/testfile.conf
This gives an error message ./test: line 4: folder=abc: command not found etc.
I have tried "$line" and $($line) and it is the same. Is it possible to do what I whant?
I have succeeded by doing:
while read -r line; do
if [[ "$line" == 'folder'* ]]; then
folder="$(echo "$line" | cut -d'=' -f 2)"
fi
if [[ "$line" == 'name'* ]]; then
name="$(echo "$line" | cut -d'=' -f 2)"
fi
done < /etc/testfile.conf
but this seems messy
for your sample, declare is the safest option:
while read -r line; do
declare "$line"
done
$ echo "$folder"
abc
$ echo "$name"
xyz
Direct approach, use eval.
Different approach, try with source or .:
$ echo "$line"
folder=abc
$ . <(echo "$line")
$ echo "$folder"
abc
But probably the good answer will be to tackle the problem in a different way.
You can clean up your approach a bit without resorting to eval.
while IFS="=" read -r name value; do
case $name in
folder) folder=$value ;;
name) name=$value ;;
esac
done < /etc/testfile.conf
why not only source de file ?
$ . infile ; echo "$name"
xyz
Related
I have got a question. How should I proceed and make this code print out and execute curl examples that I have on my external file?
How I want it to work is to match the pattern, get text between the patterns (without the pattern) and then execute it.
Is there way to do this?
Thanks for the help.
read -p "Enter a word: " instance
testfile=test.txt
case $instance in
loresipsum)
sed -n '/^loremipsum1/,${p;/^loremipsum2/q}' $testfile \
| while read -r line; do
makingcurlCall=$(eval "$line")
echo "makingcurlCall"
done < $testfile ;;
foobar)
sed -n '/^foobar1/,${p;/^foobar2/q}' $testfile \
| while read -r line; do
makingcurlCall=$(eval "$line")
echo "makingcurlCall"
done < $testfile ;;
*)
printf 'No match for "%s"\n' ":instance"
esac
Text file looks like this
loremipsum1
curl example1
curl example2
curl example3
loremipsum2
foobar1
curl foo
curl bar
curl foo
foobar2
You cannot have the while loop read from both the output of sed and directly from the file. Your current code is ignoring the output from sed and reading directly from the file. Perhaps refactor it like:
#!/bin/sh
instance=${1-loresipsum}
testfile=test.txt
case $instance in
loresipsum) sed -n '/^loremipsum1/,/^loremipsum2/p' "$testfile";;
foobar) sed -n '/^foobar1/,/^foobar2/p' "$testfile";;
*) echo "Error: no match" >&2;;
esac \
| sed -e 1d -e '$d' -e '/^\s*$/d' | while read -r line; do
# makingcurlCall=$(eval "$line")
echo "makingcurlCall: $line"
done
I'm trying to decode a file, which is mostly encoded with base64. What I want to do is to decode the following, while still maintaining the [_*_].
example.txt
wq9cXyjjg4QpXy/Crwo=
[_NOTBASE64ED_]
aGkgdGhlcmUK
[_CONSTANT_]
SGVsbG8gV29ybGQhCg==
Sometimes it'll be in this form
aGkgdGhlcmUK[_CONSTANT_]SGVsbG8gV29ybGQhCg==
Desired output
¯\_(ツ)_/¯
[_NOTBASE64ED_]
hi there
[_CONSTANT_]
Hello World!
hi there[_CONSTANT_]Hello World!
Error output
¯\_(ツ)_/¯
4��!:�#�H\�B�8ԓ��[��ܛBbase64: invalid input
What I've tried
base64 -di example.txt
base64 -d example.txt
base64 --wrap=0 -d -i example.txt
I tried to individually base64 the [_*_] using grep -o. Then find and
replacing them through a weird arrangement with arrays, but I couldn't
get it to work.
base64ing it all, then decoding. Results in double base64ed rows.
The file is significantly downsized!
Encoded using base64 --wrap=0, while loop, and if/else statement.
The [_*_] still need to be there after being decoded.
I am sure someone has a more clever solution than this. But try this
#! /bin/bash
MYTMP1=""
function printInlineB64()
{
local lines=($(echo $1 | sed -e 's/\[/\n[/g' -e 's/\]/]\n/g'))
OUTPUT=""
for line in "${lines[#]}"; do
MYTMP1=$(base64 -d <<< "$line" 2>/dev/null)
if [ "$?" != "0" ]; then
OUTPUT="${OUTPUT}${line}"
else
OUTPUT="${OUTPUT}${MYTMP1}"
fi;
done
echo "$OUTPUT"
}
MYTMP2=""
function printB64Line()
{
local line=$1
# not fully base64 line
if [[ ! "$line" =~ ^[A-Za-z0-9+/=]+$ ]]; then
printInlineB64 "$line"
return
fi;
# likely base64 line
MYTMP2=$(base64 -d <<< "$line" 2>/dev/null)
if [ "$?" != "0" ]; then
echo $line
else
echo $MYTMP2
fi;
}
FILE=$1
if [ -z "$FILE" ]; then
echo "Please give a file name in argument"
exit 1;
fi;
while read line; do
printB64Line "$line"
done < ${FILE}
and here is output
$ cat example.txt && echo "==========================" && ./base64.sh example.txt
wq9cXyjjg4QpXy/Crwo=
[_NOTBASE64ED_]
aGkgdGhlcmUK
[_CONSTANT_]
SGVsbG8gV29ybGQhCg==
==========================
¯\_(ツ)_/¯
[_NOTBASE64ED_]
hi there
[_CONSTANT_]
Hello World!
$ cat example2.txt && echo "==========================" && ./base64.sh example2.txt
aGkgdGhlcmUK[_CONSTANT_]SGVsbG8gV29ybGQhCg==
==========================
hi there[_CONSTANT_]Hello World!
You need a loop that reads each line and tests whether it's base64 or non-base64, and processes it appropriately.
while read -r line
do
case "$line" in
\[*\]) echo "$line" ;;
*) base64 -d <<< "$line" ;;
esac
done << example.txt
I would suggest using other languages other than sh but here is a solution using cut. This would handle the case where there are more than one [_constant_] in a line.
#!/bin/bash
function decode() {
local data=""
local line=$1
while [[ -n $line ]]; do
data=$data$(echo $line | cut -d[ -f1 | base64 -d)
const=$(echo $line | cut -d[ -sf2- | cut -d] -sf1)
[[ -n $const ]] && data=$data[$const]
line=$(echo $line | cut -d] -sf2-)
done
echo "$data"
}
while read -r line; do
decode $line
done < example.txt
If Perl is an option, you can say something like:
perl -MMIME::Base64 -lpe '$_ = join("", grep {/^\[/ || chomp($_ = decode_base64($_)), 1} split(/(?=\[)|(?<=\])/))' example.txt
The code below is equivalent to the above but is broken down into steps for the explanation purpose:
#!/bin/bash
perl -MMIME::Base64 -lpe '
#ary = split(/(?=\[)|(?<=\])/, $_);
foreach (#ary) {
if (! /^\[/) {
chomp($_ = decode_base64($_));
}
}
$_ = join("", #ary);
' example.txt
-MMIME::Base64 option loads the base64 codec module.
-lpe option makes Perl bahave like AWK to loop over input lines and implicitly handle newlines.
The regular expression (?=\[)|(?<=\]) matches the boundary between the base64 block and the maintaining block surrounded by [...].
The split function divides the line into blocks on the boundary and store them in an array.
Then loop over the array and decode the base64-encoded entry if found.
Finally merge the substring blocks into a line to print.
I've got this code which reads an example file of /etc/passwd:
#!/bin/bash
OLDIFS=$IFS
IFS=$'\n'
while read linea resto
do
echo $linea
echo $resto
if [[ $(echo $linea | cut -d: -f6 | egrep -c 'al-03-04') == 1 ]]
then
finger $(cut -d: -f1) 2> fich
if [[ $(egrep -c fich) == 1 ]]
then
echo $(echo $linea | cut -d: -f1). Inactive user
else
echo $(echo $linea | cut -d: -f1). Active user
fi
fi
done < <(cat fichpasswd)
IFS=$OLDIFS
and this is the example file of /etc/passwd:
jfer:x:5214:1007:Javier Lopez,,,:/home/al-03-04/jfer:/bin/bash
jperez:x:10912:1009:Juan Perez,,,:/home/al-03-04/jperez:/bin/bash
mfernan:x:10913:1009:Manuel Fernandez,,,:/home/al-02-03/mfernan:/bin/bash
The problem is that the while loop only reads the first line, ignoring the others. The script's output is:
jfer:x:5214:1007:Javier Lopez,,,:/home/al-03-04/jfer:/bin/bash
jfer. Active user
You could try something like :
#!/bin/bash
FILE="test.txt"
while IFS=":" read -a data; do
echo "${data[#]}"
if [[ $(echo ${data[5]}|egrep -c 'al-03-04') -eq 1 ]]; then
if [[ $(finger "${data[0]}" 2>&1) =~ "no such user" ]]; then
echo "${data[0]}. Inactive user"
else
echo "${data[0]}. Active user"
fi
fi
done < "$FILE"
Here's the output :
ineumann ~ $ cat test.txt
ineumann:x:5214:1007:Javier Lopez,,,:/home/al-03-04/jfer:/bin/bash
jperez:x:10912:1009:Juan Perez,,,:/home/al-03-04/jperez:/bin/bash
mfernan:x:10913:1009:Manuel Fernandez,,,:/home/al-02-03/mfernan:/bin/bash
ineumann ~ $ ./test.sh
ineumann x 5214 1007 Javier Lopez,,, /home/al-03-04/jfer /bin/bash
ineumann. Active user
jperez x 10912 1009 Juan Perez,,, /home/al-03-04/jperez /bin/bash
jperez. Inactive user
mfernan x 10913 1009 Manuel Fernandez,,, /home/al-02-03/mfernan /bin/bash
A few comments on your script :
No need to use cat to read your file in a loop.
finger $(cut -d: -f1) 2> fich : cut need an input. And no need to use a temporary file to catch the output of finger (moreover this is not thread safe).
No need to use cut in your script when you choose the right IFS to split a line in multiple parts. In your case, I think the smartest choice would be :.
You can change the IFS only inside the loop with the syntax while IFS=':' read; do ...; done. No need to re-assign IFS with OLDIFS.
You can also use the while IFS=':' read var1 var2 var3 trash; do ...; done syntax to avoid to use an array with read -a (but I'd prefer to use an array as I wrote in my version of your script).
I have a bash while read line block reading from a text file specified by $filename:
IFS=''
while read -r line
do
...
done < $filename
Instead of reading the whole file each time, I would like to supply different inputs in the redirect depending on the arguments supplied to the script.
Whole file: done < "$filename"
start at line x: done < <(tail -n +"$x" "$filename")
line x to line y: done < <(tail -n +"$x" "$filename" | head -n "$y")
start to line y: done < <(head -n "$y" "$filename")
How can I assign these inputs to a variable ahead of time to be read by the while loop?
My input file is ~4GB with some 58M lines (all with different lengths), and may grow or shrink from time to time. Reading https://unix.stackexchange.com/questions/47407/cat-line-x-to-line-y-on-a-huge-file it appears that tail | head is the fastest method to read from the middle of a file, so given the file size, I'm deliberately avoiding awk and sed for the most part.
Your data is too big to read in whole. The good news is that the contents of a process substitution is a shell script, so you can write:
while IFS= read -r line; do
...
done < <(
if [[ $x && $y ]]; then tail -n +"$x" "$filename" | head -n "$y"
elif [[ $x ]]; then tail -n +"$x" "$filename"
elif [[ $y ]]; then head -n "$y" "$filename"
else cat "$filename"
fi
)
One thing I don't like about process substitutions is that code follows the loop for which it is input. It would be nice if it appeared first. I think this will work, but is untested:
# set up file descriptor 3
exec 3< <(
if [[ $x && $y ]]; then tail -n +"$x" "$filename" | head -n "$y"
elif [[ $x ]]; then tail -n +"$x" "$filename"
elif [[ $y ]]; then head -n "$y" "$filename"
else cat "$filename"
fi
)
# iterate over lines read from fd 3
while IFS= read -u3 -r line; do
...
done
# close fd 3
exec 3<&-
I might handle all of these as part of the loop condition, with an explicitly maintained line counter.
start=10
end=30
i=0
while ((i <= end )) && IFS= read -r line; do
(( i++ >= start )) || continue
...
done < "$filename"
However, if you might skip a significant number of lines at the beginning, it might be more efficient to use sed
while IFS= read -r line; do
...
done < <(sed -n "$start,$stop p" "$filename")
or awk:
while IFS= read -r line; do
...
done < <(awk -v start "$start" -v end "$end" 'NR >= start && NR <= end' "$filename")
This then raises the question of how much of the body of the while loop can be moved into awk itself.
I have a list that looks like this:
sharename:shareX
comment:commentX
sharename:shareY
comment:commentY
sharename:shareZ
comment:commentZ
and so on...
And this is how I would like the list to look like:
shareX;commentX
shareY;commentY
shareZ;commentZ
How can I accomplish that in bash?
Pure Bash:
IFS=':'
while read a b; read c d; do # read 2 lines
echo -e "$b:$d"
done < "$infile"
one liner:
odd=0; for i in `cat list | cut -d":" -f2`; do if [ $odd -eq 0 ]; then echo -ne $i; odd=1; else echo $i; odd=0; fi; done
formatted:
odd=0;
for i in `cat list | cut -d":" -f2`;
do
if [ $odd -eq 0 ];
then
echo -ne $i";";
odd=1;
else
echo $i;
odd=0;
fi;
done
untested, the sed part may be wrong
paste -d ';' - - < filename | sed -r 's/(^|;)[^:]:/\1/g'
This might work for you:
sed '$!N;s/[^:]*:\([^\n]*\)\n[^:]*:/\1;/' file
shareX;commentX
shareY;commentY
shareZ;commentZ