I'm trying to decode a file, which is mostly encoded with base64. What I want to do is to decode the following, while still maintaining the [_*_].
example.txt
wq9cXyjjg4QpXy/Crwo=
[_NOTBASE64ED_]
aGkgdGhlcmUK
[_CONSTANT_]
SGVsbG8gV29ybGQhCg==
Sometimes it'll be in this form
aGkgdGhlcmUK[_CONSTANT_]SGVsbG8gV29ybGQhCg==
Desired output
¯\_(ツ)_/¯
[_NOTBASE64ED_]
hi there
[_CONSTANT_]
Hello World!
hi there[_CONSTANT_]Hello World!
Error output
¯\_(ツ)_/¯
4��!:�#�H\�B�8ԓ��[��ܛBbase64: invalid input
What I've tried
base64 -di example.txt
base64 -d example.txt
base64 --wrap=0 -d -i example.txt
I tried to individually base64 the [_*_] using grep -o. Then find and
replacing them through a weird arrangement with arrays, but I couldn't
get it to work.
base64ing it all, then decoding. Results in double base64ed rows.
The file is significantly downsized!
Encoded using base64 --wrap=0, while loop, and if/else statement.
The [_*_] still need to be there after being decoded.
I am sure someone has a more clever solution than this. But try this
#! /bin/bash
MYTMP1=""
function printInlineB64()
{
local lines=($(echo $1 | sed -e 's/\[/\n[/g' -e 's/\]/]\n/g'))
OUTPUT=""
for line in "${lines[#]}"; do
MYTMP1=$(base64 -d <<< "$line" 2>/dev/null)
if [ "$?" != "0" ]; then
OUTPUT="${OUTPUT}${line}"
else
OUTPUT="${OUTPUT}${MYTMP1}"
fi;
done
echo "$OUTPUT"
}
MYTMP2=""
function printB64Line()
{
local line=$1
# not fully base64 line
if [[ ! "$line" =~ ^[A-Za-z0-9+/=]+$ ]]; then
printInlineB64 "$line"
return
fi;
# likely base64 line
MYTMP2=$(base64 -d <<< "$line" 2>/dev/null)
if [ "$?" != "0" ]; then
echo $line
else
echo $MYTMP2
fi;
}
FILE=$1
if [ -z "$FILE" ]; then
echo "Please give a file name in argument"
exit 1;
fi;
while read line; do
printB64Line "$line"
done < ${FILE}
and here is output
$ cat example.txt && echo "==========================" && ./base64.sh example.txt
wq9cXyjjg4QpXy/Crwo=
[_NOTBASE64ED_]
aGkgdGhlcmUK
[_CONSTANT_]
SGVsbG8gV29ybGQhCg==
==========================
¯\_(ツ)_/¯
[_NOTBASE64ED_]
hi there
[_CONSTANT_]
Hello World!
$ cat example2.txt && echo "==========================" && ./base64.sh example2.txt
aGkgdGhlcmUK[_CONSTANT_]SGVsbG8gV29ybGQhCg==
==========================
hi there[_CONSTANT_]Hello World!
You need a loop that reads each line and tests whether it's base64 or non-base64, and processes it appropriately.
while read -r line
do
case "$line" in
\[*\]) echo "$line" ;;
*) base64 -d <<< "$line" ;;
esac
done << example.txt
I would suggest using other languages other than sh but here is a solution using cut. This would handle the case where there are more than one [_constant_] in a line.
#!/bin/bash
function decode() {
local data=""
local line=$1
while [[ -n $line ]]; do
data=$data$(echo $line | cut -d[ -f1 | base64 -d)
const=$(echo $line | cut -d[ -sf2- | cut -d] -sf1)
[[ -n $const ]] && data=$data[$const]
line=$(echo $line | cut -d] -sf2-)
done
echo "$data"
}
while read -r line; do
decode $line
done < example.txt
If Perl is an option, you can say something like:
perl -MMIME::Base64 -lpe '$_ = join("", grep {/^\[/ || chomp($_ = decode_base64($_)), 1} split(/(?=\[)|(?<=\])/))' example.txt
The code below is equivalent to the above but is broken down into steps for the explanation purpose:
#!/bin/bash
perl -MMIME::Base64 -lpe '
#ary = split(/(?=\[)|(?<=\])/, $_);
foreach (#ary) {
if (! /^\[/) {
chomp($_ = decode_base64($_));
}
}
$_ = join("", #ary);
' example.txt
-MMIME::Base64 option loads the base64 codec module.
-lpe option makes Perl bahave like AWK to loop over input lines and implicitly handle newlines.
The regular expression (?=\[)|(?<=\]) matches the boundary between the base64 block and the maintaining block surrounded by [...].
The split function divides the line into blocks on the boundary and store them in an array.
Then loop over the array and decode the base64-encoded entry if found.
Finally merge the substring blocks into a line to print.
I want to input multiple strings.
For example:
abc
xyz
pqr
and I want output like this (including quotes) in a file:
"abc","xyz","pqr"
I tried the following code, but it doesn't give the expected output.
NextEmail=","
until [ "a$NextEmail" = "a" ];do
echo "Enter next E-mail: "
read NextEmail
Emails="\"$Emails\",\"$NextEmail\""
done
echo -e $Emails
This seems to work:
#!/bin/bash
# via https://stackoverflow.com/questions/1527049/join-elements-of-an-array
function join_by { local IFS="$1"; shift; echo "$*"; }
emails=()
while read line
do
if [[ -z $line ]]; then break; fi
emails+=("$line")
done
join_by ',' "${emails[#]}"
$ bash vvuv.sh
my-email
another-email
third-email
my-email,another-email,third-email
$
With sed and paste:
sed 's/.*/"&"/' infile | paste -sd,
The sed command puts "" around each line; paste does serial pasting (-s) and uses , as the delimiter (-d,).
If input is from standard input (and not a file), you can just remove the input filename (infile) from the command; to store in a file, add a redirection at the end (> outfile).
If you can withstand a trailing comma, then printf can convert an array, with no loop required...
$ readarray -t a < <(printf 'abc\nxyx\npqr\n' )
$ declare -p a
declare -a a=([0]="abc" [1]="xyx" [2]="pqr")
$ printf '"%s",' "${a[#]}"; echo
"abc","xyx","pqr",
(To be fair, there's a loop running inside bash, to step through the array, but it's written in C, not bash. :) )
If you wanted, you could replace the final line with:
$ printf -v s '"%s",' "${a[#]}"
$ s="${s%,}"
$ echo "$s"
"abc","xyx","pqr"
This uses printf -v to store the imploded text into a variable, $s, which you can then strip the trailing comma off using Parameter Expansion.
I have a file that contains 10 lines with this sort of content:
aaaa,bbb,132,a.g.n.
I wanna walk throw every line, char by char and put the data before the " , " is met in an output file.
if [ $# -eq 2 ] && [ -f $1 ]
then
echo "Read nr of fields to be saved or nr of commas."
read n
nrLines=$(wc -l < $1)
while $nrLines!="1" read -r line || [[ -n "$line" ]]; do
do
for (( i=1; i<=$n; ++i ))
do
while [ read -r -n1 temp ]
do
if [ temp != "," ]
then
echo $temp > $(result$i)
else
fi
done
paste -d"\n" $2 $(result$i)
done
nrLines=$($nrLines-1)
done
else
echo "File not found!"
fi
}
In parameter $2 I have an empty file in which I will store the data from file $1 after I extract it without the " , " and add a couple of comments.
Example:
My input_file contains:
a.b.c.d,aabb,comp,dddd
My output_file is empty.
I call my script: ./script.sh input_file output_file
After execution the output_file contains:
First line info: a.b.c.d
Second line info: aabb
Third line info: comp
(yes, without the 4th line info)
You can do what you want very simply with parameter-expansion and substring-removal using bash alone. For example, take an example file:
$ cat dat/10lines.txt
aaaa,bbb,132,a.g.n.
aaaa,bbb,133,a.g.n.
aaaa,bbb,134,a.g.n.
aaaa,bbb,135,a.g.n.
aaaa,bbb,136,a.g.n.
aaaa,bbb,137,a.g.n.
aaaa,bbb,138,a.g.n.
aaaa,bbb,139,a.g.n.
aaaa,bbb,140,a.g.n.
aaaa,bbb,141,a.g.n.
A simple one-liner using native bash string handling could simply be the following and give the following results:
$ while read -r line; do echo ${line%,*}; done <dat/10lines.txt
aaaa,bbb,132
aaaa,bbb,133
aaaa,bbb,134
aaaa,bbb,135
aaaa,bbb,136
aaaa,bbb,137
aaaa,bbb,138
aaaa,bbb,139
aaaa,bbb,140
aaaa,bbb,141
Paremeter expansion w/substring removal works as follows:
var=aaaa,bbb,132,a.g.n.
Beginning at the left and removing up to, and including, the first ',' is:
${var#*,} # bbb,132,a.g.n.
Beginning at the left and removing up to, and including, the last ',' is:
${var##*,} # a.g.n.
Beginning at the right and removing up to, and including, the first ',' is:
${var%,*} # aaaa,bbb,132
Beginning at the left and removing up to, and including, the last ',' is:
${var%%,*} # aaaa
Note: the text to remove above is represented with a wildcard '*', but wildcard use is not required. It can be any allowable text. For example, to only remove ,a.g.n where the preceding number is 136, you can do the following:
${var%,136*},136 # aaaa,bbb,136 (all others unchanged)
To print 2016 th line from a file named file.txt u have to run a command like this-
sed -n '2016p' < file.txt
More-
sed -n '2p' < file.txt
will print 2nd line
sed -n '2011p' < file.txt
2011th line
sed -n '10,33p' < file.txt
line 10 up to line 33
sed -n '1p;3p' < file.txt
1st and 3th line
and so on...
For more detail, please have a look in this tutorial and this answer.
In native bash the following should do what you want, assuming you replace the contents of your script.sh with the below:
#!/bin/bash
IN_FILE=${1}
OUT_FILE=${2}
IFS=\,
while read line; do
set -- ${line}
for ((i=1; i<=${#}; i++)); do
((${i}==4)) && continue
((n+=1))
printf '%s\n' "Line ${n} info: ${!i}"
done
done < ${IN_FILE} > ${OUT_FILE}
This will not print the 4th field of each line within the input file, on a new line in the output file (I assume this is your requirement as per your comment?).
[wspace#wspace sandbox]$ awk -F"," 'BEGIN{OFS="\n"}{for(i=1; i<=NF-1; i++){print "line Info: "$i}}' data.txt
line Info: a.b.c.d
line Info: aabb
line Info: comp
This little snippet can ignore the last field.
updated:
#!/usr/bin/env bash
if [ ! -f "$1" -o $# -ne 2 ];then
echo "Usage: $(basename $0) input_file out_file"
exit 127
fi
input_file=$1
output_file=$2
: > $output_file
if [ "$(wc -l < $1)" -ne 0 ];then
while true
do
read -r -n1 char
if [ "$char" == "" ];then
break
elif [ $char != "," ];then
temp=$temp$char
else
echo "line info: $temp" >> $output_file
temp=""
fi
done < $input_file
else
echo "file $1 is empty"
fi
Maybe this is what you want
Did you try
sed "s|,|\n|g" $1 | head -n -1 > $2
I assume that only the last word would not have a comma on its right.
Try this (tested with you sample line) :
#!/bin/bash
# script.sh
echo "Number of fields to save ?"
read nf
while IFS=$',' read -r -a arr; do
newarr=${arr[#]:0:${nf}}
done < "$1"
for i in ${newarr[#]};do
printf "%s\n" $i
done > "$2"
Execute script with :
$ ./script.sh inputfile outputfile
Number of fields ?
3
$ cat outputfile
a.b.c.d
aabb
comp
All words separated with commas are stored into an array $arr
A tmp array $newarr removes last $n element ($n get the read command).
It loops over new array and prints result in $2, the outputfile.
This question already has answers here:
Bash tool to get nth line from a file
(22 answers)
Closed 7 years ago.
This is my script. It print every row in the file with the number of row.
Next i want to read which row user choosed and save it to some variable.
I=1
for ROW in $(cat file.txt)
do
echo "$I $ROW"
I=`expr $I + 1`
done
read var
awk 'FNR = $var {print}' file.txt
Then i want to to print / save the chosen row into the file.
How can I do this ?
when i echo $var it shows me properly the number. But when i'm trying to use this variable in awk, it print every line.
How to read the 'var' line from file?
And moreover, how to save this line in other variable?
Example file.txt
1 line1
2 line2
3 line3
4 line4
when i tap 3 i want to read third line from file.
Try this:
cat -n file.txt; read var; line="$(sed -n ${var}p file)"; echo "$line"
With more focus on Dryingsoussage's version:
#!/bin/bash
file="file.txt"
declare -i counter=0 # set integer attribute
var=0
while read -r line; do
counter=counter+1
printf "%d %s\n" "$counter" "$line"
done < "$file"
# check for number and greater-than 0 and less-than-or-equal $counter
until [[ $var =~ ^[0-9]+$ ]] && [[ $var -gt 0 ]] && [[ $var -le $counter ]]; do
read -p "Enter line number:" var
done
awk -v var="$var" 'FNR==var {print}' "$file"
You cannot use $varname inside ' ' they will not be resolved.
look at this other post it should help you:
How to use shell variables in an awk script
cat -n file.txt
read var
row="$(awk -v tgt="$var" 'NR==tgt{print;exit}' file.txt)"
First: You cannot use $var in a single quotes, as echo '$var' would be plain $var, no its value.
Second: You used = (assignment) operator instead of == (equality) operator.
Third: You don't have to write { print } if you want the line to be printed. You can write nothing instead.
Fourth: As was explained in the deleted comment below - do not allow bash expanding the variables in the awk script code, as it can lead to code injection.
So conclusion is:
awk -v var="$var" 'FNR == var' file.txt
should do what you want.
I have this block of code:
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
if [ "$line" != "" ]; then
echo -e "$lanIP\t$line" >> /tmp/ipList;
fi
done < "/tmp/includeList"
I know this must be really simple. But I have another list (/tmp/excludeList). I only want to echo the line within my while loop if the line ins't found in my excludeList. How do I do that. Is there some awk statement or something?
You can do this with grep alone:
$ cat file
blue
green
red
yellow
pink
$ cat exclude
green
pink
$ grep -vx -f exclude file
blue
red
yellow
The -v flag tells grep to only output the lines in file that are not found in exclude and the -x flags forces whole line matching.
use grep
while IFS=$'\n' read -r line || [[ -n "$line" ]]; do
if [[ -n ${line} ]] \
&& ! grep -xF "$line" excludefile &>/dev/null; then
echo -e "$lanIP\t$line" >> /tmp/ipList;
fi
done < "/tmp/includeList"
the -n $line means if $line is not empty
the grep returns true if $line is found in exclude file which is inverted by the ! so returns true if the line is not found.
-x means line matched so nothing else can appear on the line
-F means fixed string so if any metacharacters end up in $line they'll be matched literally.
Hope this helps
With awk:
awk -v ip=$lanIP -v OFS="\t" '
NR==FNR {exclude[$0]=1; next}
/[^[:space:]]/ && !($0 in exclude) {print ip, $0}
' /tmp/excludeList /tmp/includeList > /tmpipList
This reads the exclude list info an array (as the array keys) -- the NR==FNR condition is true while awk is reading the first file from the arguments. Then, while reading the include file, if the current line contains a non-space character and it does not exist in the exclude array, print it.
The equivalent with grep:
grep -vxF -f /tmp/excludeList /tmp/includeList | while IFS= read -r line; do
[[ -n "$line" ]] && printf "%s\t%s\n" "$ipList" "$line"
done > /tmp/ipList