Esteemed colleagues,
I have written a small code where i'm storing some command output into two different variables and aspiring those two values to be printed under into different columns called "PreferredList IP's" & "DefaultServerList IP's".
there are Variables PLIST & DLIST,So, when i'm running the script i only see output under the first column and unable to get the data under second column.
This looks weird to me, i don't know where i'm doing mistake..please do correct me..
#!/bin/sh
set -f # to prevent filename expansion
printf "=%.0s" $(seq 1 50)
printf "\n"
printf "%-30s : %10s\n" "PreferredList IP's" "DefaultServerList IP's" # print header
printf "=%.0s" $(seq 1 50) # print separator
printf "\n" # print newline
PLIST="$(ldapsearch -h mylap -x -b "ou=profile,o=cadence.com" "cn=*" preferredserverlist -LLL | awk '/preferredserverlist: / {print $2}')"
DLIST="$(ldapsearch -h myldap -x -b "ou=profile,o=cadence.com" "cn=*" defaultserverlist -LLL | awk '/defaultserverlist: / { print $2 }')"
printf "%-30s : %10s\n" "$PLIST" "$DLIST"
RESULT: while using debug mode, I saw the problem is both the varibale output coming into first column.
======================================================
PreferredList IP's : DefaultServerList IP's
========================================================
123.18.8.15
123.18.8.16
192.10.167.9
192.10.167.8
123.18.8.16
10.218.88.38
Below is the ldapsearch command output sample:
dn: cn=india, ou=profile, o=cadence.com
preferredServerList: 123.18.8.15 123.18.8.16
defaultServerList: 123.18.8.16 123.18.8.15
dn: cn=japan, ou=profile, o=cadence.com
preferredServerList: 192.10.167.9 192.10.167.8
defaultServerList: 123.18.8.16 10.218.88.38
$ ldapsearch -h myldap -x -b "ou=profile,o=cadence.com" "cn=*" preferredserverlist -LLL | awk '/preferredserverlist: / {print $2}' | head -2
123.18.8.15
123.18.8.16
$ ldapsearch -h myldap -x -b "ou=profile,o=cadence.com" "cn=*" defaultserverlist -LLL | awk '/defaultserverlist: / { print $2 }' | head -2
123.18.8.16
10.218.88.38
It seems like the real problem you have is formatting your columns.
You have two list of IPs stored in PLIST and DLIST as strings, separated by newlines. When you type
printf "%-30s : %10s\n" "$PLIST" "$DLIST"
It will not automatically format those into columns for you.
You really need to change the way you're parsing your LDAP results. /bin/sh is really not inherently suited to this kind of output formatting.
If you have the option of using bash (version > 4), use mapfile and restructure your program like this:
#!/bin/bash
set -f # to prevent filename expansion
# Store the output of the ldapsearches in arrays using mapfile.
mapfile -t PLIST < <(ldapsearch -h mylap -x -b "ou=profile,o=cadence.com" "cn=*" preferredserverlist -LLL | awk '/preferredserverlist: / {print $2}')
mapfile -t DLIST < <(ldapsearch -h myldap -x -b "ou=profile,o=cadence.com" "cn=*" defaultserverlist -LLL | awk '/defaultserverlist: / { print $2 }')
# Count the number of elements in each array.
count_x=${#PLIST[#]}
count_y=${#DLIST[#]}
# Print the count for debugging.
echo $count_x
echo $count_y
# Find out which of the two arrays is larger in size, assuming that's a possibility, pick the length of the bigger one.
if [[ $count_x -lt $count_y ]]
then
count=$count_y
else
count=${count_x}
fi
printf "=%.0s" $(seq 1 50)
printf "\n"
printf "%-30s : %10s\n" "PreferredList IP's" "DefaultServerList IP's" # print header
printf "=%.0s" $(seq 1 50) # print separator
printf "\n" # print newline
# Use an index 0 < i <= count, to loop over the arrays simultaneously.
for i in $(seq $count);
do
printf "%-30s : %10s\n" "${PLIST[i-1]}" "${DLIST[i-1]}"
done
This uses bash's mapfile to store the output of the ldap search commands in an indexed array and prints it out in a formatted column.
As a test, I wrote this out and replaced your ldap commands with mock seq calls to generate numbers. Here's a sample run.
Related
I have a large dataset files with two columns like
AS জীৱবিজ্ঞানবিভাগ
AS চেতনাদাস
AS বৈকল্পিক
and I want to run my command on the second column, store the result and get the output with the same column formatting:
AS jibvigyanvibhag
AS chetanadas
AS baikalpik
where my command is this pipe:
echo "$0" | indictrans -s asm -t eng --ml --build-lookup
So I'm doing like
awk -v OFS="\t" '{ print "echo "$2" | indictrans -s asm -t eng --ml --build-lookup" | "/bin/sh"}' in.txt > out.txt
but this will not preserve the columns, it just prints out the first column like this
jibvigyanvibhag
chetanadas
baikalpik
My solution was the following
awk -v OFS="\t" '{ "echo "$2" | indictrans -s asm -t eng --ml --build-lookup" | getline RES; print $1,$2,RES}' in.txt > out.txt
that will print out
AS জীৱবিজ্ঞানবিভাগ jibvigyanvibhag
AS চেতনাদাস chetanadas
AS বৈকল্পিক baikalpik
Now I want to put parametrize the command, but the escape looks odd here:
"echo "$0" | indictrans -s $SOURCE -t $TARGET --ml --build-lookup"
and it does not work. How to correctly exec this command and escape the parameters?
[UPDATE]
This is a partial solution I came out inspired by the suggested one
#!/bin/bash
SOURCE=asm
TARGET=eng
IN=$2
OUT=$3
awk -v OFS="\t" '{
CMD = "echo "$2" | indictrans -s asm -t eng --ml --build-lookup"
CMD | getline RES
print $1,RES
close(CMD)
}' $IN > $OUT
I still cannot get rid of the variables, it seems that I cannot define with -v as usual like
awk -v OFS="\t" -v source=$SOURCE -v target=$TARGET '{
CMD = "echo "$2" | indictrans -s source -t target --ml --build-lookup"
...
NOTES.
The indictrans process handles the stdin and writes to stdout in this way:
for line in ifp:
tline = trn.convert(line)
ofp.write(tline)
# close files
ifp.close()
ofp.close()
where
ifp = codecs.getreader('utf8')(sys.stdin)
ofp = codecs.getwriter('utf8')(sys.stdout)
so it takes one line from stdin, processes the data with some library trn.convert and writes the results to stdout without any parallelism.
For this reason (lack of parallelism in terms of multiline input) the performances are bound by the size of the dataset (number of rows).
An example input two column dataset (1K rows) is available here. An example sample is
KN ಐಕ್ಯತೆ ಕ್ಷೇಮಾಭಿವೃದ್ಧಿ ಸಂಸ್ಥೆ ವಿಜಯಪುರ
KN ಹೊರಗಿನ ಸಂಪರ್ಕಗಳು
KN ಮಕ್ಕಳ ಸಾಹಿತ್ಯ ಮತ್ತು ಸಾಂಸ್ಖ್ರುತಿಕ ಕ್ಷೇತ್ರದಲ್ಲಿ ಸೇವೆ ಸಲ್ಲಿಸುತ್ತಿರುವ ಸಂಸ್ಠೆ ಮಕ್ಕಳ ಲೋಕ
while the example script based on the last accepted answer is here
Don't invoke shells with awk. The shell itself avoids treating data as if it were code unless explicitly instructed to do otherwise -- but when you use system() or popen(), as the awk code is doing here, everything passed as an argument is parsed in a context where data is able to escape its quoting and be treated as code.
Simple approach: One indictrans per line
If you need a separate copy of indictrans for each line to be executed, use:
while read -r col1 rest; do
printf '%s\t%s\n' "$col1" "$(indictrans -s asm -t eng --ml --build-lookup <<<"$rest")"
done <in.txt >out.txt
Fast Approach: One indictrans processing all lines
If indictrans generates one line of output per line of input, you can do even better, by pasting together one stream with all the first columns and a second string with the translations of the remainder of the lines, thus requiring only one copy of indictrans to be run:
#!/usr/bin/env bash
# ^^^^- not compatible with /bin/sh
paste <(<in.txt awk '{print $1}') \
<(<in.txt sed -E 's/^[^[:space:]]*[[:space:]]//' \
| indictrans -s asm -t eng --ml --build-lookup) \
>out.txt
You can pipe column 2 to your command and change it with command's output like below in awk.
{
cmd = "echo "$2" | indictrans -s asm -t eng --ml --build-lookup"
cmd | getline $2
close(cmd)
} 1
If SOURCE and TARGET are awk variables
{
cmd = "echo "$0" | indictrans -s "SOURCE" -t "TARGET" --ml --build-lookup"
cmd
close(cmd)
}
In a shell script I need to assign the output of few values to different varialbes, need help please.
cat file1.txt
uid: user1
cn: User One
employeenumber: 1234567
absJobAction: HIRED
I need to assign the value of each attribute to different variables so that I can call them them in script. For example uid should be assigned to a new variable name current_uid and when $current_uid is called it should give user1 and so forth for all other attributes.
And if the output does not contain any of the attributes then that attribute value should be considered as "NULL". Example if the output does not have absJobAction then the value of $absJobAction should be "NULL"
This is what I did with my array
#!/bin/bash
IFS=$'\n'
array=($(cat /tmp/file1.txt | egrep -i '^uid:|^cn:|^employeenumber|^absJobAction'))
current_uid=`echo ${array[0]} | grep -w uid | awk -F ': ' '{print $2}'`
current_cn=`echo ${array[1]} | grep -w cn | awk -F ': ' '{print $2}'`
current_employeenumber=`echo ${array[2]} | grep -w employeenumber | awk -F ': ' '{print $2}'`
current_absJobAction=`echo ${array[3]} | grep -w absJobAction | awk -F ': ' '{print $2}'`
echo $current_uid
echo $current_cn
echo $current_employeenumber
echo $current_absJobAction
Output from sh /tmp/testscript.sh follows:
user1
User One
1234567
HIRED
#!/usr/bin/env bash
# assuming bash 4.0 or newer: create an associative array
declare -A vars=( )
while IFS= read -r line; do ## See http://mywiki.wooledge.org/BashFAQ/001
if [[ $line = *": "* ]]; then ## skip lines not containing ": "
key=${line%%": "*} ## strip everything after ": " for key
value=${line#*": "} ## strip everything before ": " for value
vars[$key]=$value
else
printf 'Skipping unrecognized line: <%s>\n' "$line" >&2
fi
done <file1.txt # or < <(ldapsearch ...)
# print all variables read, just to demonstrate
declare -p vars >&2
# extract and print a single variable by name
echo "Variable uid has value ${vars[uid]}"
Note that this must be run with bash yourscript, not sh yourscript.
By the way -- if you don't have bash 4.0, you might consider a different approach:
while IFS= read -r line; do
if [[ $line = *": "* ]]; then
key=${line%%": "*}
value=${line#*": "}
printf -v "ldap_$key" %s "$value"
fi
done <file1.txt # or < <(ldapsearch ...)
will create separate variables of the form "$ldap_cn" or "$ldap_uid", as opposed to putting everything in a single associative array.
Here's a simple example of what you are trying to do that should get you started. It assumes 1 set of data in the file. Although a tad brute-force, I believe its easy to understand.
Given a file called file.txt in the current directory with the following contents (absJobAction intentionally left out):
$ cat file1.txt
uid: user1
cn: User One
employeenumber: 1234567
$
This script gets each value into a local variable and prints it out:
# Use /bin/bash to run this script
#!/bin/bash
# Make SOURCEFILE a readonly variable. Make it uppercase to show its a constant. This is the file the LDAP values come from.
typeset -r SOURCEFILE=./file1.txt
# Each line sets a variable using awk.
# -F is the field delimiter. It's a colon and a space.
# Next is the value to look for. ^ matches the start of the line.
# When the above is found, return the second field ($2)
current_uid="$(awk -F': ' '/^uid/ {print $2}' ${SOURCEFILE})"
current_cn="$(awk -F': ' '/^cn/ {print $2}' ${SOURCEFILE})"
current_enbr="$(awk -F': ' '/^employeenumber/ {print $2}' ${SOURCEFILE})"
current_absja="$(awk -F': ' '/^absJobAction/ {print $2}' ${SOURCEFILE})"
# Print the contents of the variables. Note since absJobAction was not in the file,
# it's value is NULL.
echo "uid: ${current_uid}"
echo "cn: ${current_cn}"
echo "EmployeeNumber: ${current_enbr}"
echo "absJobAction: ${current_absja}"
~
When run:
$ ./test.sh
uid: user1
cn: User One
EmployeeNumber: 1234567
absJobAction:
$
I tried the following:
#!/bin/bash
while read device; do
name=$(echo "$device" | awk '{ print $1 }')
ip=$(echo "$device" | awk '{ print $2 }')
while read creds; do
community=$(echo "$creds" | awk '{ print $1 }')
version=$(echo "$creds" | awk '{ print $2 }')
mkdir -p walks/$name;
`echo -e "snmpwalk -v$version -c \x27$community\x27 $ip system > walks/$name/$community-$version.txt
done < <(##MySQL query that returns tuples in form: (snmp_ro,(1,2c,3))##")
done < <(cat devices.txt)
exit 0
This is meant to go through and find the snmp string and version of each device.
devices.txt is a list of devices in form: hostname ip
It doesn't create the file: walks/$name/$community-$version.txt, and it only seems to run through the walks 1 at a time, something I don't want.
Use & to put the contents you want backgrounded in, well, the background.
pids=( )
while read -r -u 3 name ip _; do
while read -r -u 4 community version _; do
mkdir -p "walks/$name"
snmpwalk -v"$version" -c "$community" "$ip" system \
</dev/null >"walks/$name/$community-$version.txt" & pids+=( "$!" )
done 4< <(: get data for "$name" and "$ip")
done 3<devices.txt
wait "${pids[#]}"
Other items of note:
read can already split fields into their own variables; using awk for this is silly.
The _ in read -r foo bar _ ensures that if more than two columns exist in the input file, the third column and onward are discarded (actually, put into a variable named _, but this is considered discard by convention) rather than appended to bar.
Make a habit of quoting expansions unless you have a specific and compelling reason to do otherwise; otherwise, you get string-splitting and glob expansion of string contents.
This example puts each input stream on its own file descriptor, and redirects each read to its own FD. This prevents any other content within your loop from consuming stdin.
How can I use awk and printf in for loop?
Here is my code
for fileName in /home/BamFiles/sample*
do
sampleIds=${fileName##*/}
for Bam in /home/BamFiles/sample*/*.bam
do
samtools idxstats $Bam | awk '{i+=$3+$4} END {printf("%s\t%d",$sampleIds Bam)}'
done
done
I get the the following error
awk: fatal: not enough arguments to satisfy format string
`%d %s'
^ ran out for this one
Expected output is
sample1 52432
sample2 32909
sample3 54000
sample5 45890
Thanks
Awk can never be able to expand a shell variable. You are also trying to pass only a single argument to printf (nothing is passed for %d.
Perhaps you want it this way:
samtools idxstats "$Bam" | awk -v "file=$fileName" '{i+=$3+$4} END {printf("%s\t%d\n", file, i)}'
Note that embedding a variable's value directly to awk's code may be possible by using double-quotes but is not recommended:
samtools idxstats "$Bam" | awk "{i+=\$3+\$4} END {printf(\"%s\\t%d\", $file, i)}"
Suggestion:
shopt -s nullglob
for sample in /home/BamFiles/sample*; do
for bam in "$sample"/*.bam; do
samtools idxstats "$bam"
done | awk -v sample="${sample##*/}" '{ i += $3 + $4 } END { printf("%s\t%d\n", sample, i) }'
done
Or
shopt -s nullglob
for sample in /home/BamFiles/sample*; do
for bam in "$sample"/*.bam; do
samtools idxstats "$bam" | \
awk -v sample="${sample##*/}" -v bam="${bam##*/}" \
'{ i += $3 + $4 } END { printf("%s\t%s\t%d\n", sample, bam, i) }'
done
done
This is my code:
nb_lignes=`wc -l $1 | cut -d " " -f1`
for i in $(seq $nb_lignes)
do
m=`head $1 -n $i | tail -1`
//command
done
Please how can i change it to get Get 20% of lines in File randomly to apply "command" on each line ?
20% or 40% or 60 % (it's a parameter)
Thank you.
This will randomly get 20% of the lines in the file:
awk -v p=20 'BEGIN {srand()} rand() <= p/100' filename
So something like this for the whole solution (assuming bash):
#!/bin/bash
filename="$1"
pct="${2:-20}" # specify percentage
while read line; do
: # some command with "$line"
done < <(awk -v p="$pct" 'BEGIN {srand()} rand() <= p/100' "$filename")
If you're using a shell without command substitution (the <(...) bit), you can do this - but the body of the loop won't be able to have any side effects in the outer script (e.g. any variables it sets won't be set anymore once the loop completes):
#!/bin/sh
filename="$1"
pct="${2:-20}" # specify percentage
awk -v p="$pct" 'BEGIN {srand()} rand() <= p/100' "$filename" |
while read line; do
: # some command with "$line"
done
Try this:
file=$1
nb_lignes=$(wc -l $file | cut -d " " -f1)
num_lines_to_get=$((20*${nb_lignes}/100))
for (( i=0; i < $num_lines_to_get; i++))
do
line=$(head -$((${RANDOM} % $nb_lignes)) $file | tail -1)
echo "$line"
done
Note that ${RANDOM} only generates numbers less than 32768 so this approach won't work for large files.
If you have shuf installed, you can use the following to get a random line instead of using $RANDOM.
line=$(shuf -n 1 $file)
you can do it with awk.see below:
awk -v b=20 '{a[NR]=$0}END{val=((b/100)*NR)+1;for(i=1;i<val;i++)print a[i]}' all.log
the above command prints 20% of all the lines starting from begining of the file.
you just have to change the value of b on command line to get the required % of lines.
tested below:
> cat temp
1
2
3
4
5
6
7
8
9
10
> awk -v b=10 '{a[NR]=$0}END{val=((b/100)*NR)+1;for(i=1;i<val;i++)print a[i]}' temp
1
> awk -v b=20 '{a[NR]=$0}END{val=((b/100)*NR)+1;for(i=1;i<val;i++)print a[i]}' temp
1
2
>
shuf will produce the file in a randomized order; if you know how many lines you want, you can give that to the -n parameter. No need to get them one at a time. So:
shuf -n $(( $(wc -l < $FILE) * $PCT / 100 )) "$file" |
while read line; do
# do something with $line
done
shuf comes standard with GNU/Linux distros afaik.