BASH - echo issues, wont print anything but while read argument - bash

We are having a wierd issue.
We have this lines :
while read line2; do
echo $line2
done < $1 | `echo grep '.*|.*|.*|.*|.*|.*|.*|.*'` | sort -nbsk1 | cut -d "|" -f1 | uniq -d
Which prints what they should print. but, when changing the echo to ->
while read line2; do
echo "Hello World"
done < $1 | `echo grep '.*|.*|.*|.*|.*|.*|.*|.*'` | sort -nbsk1 | cut -d "|" -f1 | uniq -d
It wont print anything, same result for anything different then $line2.
Whats even more wierd is :
echo " $line2 Hello"
Will print the line2 variable
echo "Hello $line2"
Print nothing
I have tried the same with printf, same results.
Any suggestions ?

What you've written is equivalent to the following shell code:
cat $1 |
while read line2; do
echo $line2
done |
`echo grep '.*|.*|.*|.*|.*|.*|.*|.*'` |
sort -nbsk1 |
cut -d "|" -f1 |
uniq -d
The while read loop takes the contents of file $1 and echoes them, which does nothing other than remove leading and trailing spaces and replace internal spaces with a single space. If you replace the echo $line2 line with echo "Hello World", that string is clearly not going to match the grep command that the output of the loop is being passed through, so producing no output is unsurprising.
When you change the echo line to echo " $line2 Hello", you tack "Hello" onto the end of the input line, which then matches the grep command and gets sliced off the end of the string with the cut command, so it makes sense that it would have essentially no ultimate effect.
If you change the echo line to echo "Hello $line2", any number at the beginning of the line becomes invisible to the sort -ns, which makes your sort call essentially a no-op. This is probably why you're not seeing anything in this situation, although you probably would see something if two identical lines appeared in the input one after the other. (In my testing on my machine, I see one such line because I happen to have two identical lines in succession in my test case.)
It's not exactly clear what you're trying to do since the while loop is almost a no-op. It's possible what you want to do is something more like this:
grep '.*|.*|.*|.*|.*|.*|.*|.*' < $1 |
sort -nbsk1 |
cut -d "|" -f1 |
uniq -d |
while read line2; do
echo $line2
done
... but I'm only speculating at this point.

Related

bash assign variable to another after operation

I'm trying to print domain and topLeveldomain variables (example.com)
$line = example.com
domain =$line | cut -d. -f 1
topLeveldomain = $line | cut -d. -f 2
However when I try and echo $domain, it doesn't display desired value
test.sh: line 4: domain: command not found
test.sh: line 5: topLeveldomain: command not found
I suggest:
line="example.com"
domain=$(echo "$line" | cut -d. -f 1)
topLeveldomain=$(echo "$line" | cut -d. -f 2)
The right code for this should be:
line="example.com"
domain=$(echo "$line" | cut -d. -f 1)
topLeveldomain=$(echo "$line" | cut -d. -f 2)
Consider the right syntax of bash:
variable=value
(there are no blanks allowed)
if you want to use the content of the variable you have to add a leading $
e.g.
echo $variable
You don't need external tools for this, just do this in bash
$ string="example.com"
# print everything upto first de-limiter '.'
$ printf "${string%%.*}\n"
example
# print everything after first de-limiter '.'
$ printf "${string#*.}\n"
com
Remove spaces around =:
line=example.com # YES
line = example.com # NO
When you create a variable, do not prepend $ to the variable name:
line=example.com # YES
$line=example.com # NO
When using pipes, you need to pass standard output to the next command. Than means, you usually need to echo variables or cat files:
echo $line | cut -d. -f1 # YES
$line | cut -d. -f1 # NO
Use the $() syntax to get the output of a command into a variable:
new_variable=$(echo $line | cut -d. -f1) # YES
new_variable=echo $line | cut -d. -f1 # NO
I would rather use AWK:
domain="abc.def.hij.example.com"
awk -F. '{printf "TLD:%s\n2:%s\n3:%s\n", $NF, $(NF-1), $(NF-2)}' <<< "$domain"
Output
TLD:com
2:example
3:hij
In the command above, -F option specifies the field separator; NF is a built-in variable that keeps the number of input fields.
Issues with Your Code
The issues with your code are due to invalid syntax.
To set a variable in the shell, use
VARNAME="value"
Putting spaces around the equal sign will cause errors. It is a good
habit to quote content strings when assigning values to variables:
this will reduce the chance that you make errors.
Refer to the Bash Guide for Beginners.
this also works:
line="example.com"
domain=$(echo $line | cut -d. -f1)
toplevel=$(cut -d. -f2 <<<$line)
echo "domain name=" $domain
echo "Top Level=" $toplevel
You need to remove $ from line in the beginning, correct the spaces and echo $line in order to pipe the value to cut . Alternatively feed the cut with $line.

exiting an IF statement after initial match bash scripting

I have a script which iterates through a file and finds matches in another file. How to I get the process to stop once I've found a match.
For example:
I take the first line in name.txt, and then try to find a match for it in file.txt.
name.txt:
7,7,FRESH,98,135,
65,10,OLD,56,45,
file.txt:
7,7,Dave,S
8,10,Frank,S
31,7,Gregg
45,5,Jake,S
Script:
while read line
do
name_id=`echo $line | cut -f1,2 -d ','`
identiferOne=`echo $name_id | cut -f1 -d ','`
identiferTwo=`echo $name_id | cut -f2 -d ','`
while IFS= read line
do
CHECK=`echo $line | cut -f4 -d','`
if [ $CHECK = "S" ]
then
symbolName=`echo $line | cut -f3 -d ','`
numberOne=`echo $line | awk -F',' '{print $1}'`
numberTwo=`echo $line | cut -f2 -d ','`
if [ "$numberOne" == $identiferOne ] && [ "$numberTwo" == $identifierTwo ]
then
echo "WE HAVE A MATCH with $symbolName"
break
fi
fi
done < /tmp/file.txt
done < /tmp/name.txt
My question is - how do I stop the script from iterating through file.txt once it has found an initial match, and then set that matched record into a variable, stop the if statement, then do some other stuff within the loop using that variable. I tried using break; but that exits the loop, which is not what I want.
You can tell grep different things:
Stop searching after the first match (option -m 1).
Read the searchkeys from a file (option -f file).
Pretend that the output of a command is a file (not really grep, bash helps here) with <(cmmnd).
Combining these will give you
grep -m1 -f <(cut -d"," -f1-2 name.txt) file.txt
Close, but not what you want. The substrings given by cut -d"," -f1-2 name.txt will match everywhere in the line, and you want to match the first two fields. Matching at the start of the line is done with ^, so we use sed to make strings like ^field1,field2 :
grep -m1 -f <(sed 's/\([^,]*,[^,]*,\).*/^\1/' name.txt) file.txt

How to process values from for loop in shell script

I have below for loop in shell script
#!/bin/bash
#Get the year
curr_year=$(date +"%Y")
FILE_NAME=/test/codebase/wt.properties
key=wt.cache.master.slaveHosts=
prop_value=""
getproperty(){
prop_key=$1
prop_value=`cat ${FILE_NAME} | grep ${prop_key} | cut -d'=' -f2`
}
#echo ${prop_value}
getproperty ${key}
#echo "Key = ${key}; Value="${prop_value}
arr=( $prop_value )
for i in "${arr[#]}"; do
echo $i | head -n1 | cut -d "." -f1
done
The output I am getting is as below.
test1
test2
test3
I want to process the test2 from above results to below script in place of 'ABCD'
grep test12345 /home/ptc/storage/**'ABCD'**/apache/$curr_year/logs/access.log* | grep GET > /tmp/test.access.txt
I tried all the options but could not able to succeed as I am new to shell scripting.
Ignoring the many bugs elsewhere and focusing on the one piece of code you say you want to change:
for i in "${arr[#]}"; do
val=$(echo "$i" | head -n1 | cut -d "." -f1)
grep test12345 /dev/null "/home/ptc/storage/$val/apache/$curr_year/logs/access.log"* \
| grep GET
done > /tmp/test.access.txt
Notes:
Always quote your expansions. "$i", "/path/with/$val/"*, etc. (The * should not be quoted on the assumption that you want it to be expanded).
for i in $prop_value would have the exact same (buggy) behavior; using arr buys you nothing. If you want using arr to increase correctness, populate it correctly: read -r -a arr <<<"$prop_value"
The redirection is moved outside the loop -- that way the second iteration through the loop doesn't overwrite the file written by the first one.
The extra /dev/null passed to grep ensures that its behavior is consistent regardless of the number of matches; otherwise, it would display filenames only if more than one matching log file existed, and not otherwise.

Inifinite loop in bash

I have written the following command to loop over a set of strings in the second column of my file and then do sorting for each string on column 11, then take the second and eleventh column and count the number of unique occurrences. Very simple but it seems that it enters an infinite loop and I can't see why. I would appreciate your help very much.
for item in $(cat file.txt | cut -f2 -d " "| uniq)
do
sort -k11,11 file.txt | cut -f2,11 -d " " | uniq -c | sort -k2,2 > output
done
There's no infinite loop here, but it is a very silly loop (that takes a long time to run, while not accomplishing the script's stated purpose). Let's look at how one might accomplish that purpose more sanely:
Using a temporary file for counts.txt to avoid needing to rerun the sort, cut and uniq steps on each iteration:
sort -k11,11 file.txt | cut -f2,11 -d " " | uniq -c >counts.txt
while read -r item; do
fgrep -e " ${item}" counts.txt
done < <(cut -f2 -d' ' <file.txt | uniq)
Even better, using bash 4 associative arrays and no temporary file:
# reads counts into an array
declare -A counts=( )
while read -r count item; do
counts[$item]=count
done < <(sort -k11,11 file.txt | cut -f2,11 -d " " | sort | uniq -c)
# reads counts back out
while read -r item; do
echo "$item ${counts[$item]}"
done < <(cat file.txt | cut -f2 -d " "| sort | uniq)
...that said, that's only if you want to use sort for ordering on pulling data back out. If you don't need to do that, the latter part could be replaced as such:
# read counts back out
for item in "${!counts[#]}"; do
echo "$item ${counts[$item]}"
done

Weird bash results using cut

I am trying to run this command:
./smstocurl SLASH2.911325850268888.911325850268896
smstocurl script:
#SLASH2.911325850268888.911325850268896
model=$(echo \&model=$1 | cut -d'.' -f 1)
echo $model
imea1=$(echo \&simImea1=$1 | cut -d'.' -f 2)
echo $imea1
imea2=$(echo \&simImea2=$1 | cut -d'.' -f 3)
echo $imea2
echo $model$imea1$imea2
Result Received
&model=SLASH2911325850268888911325850268896
Result Expected
&model=SLASH2&simImea1=911325850268888&simImea2=911325850268896
What am I missing here ?
You are cutting based on the dot .. In the first case your desired string contains the first string, the one containing &model, so then it is printed.
However, in the other cases you get the 2nd and 3rd blocks (-f2, -f3), so that the imea text gets cutted off.
Instead, I would use something like this:
while IFS="." read -r model imea1 imea2
do
printf "&model=%s&simImea1=%s&simImea2=%s\n" $model $imea1 $imea2
done <<< "$1"
Note the usage of printf and variables to have more control about what we are writing. Using a lot of escapes like in your echos can be risky.
Test
while IFS="." read -r model imea1 imea2; do printf "&model=%s&simImea1=%s&simImea2=%s\n" $model $imea1 $imea2
done <<< "SLASH2.911325850268888.911325850268896"
Returns:
&model=SLASH2&simImea1=911325850268888&simImea2=911325850268896
Alternatively, this sed makes it:
sed -r 's/^([^.]*)\.([^.]*)\.([^.]*)$/\&model=\1\&simImea1=\2\&simImea2=\3/' <<< "$1"
by catching each block of words separated by dots and printing back.
You can also use this way
Run:
./program SLASH2.911325850268888.911325850268896
Script:
#!/bin/bash
String=`echo $1 | sed "s/\./\&simImea1=/"`
String=`echo $String | sed "s/\./\&simImea2=/"`
echo "&model=$String
Output:
&model=SLASH2&simImea1=911325850268888&simImea2=911325850268896
awk way
awk -F. '{print "&model="$1"&simImea1="$2"&simImea2="$3}' <<< "SLASH2.911325850268888.911325850268896"
or
awk -F. '$0="&model="$1"&simImea1="$2"&simImea2="$3' <<< "SLASH2.911325850268888.911325850268896"
output
&model=SLASH2&simImea1=911325850268888&simImea2=911325850268896

Resources