This question already has answers here:
How to merge two files consistently line by line
(6 answers)
Closed 2 years ago.
I have 2 files
#cat file.txt
aaa
bbb
ccc
#cat input.txt
191
112
339
I need
#cat output
aaa,191
bbb,112
ccc,339
I am trying
while IFS= read -r line
do
cat file.txt | grep "\.$line$" | wc -l >> output.txt
sed -i "1 i\\$line" output.txt
done < input.txt
but it prints all lines each file in the group.
Any idea how can I archive my result?
if you are the type who prefers to write more lines rather than fewer lines you can try this:
echo -n "" > output.txt
while IFS= read -r line_input && IFS= read -r line_file <&3
do
echo "$line_input,$line_file" >> output.txt
done <input.txt 3<file.txt
if not, you can simple use #RavinderSingh13 hint ;-)
Update
There was a comment regarding that the solution I supplied is slower then the solution
paste -d ',' file.txt input.txt
So I measured the execution time and I got the following values :
#!/bin/bash
START=$(date +%s.%N)
paste -d ',' file.txt input.txt > output.txt
END=$(date +%s.%N)
DIFF=$(echo "$END - $START" | bc)
echo "$DIFF";
I got the fastest :
.004961322
| test 1 |.005088686 |
| test 2 |.005036140 |
| test 3 |.005019742 |
| test 4 |.004961322 |
| test 5 |.005030747 |
| test 6 |.004978428 |
Next I checked the solution that I have supplied
#!/bin/bash
START=$(date +%s.%N)
echo -n "" > output.txt
while IFS= read -r line_input && IFS= read -r line_file <&3
do
echo "$line_input,$line_file" >> output.txt
done <input.txt 3<file.txt
END=$(date +%s.%N)
DIFF=$(echo "$END - $START" | bc)
echo "$DIFF";
And I got
.002386748
| test 1 |.003516318 |
| test 2 |.003538960 |
| test 3 |.002386748 |
| test 4 |.002584314 |
| test 5 |.003461923 |
| test 6 |.003619142 |
So my solution is just a bit longer, but it is faster then using paste.
The test was run on the same system.
Related
I want to pipe the output of the command into two commands and paste the results together. I found this answer and similar ones suggesting using tee but I'm not sure how to make it work as I'd like it to.
My problem (simplified):
Say that I have a myfile.txt with keys and values, e.g.
key1 /path/to/file1
key2 /path/to/file2
What I am doing right now is
paste \
<( cat myfile.txt | cut -f1 ) \
<( cat myfile.txt | cut -f2 | xargs wc -l )
and it produces
key1 23
key2 42
The problem is that cat myfile.txt is repeated here (in the real problem it's a heavier operation). Instead, I'd like to do something like
cat myfile.txt | tee \
<( cut -f1 ) \
<( cut -f2 | xargs wc -l ) \
| paste
But it doesn't produce the expected output. Is it possible to do something similar to the above with pipes and standard command-line tools?
This doesn't answer your question about pipes, but you can use AWK to solve your problem:
$ printf %s\\n 1 2 3 > file1.txt
$ printf %s\\n 1 2 3 4 5 > file2.txt
$ cat > myfile.txt <<EOF
key1 file1.txt
key2 file2.txt
EOF
$ cat myfile.txt | awk '{ ("wc -l " $2) | getline size; sub(/ .+$/,"",size); print $1, size }'
key1 3
key2 5
On each line we first we run wc -l $2 and save the result into a variable. Not sure about yours, but on my system wc -l includes the filename in the output, so we strip it with sub() to match your example output. And finally, we print the $1 field (key) and the size we got from wc -l command.
Also, can be done with shell, now that I think about it:
cat myfile.txt | while read -r key value; do
printf '%s %s\n' "$key" "$(wc -l "$value" | cut -d' ' -f1)"
done
Or more generally, by piping to two commands and using paste, therefore answering the question:
cat myfile.txt | while read -r line; do
printf %s "$line" | cut -f1
printf %s "$line" | cut -f2 | xargs wc -l | cut -d' ' -f1
done | paste - -
P.S. The use of cat here is useless, I know. But it's just a placeholder for the real command.
I'm trying to find all words in a file that don't exist in the dictionary. If I look for a single word the following works
b=ther; look $b | grep -i "^$b$" | ifne -n echo $b => ther
b=there; look $b | grep -i "^$b$" | ifne -n echo $b => [no output]
However if I try to run a "while read" loop
while read a; do look $a | grep -i "^$a$" | ifne -n echo "$a"; done < <(tr -s '[[:punct:][:space:]]' '\n' <lotr.txt |tr '[:upper:]' '[:lower:]')
The output seems to contain all (?) words in the file. Why doesn't this loop only output non-dictionary words?
Regarding ifne
If stdin is non-empty, ifne -n reprints stdin to stdout. From the manpage:
-n Reverse operation. Run the command if the standard input is empty
Note that if the standard input is not empty, it is passed through
ifne in this case.
strace on ifne confirms this behavior.
Alternative
Perhaps, as an alternative:
1 #!/bin/bash -e
2
3 export PATH=/bin:/sbin:/usr/bin:/usr/sbin
4
5 while read a; do
6 look "$a" | grep -qi "^$a$" || echo "$a"
7 done < <(
8 tr -s '[[:punct:][:space:]]' '\n' < lotr.txt \
9 | tr '[A-Z]' '[a-z]' \
10 | sort -u \
11 | grep .
12 )
I assign a keyword as variable, and need to awk from a file using this variable and loop. The file has millions of lines.
i have tried the code below.
DEVICE="DEV2"
while read -r line
do
echo $line
X_keyword=`echo $line | cut -d ',' -f 2 | grep -w "X" | cut -d '=' -f2`
echo $X_keyword
done <<< "$(grep -w $DEVICE $config)"
log="Dev2_PRT.log"
while read -r file
do
VALUE=`echo $file | cut -d '|' -f 1`
HEADER=`echo $VALUE | cut -c 1-4`
echo $file
if [[ $HEADER = 'PTR:' ]]; then
VALUE=`echo $file | cut -d '|' -f 4`
echo $VALUE
XCOORD+=($VALUE)
((X++))
fi
done <<< "awk /$X_keyword/ $log"
expected result:
the log files content lots of below:
PTR:1|2|3|4|X_keyword
PTR:1|2|3|4|Y_rest .....
Filter the X_keyword and get the field no 4.
Unfortunately your shell script is simply the wrong approach to this problem (see https://unix.stackexchange.com/q/169716/133219 for some of the reasons why) so you should set it aside and start over.
To demonstrate the solution, lets create a sample input file:
$ seq 10 | tee file
1
2
3
4
5
6
7
8
9
10
and a shell variable to hold a regexp that's a character list of the chars 5, 6, or 7:
$ var='[567]'
Now, given the above input, here is the solution for how to g/re/p pattern as variable and count how many results:
$ awk -v re="$var" '$0~re{print; c++} END{print "---" ORS c+0}' file
5
6
7
---
3
If that's not all you need then please edit your question to clarify your requirements and provide concise, testable sample input and expected output.
I've got a piece of a script I'm trying to figure out, so maybe its a simple question for someone more experienced out there.
Here is the code:
#!/bin/bash
echo "obase=2;$1" | bc
Used like:
$./script 12
Outputs:
1100
My question is, how can I parse this 4 digit number into separate digits? (to then delimit with cut -d ' ' and input those into an array...)
I'd like to be able to get the following output:
1 1 0 0
Is this even possible in BASH? I know its easier with other languages.
can use sed
echo "obase=2;$1" | bc | sed 's/./& /g'
or if you prefer longer form:
echo "obase=2;$1" | bc | sed 's/\(.\)/\1 /g'
if your sed supports -r
echo "obase=2;$1" | bc | sed -r 's/(.)/\1 /g'
To print individual digits from a string you can use fold:
s=1100
fold -w1 <<< "$s"
1
1
0
0
To create an array:
arr=( $(fold -w1 <<< "$s") )
set|grep arr
arr=([0]="1" [1]="1" [2]="0" [3]="0")
I am trying a simple shell script like the following:
#!/bin/bash
up_cap=$( cat result.txt | cut -d ":" -f 6,7 | sort -n | cut -d " " -f 2 | sort -n)
down_cap=$( cat result.txt | cut -d : -f 6,7 | sort -n | cut -d " " -f 6| sort -n)
for value in "${down_cap[#]}";do
if [ $value > 80000 ]; then
cat result.txt | grep -B 1 "$value"
fi
done
echo " All done, exiting"
when I execute the above script as ./script.sh, I get the error:
./script.sh: line 5: [: too many arguments
All done, exiting
I have googled enough, and still not able to rectify this.
You want
if [ "$value" -gt 80000 ]; then
You use -gt for checking if A is bigger than B, not >. The quotation marks I merely added to prevent the script from failing in case $value is empty.
Try to declare variable $value explicitly:
declare -i value
So, with the dominikh's and mine additions the code should look like this:
#!/bin/bash
up_cap=$( cat result.txt | cut -d ":" -f 6,7 | sort -n | cut -d " " -f 2 | sort -n)
down_cap=$( cat result.txt | cut -d : -f 6,7 | sort -n | cut -d " " -f 6| sort -n)
for value in "${down_cap[#]}";do
declare -i value
if [ $value -gt 80000 ]; then
cat result.txt | grep -B 1 "$value"
fi
done
echo " All done, exiting"