I am trying to write a bash script that looks at two files with the same name, each in a different directory.
I know this can be done with diff -r, however, I would like to take everything that is in the second file that is not in the first file and output it into an new file (also with the same file name)
I have written a (nested) loop with a grep command but it's not good and gives back a syntax error:
#!/bin/bash
FILES=/Path/dir1/*
FILES2=/Path/dir2/*
for f in $FILES
do
for i in $FILES2
do
if $f = $i
grep -vf $i $f > /Path/dir3/$i
done
done
Any help much appreciated.
try this
#!/bin/bash
cd /Path/dir1/
for f in *; do
comm -13 <(sort $f) <(sort /Path/dir2/$f) > /Path/dir3/$f
done
if syntax in shell is
if test_command;then commands;fi
commands are executed if test_command exit code is 0
if [ $f = $i ] ; then grep ... ; fi
but in your case it will be more efficient to get the file name
for i in $FILES; do
f=/Path/dir2/`basename $i`
grep
done
finally, maybe this will be more efficient than grep -v
comm -13 <(sort $f) <(sort $i)
comm -13 will get everything which is in the second and not in first ; comm without arguments generates 3 columns of output : first is only in first, second only in second and third what is common.
-13 or -1 -3 removes first and third column
#!/bin/bash
DIR1=/Path/dir1
DIR2=/Path/dir2
DIR3=/Path/dir3
for f in $DIR1/*
do
for i in $DIR2/*
do
if [ "$(basename $f)" = "$(basename $i)" ]
then
grep -vf "$i" "$f" > "$DIR3/$(basename $i)"
fi
done
done
This assumes no special characters in filenames. (eg, whitespace. Use double quotes if that is unacceptable.):
a=/path/dir1
b=/path/dir2
for i in $a/*; do test -e $b/${i##*/} &&
diff $i $b/${i##*/} | sed -n '/^< /s///p'; done
Related
I have 10 subfolders under one folder and I have 10 .gz files under each one. I need to enter subfiles from the main file and take this .gz file and process it in a code order (from 91 to 99). I printed them all in order in the log3 file. How can I read them in the code again in order?
I was able to generate log 1,2,3 files sequentially with the following codes.
find . -name '*.gz' -printf '%f\0' |
sort -z |
while IFS='' read -r -d '' fname
do
printf '%s\n' "${fname:0:4}" >&3
printf '%s\n' "${fname:16:17}" >&4
printf '%s\n' "${fname:0:100}" >&5
done \
3> >(tee -a receiver_ids > log) \
4> >(tee -a doy > log2) \
5> >(tee -a data_record > log3)
but I cannot run rnxEditGde.py using these log files. I tried the code below but it can't find the -dataFile.
for j in {091..099}; do
ionex=$(pl $j log)
summary=$(pl $j log2)
dataRecordFile=$(pl $j log3)
gd2e.py -mkTreeS Trees
sed -i "s/jplg.*/$ionex/g" $dir/Trees/ppp_0.tree
rnxEditGde.py -dataFile "$dataRecordFile" -o dataRecordFile.Orig.gz
The form of your pl function
function pl { sed -n "$1p" $2 }
will look for line number $1 in the file, then print that, regardless of a match on the value of $1. You probably wanted to use the "//" delimiters to define the pattern to match before printing the line, in this manner:
sed -n "/${1}/p" ${2}
Also, not having the braces makes the referencing of $1p ambiguous, so you really should use those, especially when you did intend to use "$1p" (without //).
I want to write into a file in a bash script but I want to make sure that the file is backed up if it exists and I also want to avoid overwriting any existing backups.
So basically I have $FILE, if this exists, I want to move $FILE to $FILE.bak if it does not already exist, otherwise to $FILE.bak2, $FILE.bak3, etc.
Is there a shell command for this?
Using a function to find the next available name:
#!/usr/bin/env bash
function nextsuffix {
local name="$1.bak"
if [ -e "$name" ]; then
printf "%s" "$name"
else
local -i num=2
while [ -e "$name$num" ]; do
num+=1
done
printf "%s%d" "$name" "$num"
fi
}
mv "$1" "$(nextsuffix "$1")"
If foo.bak already exists, it just loops until a given foo.bakN filename doesn't exist, incrementing N each time.
You can just output to a file with a date.
FILE=~/test
echo "123" >> $FILE.$(date +'%Y%d%m')
If you want the numbers logrotate seems to be most ideal.
cp "$FILE" "$FILE.bak$(( $(grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 )) + 1 ))"
Breaking the commands down
sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1
List the last file in the directory which is sorted in numeric form
grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 ))
Strip out everything but the digits
(( $(grep -Eo '[[:digit:]]+' <(sort -n <(for fil in $FILE.bak*;do echo $fil;done) | tail -1 )) + 1 ))
Add one to the result
For posterity, my function with changes inspired by #Shawn's answer
backup() {
local file new n=0
local fmt='%s.%(%Y%m%d)T_%02d'
for file; do
while :; do
printf -v new "$fmt" "$file" -1 $((++n))
[[ -e $new ]] || break
done
command cp -vp "$file" "$new"
done
}
I like to cp not mv.
I have a textfile called log.txt, and it logs the file name and the path it was gotten from. so something like this
2.txt
/home/test/etc/2.txt
basically the file name and its previous location. I want to use grep to grab the file directory save it as a variable and move the file back to its original location.
for var in "$#"
do
if grep "$var" log.txt
then
# code if found
else
# code if not found
fi
this just prints out to the console the 2.txt and its directory since the directory has 2.txt in it.
thanks.
Maybe flip the logic to make it more efficient?
f=''
while read prev
do case "$prev" in
*/*) f="${prev##*/}"; continue;; # remember the name
*) [[ -e "$f" ]] && mv "$f" "$prev";;
done < log.txt
That walks through all the files in the log and if they exist locally, move them back. Should be functionally the same without a grep per file.
If the name is always the same then why save it in the log at all?
If it is, then
while read prev
do f="${prev##*/}" # strip the path info
[[ -e "$f" ]] && mv "$f" "$prev"
done < <( grep / log.txt )
Having the file names on the same line would significantly simplify your script. But maybe try something like
# Convert from command-line arguments to lines
printf '%s\n' "$#" |
# Pair up with entries in file
awk 'NR==FNR { f[$0]; next }
FNR%2 { if ($0 in f) p=$0; else p=""; next }
p { print "mv \"" p "\" \"" $0 "\"" }' - log.txt |
sh
Test it by replacing sh with cat and see what you get. If it looks correct, switch back.
Briefly, something similar could perhaps be pulled off with printf '%s\n' "$#" | grep -A 1 -Fxf - log.txt but you end up having to parse the output to pair up the output lines anyway.
Another solution:
for f in `grep -v "/" log.txt`; do
grep "/$f" log.txt | xargs -I{} cp $f {}
done
grep -q (for "quiet") stops the output
I don't know why this is not working:
for g in *.txt; do for f in $(cat $g); do grep $f annotations.csv; done > ../$f_annot; done
I want to loop through each file in a folder, for each file I want to loop through each line and apply the grep command. When I do
for f in $(cat file1.txt); do grep $f annotations.csv; done > ../$f_annot
It works, it is the nested loop that doesn't output anything, it seems like it is running but it lasts forever and does nothing.
When you hava an empty txt file, grep $f annotations.csv will be translated into a grep command reading from stdin.
You might want to use something like
for g in *.txt; do
grep -f $g annotations.csv > ../$g_annot
done
SOLVED:
for file in *list.txt; do
while read -r line; do
grep "$line" annotations.csv
done < "$file"
> ${file}_annot.txt
done
:)
I get
Cannot write to a directory.
ksh: ../: 0403-005 Cannot create the specified file.
But this is because $f_annot evaluates to what we expect. It should be better with ${f}_annot:
for g in *.txt; do for f in $(cat $g); do grep $f annotations.csv; done > ../${f}_annot ; done
But there is an issue in your script because it erase the result of some loops: $f always has the last value of the loop. Maybe this suits your need better:
for g in *.txt; do for f in $(cat $g); do grep $f annotations.csv; done ; done
I'm trying to sort a folder on a Mac with hundreds of jpgs into separate folders
after their pixelWidth with /usr/bin/sips. When I run the script it shows an
Error 4: no file was specified. The variable $f is empty in the first line?
I thought the "f" in for f in *jpg referred to the file as it loops through the folder.
imgWid=$( /usr/bin/sips -g pixelWidth $f | sed -n 2p | cut -d":" -f2- )
for f in *jpg; do if [ "$1" == "$imgWid" ]; then; mv $f $dir1 2>/dev/null; fi; done
I would appreciate help on how I can fix this.
$() runs the command at the point specified. (So line 1), $f is only defined in the loop (line 3).
Maybe:
function determineImageWidth()
{
/usr/bin/sips -g pixelWidth $1 | sed -n 2p | cut -dj":" -f2-
}
for f in *.jpg
do
width=$(determineImageWidth $f)
if [ -n $width ]
then
mkdir -p $width
mv $f $width/
else
echo "Can't determine the width for $f"
fi
done
You didn't specify what $f is in the first line of your script. As far as I can see it is just specified within the for loop.
You might want to change $f to $1 in the first line, to use an argument given to your script.