I have the following scenario:
all: file4.txt
file1.txt:
./program parameter1
file2.txt: file1.txt
./program parameter2
file3.txt: file2.txt
./program parameter3
file4.txt: file3.txt
./program parameter4
So, ./program parameter1 creates file1.txt, ./program parameter2 creates file2.txt, etc. And file3.txt can only be created if file2.txt already exists and file2.txt can only be created when file1.txt exists. But sometimes ./program parameters2 is able to create both file2.txt AND file3.txt. So there is no need to run the file3.txt target anymore but instead file4.txt should be created directly. In fact, rerunning the file3.txt target would break file3.txt in this case.
Is there a way to re-evaluate dependencies or somehow skip over target file3.txt if this file has been created? If I run make file2.txt and then make file4.txt this works and the target file3.txt is never executed. But running make file4.txt from the beginning will trigger the file3.txt target.
The biggest question is "how do you know if file3.txt needs to be updated or not?" If the only way is if the param2 run does NOT write it, then you'll have trouble if an old version is lying around. So about the only way to do it safely is to explicitly remove the old one and check if it is created. So you end up wanting something like:
file2.txt: file1.txt
rm file3.txt # make sure not to leave an old version around
./program parameter2
file3.txt: file2.txt
if [ -r $# ]; then \
touch $#; \
else \
./program parameter3; \
fi
Related
I have very large TXT files that I need to merge horizontally without leaving whitespaces. Please see the example below.
FILE1.txt
1
2
3
FILE2.txt
A
B
C
NEEDED OUTPUT FILE3.txt
1A
2B
3C
I currently use these 2 commands below to get the desired result, but would like to use just 1 command line because of the huge file size. I also need the results to be saved to an output file without being visible in Terminal.
paste -d ' ' FILE1.txt FILE2.txt | tee -a FILE3.txt
cat FILE3.txt | tr -d "[:blank:]"
Using the paste tool with \0 as a delimitator will do this.
paste -d'\0' file1.txt file2.txt > file3.txt
Consider this simple makefile:
all: output.txt
# The actual build command won't be this simple.
# It'll more be like "some-compiler file1.txt",
# which includes file2.txt automatically.
output.txt: file1.txt
cat file1.txt file2.txt > output.txt
file2.txt:
echo "heyo" > file2.txt
file1.txt: file2.txt
On first run, Make recognizes that file2.txt is a dependency of file1.txt, and so it needs to be built for output.txt to be built. Thus, it runs echo "heyo" > file2.txt and then cat file1.txt file2.txt > output.txt.
However, on subsequent runs, if file2.txt is changed, Make doesn't rebuild! If file1.txt is changed it does, but not for file2.txt. It just gives the dreaded make: Nothing to be done for 'all'. message.
One hacky solution I've seen people suggest is to do the following:
all: output.txt
output.txt: file1.txt file2.txt
cat file1.txt file2.txt > output.txt
However, that's not possible in my case, as my secondary dependencies (the lines like file1.txt: file2.txt) are dynamically generated using include.
How do I make sure Make checks for modifications all the way up the tree when I have multiple levels of dependencies?
I think the problem here is that your makefile is slightly too simple.
Let a -> b denote a depends on b. From your makefile you have...
output.txt -> file1.txt -> file2.txt
When make tries to update output.txt it sees that output.txt depends on file1.txt. It then notices that file1.txt depends on file2.txt. At that point the dependency chain stops. If make sees that file2.txt is newer than file1.txt it will run the command(s) that is associated with the file1.txt: file2.txt delendency. In this case, however, there aren't any commands -- just the dependency itself. That's fine as things go, but it does mean that even if file2.txt is updated file1.txt won't be. Hence, when make moves up the dependency chain to...
output.txt: file1.txt
it sees that output.txt is still newer than file1.txt so there is no need to run any command associated with that dependency.
If you add the touch command...
file1.txt: file2.txt
touch $#
then file1.txt will be updated and so the dependency chain works as you expect.
Your makefile does neither generate nor update file1.txt at all (i.e.: file1.txt must exist at the moment of running make). It contains no recipe for generating file1.txt from file2.txt. It has just an empty rule (i.e.: a rule without recipe):
file1.txt: file2.txt
Since file1.txt a prerequisite of output.txt, this empty rule just implies that file2.txt must exist for output.txt to be built, it does not even update file1.txt when file2.txt is generated.
Since file1.txt is the only prerequisite of output.txt and file1.txt is never updated by make, once output.txt is generated, it remains always up-to-date (provided file1.txt is not externally updated).
file2.txt being changed never causes output.txt to be rebuilt because:
it is not a prerequisite of output.txt.
it does not update file1.txt (which is the only prerequisite of output.txt).
Solution
Given your current output.txt rule:
output.txt: file1.txt
cat file1.txt file2.txt > output.txt
If you want output.txt to be built every time file2.txt changes, then you need file1.txt to be built every time file2.txt changes. This can be achieved by means of a rule whose recipe actually updates file1.txt and has file2.txt as prerequisite, e.g.:
file1.txt: file2.txt
touch $#
So I have a file1.txt with a list of names, and a file2.txt with another list of names and I need a list with the names that are in both files.
I tried grep-f file1.txt file2.txt > newlist.txt but for some reason it isn't working, and the newlist.txt has names that are not in file1.
Does anyone know why this is happening and what i could do to get only the names that are on both lists?
thank you.
If file1.txt and file2.txt are sorted, you could use 'comm'
comm -12 file1.txt file2.txt > newlist.txt
Your grep -f file1.txt file2.txt > newlist.txt is a nice thought, but will give too much hits when file1.txt has "s10" and file2.txt has "slass100". You want to match the complete line, so try
grep -Fxf file1.txt file2.txt > newlist.txt
This should be faster than a solution that requires sorting first.
If each the names in each list are unique, then you can find their intersection as follows:
sort file1.txt file2.txt | uniq -d > newlist.txt
I need some assistance on the below.
File1.txt
aaa:/path/to/aaa:777
bob:/path/to/bbb:700
ccc:/path/to/ccc:600
File2.txt
aaa:/path/to/aaa:700
bbb:/path/to/bbb:700
ccc:/path/to/ccc:644
I should iterate file2.txt and if aaa exists in File1.txt, then i should compare the file permission. If the file permission is same for aaa in both the files then ignore.
If they are different then write them in the output.txt
So in above case
Output.txt
aaa:/path/to/aaa:700
ccc:/path/to/ccc:644
How can i achieve this in unix shell script? Please suggest
I agree with the comment of #Marc that you should try something before asking here.
However, the following answer is difficult to find when you never have seen the constructions, so I give you something to study.
When you want to parse line by line, you can start with
while IFS=: read -r file path mode; do
comparewith=$(grep "^${file}:${path}:" File2.txt | cut -d: -f3)
# compare and output
done < File1.txt
For large files that will become very slow.
You can first filter the lines you want to compare from File2.txt.
You want to grep strings like aaa:/path/to/aaa:, including the last :. With cut -d: -f1-2 you might be fine with your inputfile, but maybe it is better to remove the last three characters:
sed 's/...$//' File1.txt.
You can let grep use the output as a file with expressions using <():
grep -f <(sed 's/...$//' File1.txt) File2.txt
Your example files don't show the situation when both files have identical lines (that you want to skip), you will need another process substitution to get that working:
grep -v -f File1.txt <(grep -f <(sed 's/...$//' File1.txt ) File2.txt )
Another solution, worth trying yourself, is using awk (see What is "NR==FNR" in awk? for accessing 2 files).
comm - compare two sorted files line by line
According to manual, comm -13 <file1> <file2> must print only lines unique to <file2>:
$ ls
File1.txt File2.txt
$ cat File1.txt
aaa:/path/to/aaa:777
bbb:/path/to/bbb:700
ccc:/path/to/ccc:600
$ cat File2.txt
aaa:/path/to/aaa:700
bbb:/path/to/bbb:700
ccc:/path/to/ccc:644
$ comm -13 File1.txt File2.txt
aaa:/path/to/aaa:700
ccc:/path/to/ccc:644
$ # Nice!
But it doesn't check for lines in <file1> that are "similar" to corresponding lines of <file2>. I. e. it won't work as you want if File1.txt has line BOB:/path/to/BOB:700 and File2.txt has BBB:/path/to/BBB:700 since it will print the latter (while you want it not to be printed).
It also won't do what you want if strings bbb:/path/to/bbb:700 and bbb:/another/path/to/bbb:700 are supposed to be "identical".
I want to compare two catalogues. They look like this:
Catalogue1 Catalogue 2
file1.txt file1.txt
file2.txt file2.txt
file3.txt file3.txt
file4.txt file4.txt
file5.txt file5.txt
file6.txt file6.txt
files.zip files.zip
I just want to have the *.txt files compared, but the command
diff -r --exclude=".zip" $CURRENT_BUILD_DIR $PREVIOUS_BUILD_DIR >> $DIFF_DIR/DIFF.TXT
is of no use here. The effect of this command looks like this
Files /home/blah/workspace/Catalogue1/files.zip and /home/blah/workspace/Catalogue2/files.zip are different. // not sure about that, translated it from German.
diff -r '--exclude=.zip' /home/blah/workspace/Catalogue1/file1.TXT /home/blah/workspace/Catalogue2/file1.TXT
1c1
< 1
---
> 2
Why is diff still trying to compare the zip files even with the exclude option?
diff -r --exclude="*.zip" "$CURRENT_BUILD_DIR" "$PREVIOUS_BUILD_DIR" >> "$DIFF_DIR"/DIFF.TXT