Firstly, here's my Makefile, it's really, really cool.
It generates an executable for every 'c' file in the current directory.
Trivial with shell, PITA with Make... Did I mention, I haven't 'c'd in a decade so please don't laugh...
CC = gcc
#List all 'c' files by wildcard.
SOURCES=$(wildcard *.c)
#Get the 'exe' equivalent of said file
EXECUTABLES=$(SOURCES:%.c=%)
#For each of the '*.exe' files
#append a 'c' suffix to the target
build: $(EXECUTABLES)
$(CC) -c $(<:%=%.c) -o $<
clean:
find ./ ! -name "*.c" -a ! -name "Makefile" -type f -delete
rebuild: clean build
Here are contents of my directory:
% ls -la
total 32
drwxr-xr-x 2 yomom yomom 4096 2012-01-27 13:38 ./
drwx------ 94 yomom yomom 4096 2012-01-27 13:38 ../
-rw-r--r-- 1 yomom yomom 990 2012-01-27 12:47 array_example.c
-rw-r--r-- 1 yomom yomom 341 2012-01-27 13:38 Makefile
-rw-r--r-- 1 yomom yomom 430 2011-12-05 13:08 pointers.c
-rw-r--r-- 1 yomom yomom 319 2012-01-27 12:45 should_create_warning.c
-rw-r--r-- 1 yomom yomom 1472 2011-12-19 16:16 socket-client.c
-rw-r--r-- 1 yomom yomom 1150 2011-12-19 16:15 socket-server.c
Now I run make
% make
gcc array_example.c -o array_example
gcc pointers.c -o pointers
gcc should_create_warning.c -o should_create_warning
gcc socket-client.c -o socket-client
gcc socket-server.c -o socket-server
gcc -c array_example.c -o array_example
Great, it re-compiled everything, the last line of output looked a bit inconsistent though.
How come it's different to the others?
% ls -la
total 68
drwxr-xr-x 2 yomom yomom 4096 2012-01-27 13:38 ./
drwx------ 94 yomom yomom 4096 2012-01-27 13:38 ../
-rw-r--r-- 1 yomom yomom 1428 2012-01-27 13:38 array_example
-rw-r--r-- 1 yomom yomom 990 2012-01-27 12:47 array_example.c
-rw-r--r-- 1 yomom yomom 341 2012-01-27 13:38 Makefile
-rwxr-xr-x 1 yomom yomom 7164 2012-01-27 13:38 pointers*
-rw-r--r-- 1 yomom yomom 430 2011-12-05 13:08 pointers.c
-rwxr-xr-x 1 yomom yomom 7139 2012-01-27 13:38 should_create_warning*
-rw-r--r-- 1 yomom yomom 319 2012-01-27 12:45 should_create_warning.c
-rwxr-xr-x 1 yomom yomom 7515 2012-01-27 13:38 socket-client*
-rw-r--r-- 1 yomom yomom 1472 2011-12-19 16:16 socket-client.c
-rwxr-xr-x 1 yomom yomom 7579 2012-01-27 13:38 socket-server*
-rw-r--r-- 1 yomom yomom 1150 2011-12-19 16:15 socket-server.c
I run make again, expecting it to NOP, after all, I haven't changed anything
% make
gcc -c array_example.c -o array_example
But it producted output, what gives ?
% ls -la
total 68
drwxr-xr-x 2 yomom yomom 4096 2012-01-27 13:51 ./
drwx------ 94 yomom yomom 4096 2012-01-27 13:51 ../
-rw-r--r-- 1 yomom yomom 1428 2012-01-27 13:51 array_example
-rw-r--r-- 1 yomom yomom 990 2012-01-27 12:47 array_example.c
-rw-r--r-- 1 yomom yomom 341 2012-01-27 13:38 Makefile
-rwxr-xr-x 1 yomom yomom 7164 2012-01-27 13:38 pointers*
-rw-r--r-- 1 yomom yomom 430 2011-12-05 13:08 pointers.c
-rwxr-xr-x 1 yomom yomom 7139 2012-01-27 13:38 should_create_warning*
-rw-r--r-- 1 yomom yomom 319 2012-01-27 12:45 should_create_warning.c
-rwxr-xr-x 1 yomom yomom 7515 2012-01-27 13:38 socket-client*
-rw-r--r-- 1 yomom yomom 1472 2011-12-19 16:16 socket-client.c
-rwxr-xr-x 1 yomom yomom 7579 2012-01-27 13:38 socket-server*
-rw-r--r-- 1 yomom yomom 1150 2011-12-19 16:15 socket-server.c
And it's re-compiled that array_example file, why always that one?
THE SOLUTION (Thanks to Eldar Abusalimov)
CC = gcc
#List all 'c' files by wildcard.
SOURCES=$(wildcard *.c)
#Get the 'exe' equivalent of said file
EXECUTABLES=$(SOURCES:%.c=%)
all: $(EXECUTABLES)
.PHONY: all
$(EXECUTABLES): % : %.c
$(CC) -c $< -o $#
.PHONY clean:
clean:
find ./ ! -name "*.c" -a ! -name "Makefile" -type f -delete
rebuild: clean all
FINAL SESSION
Remove all generated files
% make clean
find ./ ! -name "*.c" -a ! -name "Makefile" -type f -delete
% ls -la
total 48
drwxr-xr-x 2 yomom yomom 4096 2012-01-27 18:07 ./
drwx------ 94 yomom yomom 4096 2012-01-27 18:07 ../
-rw-r--r-- 1 yomom yomom 990 2012-01-27 17:38 array_example.c
-rw-r--r-- 1 yomom yomom 428 2012-01-27 17:38 array_of_pointers.c
-rw-r--r-- 1 yomom yomom 274 2012-01-27 17:38 const_ptr.c
-rw-r--r-- 1 yomom yomom 293 2012-01-27 17:38 function_pointers.c
-rw-r--r-- 1 yomom yomom 313 2012-01-27 18:06 Makefile
-rw-r--r-- 1 yomom yomom 430 2012-01-27 17:38 pointers.c
-rw-r--r-- 1 yomom yomom 228 2012-01-27 17:38 pointer_to_constant.c
-rw-r--r-- 1 yomom yomom 253 2012-01-27 17:38 pointer_to_pointer.c
-rw-r--r-- 1 yomom yomom 1472 2012-01-27 17:38 socket-client.c
-rw-r--r-- 1 yomom yomom 1150 2012-01-27 17:38 socket-server.c
Run make again
% make
gcc -c array_example.c -o array_example
gcc -c array_of_pointers.c -o array_of_pointers
gcc -c const_ptr.c -o const_ptr
gcc -c function_pointers.c -o function_pointers
gcc -c pointers.c -o pointers
gcc -c pointer_to_constant.c -o pointer_to_constant
gcc -c pointer_to_pointer.c -o pointer_to_pointer
gcc -c socket-client.c -o socket-client
gcc -c socket-server.c -o socket-server
Now it doesn't do silly things anymore...
% make
make: Nothing to be done for `all'.
Just remove the recipe of build: $(EXECUTABLES). First, such rule is not supposed to create a build file. Second, and answering to your question about why only array_example is being always recompiled, $(<:%=%.c) results in the first listed prerequisite, which is likely to be the first one returned by wildcard (which sorts the result in the lexicographical order, not guaranteed, but usually it does). That is,
SOURCES=$(wildcard *.c)# returns 'array_example.c pointers.c ...'.
EXECUTABLES=$(SOURCES:%.c=%)# is 'array_example pointers ...'.
build: $(EXECUTABLES)
# Here '$<' is the first item of '$(EXECUTABLES)', which is 'array_example'.
# And '$(<:%=%.c)' returns 'array_example.c'
$(CC) -c $(<:%=%.c) -o $<
Finally, don't forget to add .PHONY target.
.PHONY: build
build: $(EXECUTABLES)
Make builds all programs listed in $(EXECUTABLES) using an implicit rule to make executables from C sources. There is no need to specify anything else.
In case when the implicit rule is not OK, you can, however, override it, e.g. as follows:
$(EXECUTABLES) : % : %.c
$(CC) -o $# $<
Related
Directory content
rtetteh#PW02R9F3:~/Projects$ ls -al
total 68
drwxr-xr-x 5 rtetteh rtetteh 4096 Oct 27 13:45 .
drwxr-xr-x 18 rtetteh rtetteh 4096 Oct 27 13:45 ..
-rw-r--r-- 1 rtetteh rtetteh 12288 Sep 30 15:19 .ThreadTest.cpp.swp
drwxr-xr-x 2 rtetteh rtetteh 4096 Oct 27 13:45 .recycleBin
-rw-r--r-- 1 rtetteh rtetteh 953 Oct 27 13:44 ThreadTest.cpp
-rwxr-xr-x 1 rtetteh rtetteh 24984 Sep 30 15:14 ThreadTest_exe
drwxr-xr-x 2 rtetteh rtetteh 4096 Aug 17 18:26 hello
drwxr-xr-x 6 rtetteh rtetteh 4096 Oct 21 15:12 python-account-manager
-rwxr-xr-x 1 rtetteh rtetteh 168 Aug 19 20:23 test_pos_param.sh
Test 1
rtetteh#PW02R9F3:~/Projects$ ls -al *.sh
-rwxr-xr-x 1 rtetteh rtetteh 168 Aug 19 20:23 test_pos_param.sh
Test 2
rtetteh#PW02R9F3:~/Projects$ ls -al *.swp
ls: cannot access '*.swp': No such file or directory
Why does Test 1 work and not Test 2.
How do I get Test 2 to work i.e show files with .swp extension
Your incorrect assumption is that asterisk expands to include "." at the start of the string. Shell defaults don't make that match in regexp.
You need to specifically specify ".*.swp" in order to make the correct expansion. That also doesn't need the -a switch for ls, because you specified the "." prefix.
I have a folder such like:
drwxrwsr-x+ 1 dz33 dcistat 212 Sep 22 13:34 ./
drwxrwsr-x+ 1 dz33 dcistat 46 Sep 7 13:51 ../
-rw-rw---- 1 qg25 dcistat 542 Sep 15 13:55 createsamplelist.R
-rwxrwxr-x 1 dz33 dcistat 3717 Sep 7 14:15 Freedman-HuEx1.0v2-Analysis.Rnw*
drwxrws---+ 1 dz33 dcistat 0 Sep 22 13:34 Gao/
-rw-rw---- 1 qg25 dcistat 530 Sep 14 17:04 .log
-rwxrwxr-x 1 dz33 dcistat 154 Sep 7 13:44 Makefile*
-rwxrwx--x 1 qg25 dcistat 1191 Sep 15 09:04 pacaroma.R*
-rw-rw---- 1 qg25 dcistat 1741 Sep 14 17:23 pacaroma.Rout
-rw-rw---- 1 qg25 dcistat 4426 Sep 15 16:54 pacmeap.R
-rw-rw---- 1 qg25 dcistat 3230 Sep 14 17:15 .RData
-rw-rw---- 1 qg25 dcistat 0 Sep 14 17:04 .txt
My question is how to move all files belong to user qg25 to the directory Gao/.
find -maxdepth 1 -user qg25 -exec mv {} Gao/ \;
find /PATH_NAME -group qg25 -exec mv -t /NEW_PATH_NAME {} +
This should do the trick but I would test it on some dummy data.
I have 575 bz2 files with average size 3G and need to convert them to .gz format to make them compatible with a downstream pipeline.
$ ll -h | head
total 1.4T
drwxrws---+ 1 dz33 dcistat 24K Aug 23 09:21 ./
drwxrws---+ 1 dz33 dcistat 446 Aug 22 11:57 ../
-rw-rw---- 1 dz33 dcistat 2.0G Aug 22 11:38 DRR091550_1.fastq.bz2
-rw-rw---- 1 dz33 dcistat 2.0G Aug 22 11:38 DRR091550_2.fastq.bz2
-rw-rw---- 1 dz33 dcistat 2.0G Aug 22 11:38 DRR091551_1.fastq.bz2
-rw-rw---- 1 dz33 dcistat 2.0G Aug 22 11:38 DRR091551_2.fastq.bz2
-rw-rw---- 1 dz33 dcistat 1.9G Aug 22 11:38 DRR091552_1.fastq.bz2
-rw-rw---- 1 dz33 dcistat 1.9G Aug 22 11:38 DRR091552_2.fastq.bz2
-rw-rw---- 1 dz33 dcistat 1.8G Aug 22 11:38 DRR091553_1.fastq.bz2
$ ll | wc -l
575
For a single file I probably can do bzcat a.bz2 | gzip -c >a.gz, but I am wondering how to convert them entirely with one command or loop in bash/linux.
Do them simply and fast in parallel with GNU Parallel:
parallel --dry-run 'bzcat {} | gzip -c > {.}.gz' ::: *bz2
Sample Output
bzcat a.bz2 | gzip -c > a.gz
bzcat b.bz2 | gzip -c > b.gz
bzcat c.bz2 | gzip -c > c.gz
If you like how it looks, remove the --dry-run. Maybe add a progress meter with --bar or --progress.
In a terminal, change directory to the one containing the .bz files, then use the following command:
for f in *.bz; do bzcat "$f" | gzip -c >"${f%.*}.gz"; done
This will process each file, one at a time, and give the .gz file the name of the .bz file.
Example: DRR091550_1.fastq.bz2 will become DRR091550_1.fastq.gz.
Issue : Folders are treated as Files
My Code:
#!/bin/bash
for var in $(ls)
do
echo $var
if [ -e $var ]
then
echo "This is a file"
else
echo "This is not a file"
fi
done
echo All Done
Current Contents of the Root Folder:
-rw-r--r-- 1 stibo stibo 401 Sep 17 2015 id_rsa.pub
drwxrwxr-x 24 stibo stibo 4096 Jul 26 09:25 step
-rwxr-xr-x 1 stibo stibo 51 Jul 27 12:51 test.txt
drwxrwxrwx 2 stibo stibo 4096 Aug 2 10:32 deletionFile
-rwxrw-r-- 1 stibo stibo 225 Aug 2 12:32 deletionScript.vi
-rw-rw-r-- 1 stibo stibo 235 Aug 2 12:33 logdetails.txt
-rw-rw-r-- 1 stibo stibo 123 Aug 2 12:42 path1.txt
-rw-rw-r-- 1 stibo stibo 285 Aug 2 16:18 path2.txt
drwxrwxr-x 2 stibo stibo 4096 Aug 3 10:14 archival
-rw-rw-r-- 1 stibo stibo 0 Aug 3 13:42 ls
-rw-rw-r-- 1 stibo stibo 164732 Aug 3 14:11 messages
-rw-rw-r-- 1 stibo stibo 164732 Aug 3 14:11 wtmp
drwxrwxrwx 2 stibo stibo 4096 Aug 3 14:21 backup
-rwxrwxr-x 1 stibo stibo 160 Aug 4 15:34 newScript.vi
-rw-rw-r-- 1 stibo stibo 160 Aug 4 15:41 Code.txt
-rw-rw-r-- 1 stibo stibo 0 Aug 4 15:43 Details.txt
Where there are 4 folders and 12 files.
But when I run the script, I could see every thing is considered as a file, even if there are folders in it.
Can you please let me know where I am going wrong?
You've tested if a given entry exists with -e. Use -d for testing for directories and -f for files
I have heard the compiler strips out unused code. But when I include a libupnpx.a static library archive of 6 MB, will this make app 6 MB larger? Or is this same as including the library source code directly?
The answer is not as simple as you may think:
Let's do an experiment:
We will create a libfoo.a with the following structure:
Makefile:
LIBOUT=libfoo.a
SRC=foo1.c foo2.c foo3.c
OBJ=$(SRC:%.c=%.o)
default: test1 test2 test3
test1: $(LIBOUT)
$(CC) -o $# test1.c -L. -lfoo
test2: $(LIBOUT)
$(CC) -o $# test2.c -L. -lfoo
test3: $(LIBOUT)
$(CC) -o $# test3.c -L. -lfoo
$(LIBOUT): $(OBJ)
$(AR) ruv $(LIBOUT) $(OBJ)
foo1.c:
#include <stdio.h>
int foo1_service_a() {
printf("Performing foo1_service_a\n");
return 0;
}
foo2.c:
#include <stdio.h>
int foo2_service_a() {
printf("Performing foo2_service_a\n");
return 0;
}
foo3.c:
#include <stdio.h>
int foo3_service_a() {
printf("Performing foo3_service_a\n");
return 0;
}
int foo3_service_b() {
printf("Performing foo3_service_b\n");
return 0;
}
And three test programs:
test1.c:
int main() {
foo1_service_a();
return 0;
}
test2.c:
int main() {
foo2_service_a();
return 0;
}
test3.c:
int main() {
foo3_service_a();
return 0;
}
When we compile them we get:
-rw-r--r-- 1 masud users 285 Aug 3 16:42 Makefile
-rw-r--r-- 1 masud users 96 Aug 3 16:38 foo1.c
-rw-r--r-- 1 masud users 1496 Aug 3 16:39 foo1.o
-rw-r--r-- 1 masud users 97 Aug 3 16:39 foo2.c
-rw-r--r-- 1 masud users 1496 Aug 3 16:39 foo2.o
-rw-r--r-- 1 masud users 173 Aug 3 16:39 foo3.c
-rw-r--r-- 1 masud users 1688 Aug 3 16:40 foo3.o
-rw-r--r-- 1 masud users 5008 Aug 3 16:40 libfoo.a
-rwxr-xr-x 1 masud users 9132 Aug 3 16:43 test1
-rw-r--r-- 1 masud users 45 Aug 3 16:43 test1.c
-rwxr-xr-x 1 masud users 9132 Aug 3 16:43 test2
-rw-r--r-- 1 masud users 45 Aug 3 16:43 test2.c
-rwxr-xr-x 1 masud users 9251 Aug 3 16:43 test3
-rw-r--r-- 1 masud users 45 Aug 3 16:43 test3.c
As you can tell that each of test1 test2 and test3 have foo1.o; foo2.o and foo3.o dictating their end sizes.
So it's the archive's content division that determines the size of the end binary.
Exercise for the reader
What happens to the sizes when you strip the binaries? do test1 test2 and test3 become the same size?
Why or why not ? :)
You can generally expect that static linking and "including the library source code directly" will have equivalent impacts on the size of the resulting executable. There are be some minor cases in which this may not be strictly true, but generally speaking, they should be approximately equivalent.