I'm trying to create a makefile for this project of mine, but I'm quite new to the concept. I have a makefile for each project, and a overarching make file in my main directory which I can call that communicates with all the other makefiles.
I have a few files I have named "test" to help me debug my project. By default, I want to have these test files included in my build, but with macro (ex: make TEST_FILES=false), I want to omit the files from the build.
Is there a convenient way to omit all files named "test"?
Thank you in advance!
Try something like this:
# Makefile
ifeq ($(TEST_FILES),false)
SOURCES := $(filter-out test%, $(wildcard *.txt))
else
SOURCES := $(wildcard *.txt)
endif
all:
#echo $(SOURCES)
which does this:
$ LC_ALL=C ls -nlah && \
> make all && \
> TEST_FILES=false make all
total 72K
drwx------ 2 10335 11111 4.0K May 26 15:36 .
drwxrwxrwt 585 0 0 60K May 26 15:31 ..
-rw-r--r-- 1 10335 11111 157 May 26 15:36 Makefile
-rw-r--r-- 1 10335 11111 0 May 26 14:57 bar.txt
-rw-r--r-- 1 10335 11111 0 May 26 14:57 foo.txt
-rw-r--r-- 1 10335 11111 0 May 26 14:57 qux.txt
-rw-r--r-- 1 10335 11111 0 May 26 15:30 test_bar.txt
-rw-r--r-- 1 10335 11111 0 May 26 15:30 test_foo.txt
-rw-r--r-- 1 10335 11111 0 May 26 15:27 test_qux.txt
bar.txt foo.txt qux.txt test_bar.txt test_foo.txt test_qux.txt
bar.txt foo.txt qux.txt
References:
https://www.gnu.org/software/make/manual/make.html#Text-Functions
https://www.gnu.org/software/make/manual/make.html#Wildcard-Function
https://www.gnu.org/software/make/manual/make.html#Conditional-Syntax
Related
I have directory of files called:
foo--ext1
foo--ext2
foo--ext3
...
I would like to rename them to:
foo-bar-ext1
foo-bar-ext2
foo-bar-ext3
....
How can I do this renaming in bash?
I have attempted to understand mywiki.wooledge.org/BashFAQ/030 but I can't work what should go in place of ${f%.foo}.bar"; in the first example.
So I have started with:
for f in foo--*; do echo mv -- "$f"
but what do I put next?
There are several ways to approach this.
I recommend bookmarking this page and referencing it often.
I would use this:
$: for f in foo--*; do mv "$f" "${f//foo--/foo-bar-}"; done
This uses string substitution in the current filename to construct the new name, replacing foo-- with foo-bar-.
Note the // in the replacement. This will replace every occurrence of foo-- with foo-bar- in each filename.
$: ls -l foo-*
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:32 foo-bar-ext1
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:32 foo-bar-ext2
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:32 foo-bar-ext3
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:33 foo-bar-foo-bar-etx4
Remove one of the leading slashes to make it only handle the first occurrence -
$: for f in foo--*; do mv "$f" "${f/foo--/foo-bar-}"; done
$: ls -l foo-*
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:34 foo-bar-ext1
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:34 foo-bar-ext2
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:34 foo-bar-ext3
-rw-r--r-- 1 paul 1049089 0 Jul 26 14:34 foo-bar-foo--ext4
Another simple method avoiding the loop is to use rename (from the util-linux package). There all that is needed is:
rename "foo-" "foo-bar" foo--*
You can check what will be done before actually doing the rename with the -n (no-act) and -v (verbose) options. For your example files, that would be:
$ rename -nv "foo-" "foo-bar" foo--*
`foo--ext1' -> `foo-bar-ext1'
`foo--ext2' -> `foo-bar-ext2'
`foo--ext3' -> `foo-bar-ext3'
There are two versions of rename that you will find provided in Linux distributions, the rename above from the util-linux package and perl-rename, which some Linux distros use instead which will also install as rename. Both are capable of handling the rename, but the options will be different. Check which you have with rename --version before use.
I have this small script:
#!/bin/bash
for file in "$(ls | grep -v $0)";do
cat $file > "${file}-test"
done
On this directory:
total 40
-rwxr-xr-x 1 root root 783 Dec 11 09:19 appendToLog.sh
-rwxr-xr-x 1 root root 3995 Dec 11 13:22 con2dd.py
-rwxr-xr-x 1 root root 362 Dec 11 13:26 dd.py
-rw-r--r-- 1 root root 566 Dec 11 13:26 dd.pyc
-rw-r--r-- 1 root root 18558 Dec 25 11:24 moshe.log
-rw------- 1 root root 0 Dec 11 09:20 nohup.out
-rwxr-xr-x 1 root root 88 Dec 25 11:28 task.sh
-rwxr-xr-x 1 root root 560 Dec 11 10:33 test.py
Nevermind that I can achieve that with cp, I want to understand why this exactly is producing this file:
-rw-r--r-- 1 root root 24912 Dec 25 11:28 appendToLog.sh?con2dd.py?dd.py?dd.pyc?moshe.log?nohup.out?task.sh?test.py-test
And nothing else.
The problem is parsing output of ls is just wrong (see Why you shouldn't parse the output of ls(1), filenames in unix can have almost any special characters including whitespace, newlines, commas, pipe symbols. Its because you've quoted the output of ls in one construct, you have a single list of all the files concatenated as one string in the value of "${file}-test" which is quite not what you wanted to do.
Also notice how ls sometimes garbles your filename data (in our case, it turned the \n character in between the words into a ? question mark (could indicate a character that cannot be displayed).
Just use the glob expansion in bash to list the files and do actions on them.
for f in *; do
[[ -e $f ]] || continue
...
done
That said, You could probably have some non-printable characters on end of lines (eg. CRLF imported rom Windows)
Run cat -A scriptname it'll show you all characters in your script. Then, you can convert to unix-like format running dos2unix scriptname.
Below are the files in my local server
-rw-r----- 1 root root 0 Sep 25 15:03 one.xml
-rw-r----- 1 root root 0 Sep 25 15:03 two.xml
-rw-r----- 1 root root 0 Sep 25 15:03 data.csv
-rw-r----- 1 root root 0 Sep 25 15:03 free.png
-rw-r----- 1 root root 0 Sep 25 15:04 loaded.jpeg
I know transfering files of same extension as below
/usr/bin/sftp ${user}#${HostName} <<EOF
cd $InputPath
lcd $OutputPath
put *.csv
exit
EOF
scp ${InputPath}/*.csv ${user}#${HostName}:$OutputPath
But every time when i run the script i need to transfer only files with xml and jpeg extensions. ssh,scp,SFTP can be used. Any hep please??
You can try with something like this:
*.{jpeg,xml}
All together:
scp ${InputPath}/*.{jpeg,xml} ${user}#${HostName}:$OutputPath
Test
$ ls
a.jpeg a.png a.xml
$ ls *{jpeg,xml}
a.jpeg a.xml
I have a directory with the following files
➜ hrrep git:(master) ✗ ls -l
total 1000
drwxrwxrwx 22 zmjones staff 748 May 28 23:28 data
-rw-rw-rw- 1 zmjones staff 7180 May 20 16:17 data.R
drwxrwxrwx 9 zmjones staff 306 May 28 23:38 figures
-rw-r--r-- 1 zmjones staff 85841 May 28 23:23 hill_jones_hr.bib
-rw-r--r-- 1 zmjones staff 29193 May 28 22:23 hill_jones_hr.tex
-rw-r--r-- 1 zmjones staff 572 May 28 23:46 makefile
-rw-rw-rw- 1 zmjones staff 9588 May 28 22:47 models.R
drwxrwxrwx 3 zmjones staff 102 May 28 23:28 papers
drwxrwxrwx 9 zmjones staff 306 May 28 23:28 tex
-rw-r--r-- 1 zmjones staff 1483 May 20 12:58 un_data.py
These files are described by a makefile (using GNU Make 3.81)
all: ./data/cat_un_data.csv ./data/rep.csv ./figures/*.png ./hj-hr.pdf
./data/cat_un_data.csv: un_data.py #refreshing one will refresh all
python un_data.py
./data/rep.csv: data.R ./data/cat_un_data.csv
R CMD BATCH --no-save --no-restore data.R
./figures/*.png: models.R ./data/rep.csv
R CMD BATCH --no-save --no-restore models.R
TEXMCD := pdflatex -interaction=batchmode
hill_jones_hr.pdf: hill_jones_hr.tex ./figures/*.png
$(TEXCMD) $<
]bibtex *.aux
$(TEXCMD) $<
$(TEXCMD) $<
find . | egrep ".*((\.(aux|log|blg|bbl|out|Rout|Rhistory|DS_Store))|~)$$" | xargs rm
rm -rf auto
The makefile seems to work fine until it gets to the hll_jones_hr.pdf target, where it fails with an error:
hill_jones_hr.tex
make: hill_jones_hr.tex: No such file or directory
make: *** [hill_jones_hr.pdf] Error 1
I don't understand what the problem is. The document compiles fine when I execute pdflatex and bibtex manually. I tried adding the ignore error flag to no avail. I presume this is some stupid make error that I am making. This folder is in Dropbox if that makes any difference. I also tried adding the ./ prefix to both the .tex file and the .pdf target; the error was the same.
The problem is that the $(TEXCMD) variable is not defined, so your command:
$(TEXCMD) $<
is expanding to just $< or just the filename, which is not a legal command by itself.
use ./hill_jones_hr.tex instead of hill_jones_hr.tex, and try again.
What does the command cp $1/. $2 do? I know cp is used for copying from source(stored in variable $1) to destination(stored in variable $2). I am just confused with the /. used along with the variable. Can someone please help me understand this?
The command:
$ cp -R $1/. $2
copies contents of directory pointed by $1 to the directory $2.
Without -R switch this command would fail both when $1 is a file or directory.
In general, . points to the current directory. You can see that by comparing inode's shown by ls:
$ mkdir test
$ ls -ali
9525121 drwxr-xr-x 3 IU wheel 102 23 mar 12:31 .
771046 drwxrwxrwt 21 root wheel 714 23 mar 12:30 ..
9525312 drwxr-xr-x 2 IU wheel 68 23 mar 12:31 test
$ cd test
$ ls -ali
9525312 drwxr-xr-x 2 IU wheel 68 23 mar 12:31 .
9525121 drwxr-xr-x 3 IU wheel 102 23 mar 12:31 ..
Note that inode 9525312 points to test when viewed from the parent directory, and points to . when viewed from inside the test directory.