After some reading and trying rsync copy over only certain types of files using include option I can't get seem to get it to work.
I run the following command:
rsync -zarv -vvv -e ssh --prune-empty-dirs --delete --include="*/" --include="*.csv"
--include="*.hdf5" --include="*.pickle" --include="*.tar.gz" --include="*.bin"
--include="*.zip" --include="*.npz" --exclude="*" . user#host.com:/rsync
But at the target it backups any file I have in the directory and subdirectories. delete-before and delete-after does not delete files like .txt or .py. I have also tried the --exclude="*" before the extension includes but I am running 2.6.9 so it should be after as far as I have understood it.
Deleting files on the host machine will just sync them again for whatever reason I don't know.
Your command looks fine, although try using --delete-excluded instead of --delete.
--delete-excluded - also delete excluded files from destination dirs
It should eliminate any files that are --excluded and not --included on the destination.
Sorry to have bothered. This was a bash issue and not a command issue.
As I was using the command:
exec $COMMAND
instead of
eval $COMMAND
This made god knows what for error but executing it manually (after print) and correctly in bash made it work. Deleting items still seems flaky but that I can try some.
I like to create tar-files to distribute some scripts using bash.
For every script certain configuration-files and libraries (or toolboxes) are needed,
e.g. a script called CheckTool.py needs Checks.ini, CheckToolbox.py and CommontToolbox.py to run, which are stored in specific folders on my harddisk and need to be copied in the same manner on the users harddisk.
I can create a tarfile manually for each script, but i like to have it more simple.
For this i have the idea to define a list of all needed files and their pathes for a specific script and read this in a bashscript, which creates the tar file.
I started with:
#!/bin/bash
while read line
do
echo "$line"
done < $1
Which is reading the files and pathes. In my example the lines are:
./CheckTools/CheckMesh.bs
./Configs/CheckMesh.ini
./Toolboxes/CommonToolbox.bs
./Toolboxes/CheckToolbox.bs
My question is how do I have to organize the data to make a tar file with the specified files using bash?
Or is there someone having a better idea?
No need for a complicated script, use option -T of tar. Every file listed in there will be added to the tar file:
-T, --files-from FILE
get names to extract or create from FILE
So your script becomes:
#!/bin/bash
tar -cvpf something.tar -T listoffiles.txt
listoffiles.txt format is super easy, one file per line. You might want to put full path to ensure you get the right files:
./CheckTools/CheckMesh.bs
./Configs/CheckMesh.ini
./Toolboxes/CommonToolbox.bs
./Toolboxes/CheckToolbox.bs
You can add tar commands to the script as needed, or you could loop on the list files, from that point on, your imagination is the limit!
I am using git bash on Windows - that is git for Windows via the integrated bash. Apparently it uses the MINGW/MSYS underpinning. (Update from #VonC: It now uses msys2 since msysgit is obsolete since Q4 2015.)
So there are already a lot of MSYS tools installed - from awk to zcat. However I miss the man command and zip to compress multiple files into a zip file (unzip exists!).
Where from can I install them? I do not want to install another copy of the MINGW system! Any way just to add some pre-compiled tools to the git bash installation?
Here's another, slightly different, set of instructions to install zip for git bash on windows:
Navigate to this sourceforge page
Download zip-3.0-bin.zip
In the zipped file, in the bin folder, find the file zip.exe.
Extract the file zip.exe to your mingw64 bin folder (for me: C:\Program Files\Git\mingw64\bin)
Navigate to to this sourceforge page
Download bzip2-1.0.5-bin.zip
In the zipped file, in the bin folder, find the file bzip2.dll
Extract bzip2.dll to your mingw64\bin folder (same folder as above: C:\Program Files\Git\mingw64\bin)
7-zip can be added to gitbash as follows:
Install 7-zip on windows.
add 7-zip folder (C:\Program Files\7-Zip) to PATH
On gitbash exp: export PATH=$PATH:"C:\Program Files\7-Zip" (temporary)
On Windows, adding PATH like image below (permanent)
duplicate a copy of 7z.exe to be zip.exe
reopen gitbash again. done!
This way, it works on my laptop.
If you skip step 3. you still can call zip command as 7z instead of zip
Conclusion: Gitbash is running base on windows Path, I think you can run any command that you have added to your Windows PATH.
2016: The zip command can be installed from GoW (Gnu On Windows). man is not provided (too big).
It is to note, however, that if you only want to add the zip command from GoW, still the whole GoW system has to be downloaded and installed. Then you can delete the other commands from the bin directory, however make sure to keep the needed dlls in the directory.
Update 2021: tar/zip are by default installed on Windows 10.
7-zip based solutions are available below.
git-archive, is prepared without any installation, can create zip-archive.
mkdir workrepo
cd workrepo
git init
cp -r [target_file_or_dir] .
git add .
git commit -m commit
git archive -o ../myarchive.zip #
cd ..
rm -rf workrepo
Following script may be usable:
zip.sh foo.zip target_file_or_dir
#!/usr/bin/bash
set -eu
unset workdir
onexit() {
if [ -n ${workdir-} ]; then
rm -rf "$workdir"
fi
}
trap onexit EXIT
workdir=$(mktemp --tmpdir -d gitzip.XXXXXX)
cp -r "$2" "$workdir"
pushd "$workdir"
git init
git config --local user.email "zip#example.com"
git config --local user.name "zip"
git add .
git commit -m "commit for zip"
popd
git archive --format=zip -o "$1" --remote="$workdir" HEAD
I am so glad to share my experience on this issue that I haven't known for two years since the first day I played with Groovy. My method needs to have git for Windows installed in Windows OS.
These steps are for installing 7z command-line utility, which behaves a bit differently from zip:
Download and install 7-Zip from its official website. By default, it is installed under the directory /c/Program Files/7-Zip in Windows 10 as my case.
Run git Bash under Administrator privilege and navigate to the directory /c/Program Files/Git/mingw64/bin, you can run the command ln -s "/c/Program Files/7-Zip/7z.exe" 7z.exe
I am pretty sure it could help you a lot. Trust me!
On Windows, you can use tar instead of zip.
tar -a -c -f output.zip myfile.txt
which is same as,
zip output.zip myfile.txt
no need to install any external zip tool.
I use choco as my Windows Package Manager.
I install 7zip with choco using PowerShell (you must be admin to use Choco)
PS > choco install 7zip.install
Open another gitbash Terminal and locate the 7z.exe executable
$ which 7z
/c/ProgramData/chocolatey/bin/7z
Do a straight copy of 7z.exe to zip.exe and voila
$ cp /c/ProgramData/chocolatey/bin/7z.exe /c/ProgramData/chocolatey/bin/zip.exe
You can mimic a small subset of man behavior in the shell by mapping man <command> to <command> --help | less
Unfortunately, on my machine bash aliases won't add flags to positional arguments, it will try to run the flag as a command and fail (alias man="$1 --help" doesn't work).
And a function called man() is not allowed!
Luckily a combination of bash functions and aliases can achieve this mapping. Put the code below in your ~/.bashrc (create one if it is not there). Don't forget to source ~/.bashrc.
# man command workaround: alias can't pass flags, but can't name function man
m() {
"$1" --help | less
}
alias man="m"
It doesn't get you the full man page, but if all you're looking for is basic info on a command and its flags, this might be all you need.
You can install individual GNU tools from http://gnuwin32.sourceforge.net/packages.html such as zip.
Then add "/c/Program Files (x86)/GnuWin32/bin" to PATH in your startup script like .profile, .bash_profile, .bashrc, etc.
Here are the steps you can follow.
Go to the following link
https://sourceforge.net/projects/gnuwin32/files/
Find out whatever command you are missing
Here I need zip and bzip2 for zip command. Because zip command relies on bzip2.dll to run. Otherwise you will get error “error while loading shared libraries: ?: cannot open shared object file: No such file or directory”.
Unzip the downloaded files
Here I am downloading “zip-3.0-bin.zip” for “zip.exe” and “bzip2-1.0.5-bin.zip” for “bzip2.dll” in the bin folder. /bin/.exe
Copy the command exe file into git-bash folder
Here I am copying “zip.exe” and “bzip2.dll” to \Git\usr\bin.
Reference Link
https://ranxing.wordpress.com/2016/12/13/add-zip-into-git-bash-on-windows/
ln -s /mingw64/bin/ziptool.exe /usr/bin/zip
steps to install SDKMAN on windows
Run Windows Terminal in Admin rights. open git bash inside. (Ctrl + Shift + 4)
winget install -e --id GnuWin32.Zip
mkdir ~/bin
cp /usr/bin/unzip ~/bin/zip
curl -s "https://beta.sdkman.io" | bash
source "/c/Users/ajink/.sdkman/bin/sdkman-init.sh"
sdk selfupdate force
After you can install Java like this.
sdk install java 17.0.2-open
Done ! :)
In msys2, I restored the functionality of git help <command> by installing man-db:
|prompt> pacman -Syu man-db
|prompt> git help archive
For zip functionality, I also use git archive (similar to yukihane's answer).
Here's yet another 7-Zip option that I didn't notice:
Create a script named zip:
$ vi ~/bin/zip
Reference 7z specifying the add command followed by the args:
#!/bin/bash
/c/Progra~1/7-Zip/7z.exe a "$#"
Finally make it executable
$ chmod ugo+x ~/bin/zip
This helped to make a ytt build script happy.
+ zip ytt-lambda-website.zip main ytt
7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
Scanning the drive:
2 files, 29035805 bytes (28 MiB)
Creating archive: ytt-lambda-website.zip
Add new data to archive: 2 files, 29035805 bytes (28 MiB)
Though this question as been answered quite thoroughly in regards to man there is one alternative to zipping that has not been highlighted here yet. #Zartc brought to my attention that there is a zip compression utility built-in: ziptool. In trying to use it however I found out it is no where near a drop-in replacement and you need to specify each individual file and folder. So I dug into the docs and experimented until I had a bash-function that can do all the heavy lifting and can be used very similar to a basic zip -qrf name * compression call:
zipWithZiptool() {
# Docs: https://libzip.org/documentation/ziptool.html
targetFilePath="$1"
shift
args=() # collect all args in an array so spaces are handled correctly
while IFS=$'\n\r' read -r line; do
if [[ -d "$line" ]]; then
args+=("add_dir" "$line") # Add a single directory by name
else
# add_file <pathInZip> <pathToFile> <startIndex> <length>
args+=("add_file" "$line" "$line" 0 -1)
fi
done <<< "$(find "$#")" # call find with every arg to return a recursive list of files and dirs
ziptool $targetFilePath "${args[#]}" # quotation is important for handling file names with spaces
}
You can then for example zip the contents of the current directory by calling it like this:
zipWithZiptool "my.zip" *
If you are willing to install CygWin also, you can add the CygWin path to your GitBash path, and if zip is there, it will work. e.g. add
PATH=$PATH:/c/cygwin/bin
export PATH
to your .bashrc; NOTE: I would put it at the end of the path as shown, not the beginning.
Since CygWin has a UI-based installer, it's easy to add or remove applications like zip or man.
You can figure out the windows paths of each by saying
`cygpath -w /bin`
in each respective shell.
Regarding zip, you can use a following perl script to pack files:
#!/usr/bin/perl
use IO::Compress::Zip qw(:all);
$z = shift;
zip [ #ARGV ] => $z or die "Cannot create zip file: $ZipError\n";
If you make it executable, name it zip, and put it in your $PATH, you can run it like this:
zip archive.zip files...
however it will not work for directories. There is no need to install anything, as perl and all required modules are already there in the Git for Windows installation.
Regarding man, at least for git there is a documentation invoked like this:
git option --help
it will open in your default browser.
Here is my experience, I cant run and exe or .msi files in my laptop. so downloaded filed from https://github.com/bmatzelle/gow/wiki > go to download Now and Downloaded Source Code (Zip) and unzipped this file in a folder and updated path variable with folder name.
This worked out for me.
If you want to zip files without needing to install any additional tools on Windows, in a way that works both on git bash and on other *nix systems, you might be able to use perl.
Per Josip Medved's blog, the following script creates an .epub (which is a zip file), and includes a filter for stripping src/ from the files added to the zip:
perl -e '
use strict;
use warnings;
use autodie;
use IO::Compress::Zip qw(:all);
zip [
"src/mimetype",
<"src/META-INF/*.*">,
<"src/OEBPS/*.*">,
<"src/OEBPS/chapters/*.*">
] => "bin/book.epub",
FilterName => sub { s[^src/][] },
Zip64 => 0,
or die "Zip failed: $ZipError\n";
'
install zip
https://gnuwin32.sourceforge.net/packages/zip.htm
copy zip.exe and bzip2.dll from C:\Program Files (x86)\GnuWin32\bin to C:\Program Files\Git\mingw64\bin
reopen git-bash
Solutions for me were just to install zip on my terminal(bash):
$ sudo apt-get update
$ sudo apt-get install zip unzip
After I finish writing scripts on my local machine, I need to copy them in the cluster to execute the codes. For example, I want to copy all the matlab files in my current directory in a directory at the server id#server.
Can anyone help to write a very basic Makefile to fulfill this purpose?
Thanks a lot!
John
Here is an adaptation of Jens's answer, together with my answer here, that takes advantage of the capabilities of Make to only copy across those files that have been modified since the last time you copied the files to the server. That way, if you have hundreds of .m files and you modify one of them, you won't copy all of them across to the server.
It makes use of an empty hidden file, .last_push, that serves only to record (through its own timestamp) the time at which we last copied files to the server.
FILES = $(shell find . -name *.m)
SCP = scp id#server:path/relative/to/your/serverhomedir
LAST_PUSH = .last_push
.PHONY : push
push : $(LAST_PUSH)
$(LAST_PUSH) : $(FILES)
$(SCP) $?
touch $(LAST_PUSH)
Run this with make or make push. The key is the variable $?, which is populated with the list of all prerequisites that are newer than the target - in this case, the list of .m files that have been modified more recently than the last push.
How do you copy files to the server? Assuming you have ssh/scp available:
FILES = file1 file2 *.matlab
copy:
scp $(FILES) id#server:path/relative/to/your/serverhomedir
Run with
$ make copy
As a shell script, it could look like this:
#!/bin/sh
set -- file1 file2 *.matlab
scp "$#" id#server:path/relative/to/your/serverhomedir
Don't forget to chmod u+x yourscript.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Delete all but the most recent X files in bash
The scenerio is as follows:
Two tar files will be created in one directory per day, but I need only the latest two files, so how to delete the other files automatically each day?
Is i possible to write this script using pure shell commands, and not with high level language such as perl, python or ruby...
This issue is a bit similar to FTP - Only want to keep latest 10 files - delete LRU and how to delete all files except the latest three in a folder
but mine also needs to test if a tar file is corrupt
If newer tar file is corrupt, I would not keep it, but reserve the older ones, so what the script should be like?
Obviously this is solvable using a two pass approach.
Detect and remove corrupt tar files.
Remove all but the latest N files as described in the referenced question.
The first pass could be performed with something along
for t in *.tar; do
if program_that_checks_tar_file_for_integrity $t; then
: # OK
else
rm $t
fi
done