My example folder layout is the following:
/games/scripts/myscript.sh
/games/doom/doom.sh
/games/quake/quake.sh
/books/scripts/manuscript.txt
/books/text/hamlet.txt
/books/text/shakespeare.txt
I want to include /books/scripts, but not copy /games/scripts/
In my rsync script, I tell it to loop over a file containing the following
directories.txt
/games
/books
In my exclude file, it contains the following
exclude.txt
/games/scripts/
My command is as follows:
/usr/bin/rsync -zaxvrlog --stats \
--exclude-from="exclude.txt" \
--delete --rsh="/usr/bin/ssh -q" \
"kevin"#"myhost.internal":"${directory}" "${directory_stripped}"
When it runs, neither directory has the /scripts/
I understand that rsync uses relative directories for the exclude, but I want it to run absolute if possible. I have tried formatting the rsync differently thinking it might change the behaviour, but it still ignores the /scripts/ path on both directories
Related
I have an rsync command like this:
rsync -avz --exclude-from 'exclude-list.txt' -r /source/ destination
the exclude-list.txt looks like this:
.git
src
config.js
.gitlab-ci.yml
.gitignore
local-*
All the listed files and directories are in the same directory as the exclude-list.txt file.
So, the .git, src and config.js are skipped over and not copied (like I want), but the rest of the items are still copied. What's wrong with my syntax? (local-* is supposed to be a wildcard to skip all files beginning with local-).
I'm trying to use LFTP in my GitLab continuous integration setup so I can mirror JSON files with my destination. However, I'd like to only mirror on a set of folders using a wildcard, but I cannot seem to get this working.
I tried this mirror command configuration in LFTP, but this results in a "No such file or directory" error. I assume I'm parsing the wildcard wrong somehow.
What I tried: lftp -c "set sftp:auto-confirm true; open sftp://$DEVELOPMENT_DEPLOY_USER:$DEVELOPMENT_DEPLOY_PASSWORD#$DEVELOPMENT_DEPLOY_HOST:$DEVELOPMENT_DEPLOY_PORT; mirror -Rev ./somefolder_* $DEVELOPMENT_DESTINATION_FOLDER --ignore-time --parallel=10 --exclude .* --exclude .*/ --include ./*.json"
Results in:
/home/gitlab-runner/builds/82ffc821/0/somegroup/someproject/somefolder_*: No such file or directory
I'm probably missing something obvious. Would appreciate any help.
Maybe old but find that in man :
Include and exclude options can be specified multiple times. It means
that a file or direc‐
tory would be mirrored if it matches an include and does not match to excludes after the
include, or does not match anything and the first check is exclude. Directories are matched
with a slash appended.
So first it will execute "include" argument, lastly "exclude" with is " --exclude .* --exclude .*/". After that glob, no file math to mirror. Use "--verbose" to check what files are touch by lftp
I backup my data with rsync and would like to exclude a specific filetype in only one directory (and its subdirectories). For example I have:
$ ls Source/
Folder1/a.tar
Folder1/b.dat
Folder2/c.tar
Folder2/d.dat
Folder2/Subfolder3/e.tar
Folder2/Subfolder3/f.dat
Folder2/Subfolder3/g.pdf
Now I would like to sync all files except for the .tar files in Folder2 and its subfolder. At the end it should look like this:
$ ls Target/
Folder1/a.tar
Folder1/b.dat
Folder2/d.dat
Folder2/Subfolder3/f.dat
Folder2/Subfolder3/g.pdf
Does someone know how to do that? I played around with the --exclude option, but without luck.
rsync manual says
INCLUDE/EXCLUDE PATTERN RULES
...
o use ’**’ to match anything, including slashes.
so you can do
rsync -a --exclude='Folder2/**.tar' Source/ Target
Note this is different from bash's globstar option where you would use Folder/**/*.tar.
I am trying to use include and exclude options in rsync to copy a directory structure, excluding most but not all of the subdirectories, based on a pattern in the directory names. But, it isn't working. It is trying to copy everything over instead of just the subfolders I want. Is my syntax wrong?
I have tried:
rsync -am --include='*/*/*MPRAGE*/' --exclude='*' /parent_directory/ /destination
Also:
rsync -am --include='*/' --include='*/*/*MPRAGE*/' --exclude='*' /parent/ /dest
MPRAGE is the pattern that is in the name of each folder I want copied. But these folders are three levels deep in the structure, and I want to keep the well-organized directory structure intact for these folders I want copied.
Thanks in advance for any tips.
I've been trying to figure out how to backup the contents of my file server's (CentOS via smb) user's folder, ignoring certain file types and directories. It seems like this should be easy, but I'm not getting anywhere on figuring out how to ignore multiple directories.
I'd like to ignore the following:
all files and directories starting with a . or a _
all MS Office temp files (eg ~$*)
lock files (eg .lock)
I've tried a bunch of different combinations of the --exclude flag, but can't get any to work right.
This is the command that makes the most sense, but it's not excluding anything:
s3cmd sync --dry-run --verbose --delete-removed --exclude '.*' '_*' '~$*' '*.lock' /home/user-folder s3://bucket-name/
If you are already using .gitignore, you can do something like
s3cmd sync --exclude '.git/*' --exclude-from .gitignore <local_dir> s3://<bucket>/
as stated in this blog post and confirmed by the documentation for --exclude-from from the official docs (Ctrl+F and search for "exclude-from").
It works great, with one minor drawback: if you're excluding a folder within .gitignore, you must exclude its contents also, or s3cmd will grab its contents. However, this is easy, you can just add a line like <foldername>/* inside the .gitignore and everything will be ok.
EDIT:
Well, better than this. Set up a .s3ignore file and just refer to it from the sync command:
s3cmd sync --exclude-from .s3ignore <local_dir> s3://<bucket>/
.s3ignore example:
.git
.git/*
.gitignore
node_modules
node_modules/*
*.swo
*.swp
*.pyo
*.pyc
I've do something similar. The key is to use --exclude before each pattern you want to match:
s3cmd -v --recursive --exclude ".ts" --exclude ".aac" --exclude "/thumbnails" put /var/www/folder s3://bucket/
Also I managed to use .ts without the wildcard symbol and it worked in my case!
Other answers mention passing --exclude <pattern> for each pattern, and packing all patterns into a file to pass with --exclude-from <file>
Using regex:
You can also pack all patterns into a regular expression and pass it with the --rexclude option:
The Regex pattern for the question above: ".^\.*|._*|.~$*|.*.lock"
s3cmd sync --dry-run --verbose --delete-removed --rexclude ".^\.*|._*|.~$*|.*.lock" /home/user-folder s3://bucket-name/