I'm trying to use LFTP in my GitLab continuous integration setup so I can mirror JSON files with my destination. However, I'd like to only mirror on a set of folders using a wildcard, but I cannot seem to get this working.
I tried this mirror command configuration in LFTP, but this results in a "No such file or directory" error. I assume I'm parsing the wildcard wrong somehow.
What I tried: lftp -c "set sftp:auto-confirm true; open sftp://$DEVELOPMENT_DEPLOY_USER:$DEVELOPMENT_DEPLOY_PASSWORD#$DEVELOPMENT_DEPLOY_HOST:$DEVELOPMENT_DEPLOY_PORT; mirror -Rev ./somefolder_* $DEVELOPMENT_DESTINATION_FOLDER --ignore-time --parallel=10 --exclude .* --exclude .*/ --include ./*.json"
Results in:
/home/gitlab-runner/builds/82ffc821/0/somegroup/someproject/somefolder_*: No such file or directory
I'm probably missing something obvious. Would appreciate any help.
Maybe old but find that in man :
Include and exclude options can be specified multiple times. It means
that a file or direc‐
tory would be mirrored if it matches an include and does not match to excludes after the
include, or does not match anything and the first check is exclude. Directories are matched
with a slash appended.
So first it will execute "include" argument, lastly "exclude" with is " --exclude .* --exclude .*/". After that glob, no file math to mirror. Use "--verbose" to check what files are touch by lftp
Related
My example folder layout is the following:
/games/scripts/myscript.sh
/games/doom/doom.sh
/games/quake/quake.sh
/books/scripts/manuscript.txt
/books/text/hamlet.txt
/books/text/shakespeare.txt
I want to include /books/scripts, but not copy /games/scripts/
In my rsync script, I tell it to loop over a file containing the following
directories.txt
/games
/books
In my exclude file, it contains the following
exclude.txt
/games/scripts/
My command is as follows:
/usr/bin/rsync -zaxvrlog --stats \
--exclude-from="exclude.txt" \
--delete --rsh="/usr/bin/ssh -q" \
"kevin"#"myhost.internal":"${directory}" "${directory_stripped}"
When it runs, neither directory has the /scripts/
I understand that rsync uses relative directories for the exclude, but I want it to run absolute if possible. I have tried formatting the rsync differently thinking it might change the behaviour, but it still ignores the /scripts/ path on both directories
I want to work localy on some programm, which I want to test and run on a remote server. The only files I am editing are the *.hpp and *.cpp in the src and the include directory.
For that I tried this rsync command, to only upload the necessarry source files:
rsync --dry-run -av --exclude '*' --include 'src/*.cpp' --include 'include/*.hpp' Programm/ user#remote:/home/user/Programm
But for some reason no files are commited to the server after some local changes.
Any hints appreciated!
Thank you
Here's excerpt from the rsync man page, which tackles your exact problem.
Note that, when using the --recursive (-r) option (which is implied by
-a), every subdir component of every path is visited left to right, with each directory having a chance for exclusion before its content.
In this way include/exclude patterns are applied recursively to the
pathname of each node in the filesystem's tree (those inside the
transfer). The exclude patterns short-circuit the directory traversal
stage as rsync finds the files to send.
For instance, to include "/foo/bar/baz", the directories "/foo" and
"/foo/bar" must not be excluded. Excluding one of those parent
directories prevents the examination of its content, cutting off
rsync's recursion into those paths and rendering the include for
"/foo/bar/baz" ineffectual (since rsync can't match something it never
sees in the cut-off section of the directory hierarchy).
The concept path exclusion is particularly important when using a
trailing '*' rule. For instance, this won't work:
+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *
This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or
"some/path" directories. One solution is to ask for all directories in
the hierarchy to be included by using a single rule: "+ */" (put it
somewhere before the "- *" rule), and perhaps use the
--prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance,
this set of rules works fine:
+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *
I backup my data with rsync and would like to exclude a specific filetype in only one directory (and its subdirectories). For example I have:
$ ls Source/
Folder1/a.tar
Folder1/b.dat
Folder2/c.tar
Folder2/d.dat
Folder2/Subfolder3/e.tar
Folder2/Subfolder3/f.dat
Folder2/Subfolder3/g.pdf
Now I would like to sync all files except for the .tar files in Folder2 and its subfolder. At the end it should look like this:
$ ls Target/
Folder1/a.tar
Folder1/b.dat
Folder2/d.dat
Folder2/Subfolder3/f.dat
Folder2/Subfolder3/g.pdf
Does someone know how to do that? I played around with the --exclude option, but without luck.
rsync manual says
INCLUDE/EXCLUDE PATTERN RULES
...
o use ’**’ to match anything, including slashes.
so you can do
rsync -a --exclude='Folder2/**.tar' Source/ Target
Note this is different from bash's globstar option where you would use Folder/**/*.tar.
Right now I'm successfully running:
rsync -uvma --include="*/" --include="*.css" --exclude="*" $spec_dir $css_spec_dir
In a shell script which copies all of the files in the source directory, that are .css files, into a target directory.
I want to do the same for HTML files, but only where they are in a subfolder with the name 'template'.
So I'm in directory ~/foo, and I want to rsync where the --include="*/" only matches on subfolders with the name 'template'. So ~/foo/bar/template/baz/somefile.html would match, and so would ~foo/bar/baz/qux/template/someotherfile.html, but NOT ~/foo/bar/thirdfile.html
Although it looks a little bit strange, this works for me:
rsync -uvma --include="*/" --include="*/template/*/*.html" --include="*/template/*.html" --include="template/*.html" --include="template/*/*.html" --exclude="*" $spec_dir $html_spec_dir
This one works for me:
rsync -umva --include="**/templates/**/*.html" --exclude="*.html" source/ target
Were you looking for **? Here you have to be careful about choosing your exclude pattern, * won't work as it matches directories on the way. If rsync finds foo/templates/some.html, it will first copy foo, then foo/templates and then foo/templates/some.html, but before it gets there * already matched foo and nothing gets copied.
Here's what worked:
rsync -uvma --include="*/" --include="templates/**.html" --exclude="*" $html_all_dir $html_dir
My guess is, your format and mine probably accomplish the same thing. I know I tried about 20 different patterns before this one, and this is the only one that worked properly. I don't think I tried your format though :)
I've been trying to figure out how to backup the contents of my file server's (CentOS via smb) user's folder, ignoring certain file types and directories. It seems like this should be easy, but I'm not getting anywhere on figuring out how to ignore multiple directories.
I'd like to ignore the following:
all files and directories starting with a . or a _
all MS Office temp files (eg ~$*)
lock files (eg .lock)
I've tried a bunch of different combinations of the --exclude flag, but can't get any to work right.
This is the command that makes the most sense, but it's not excluding anything:
s3cmd sync --dry-run --verbose --delete-removed --exclude '.*' '_*' '~$*' '*.lock' /home/user-folder s3://bucket-name/
If you are already using .gitignore, you can do something like
s3cmd sync --exclude '.git/*' --exclude-from .gitignore <local_dir> s3://<bucket>/
as stated in this blog post and confirmed by the documentation for --exclude-from from the official docs (Ctrl+F and search for "exclude-from").
It works great, with one minor drawback: if you're excluding a folder within .gitignore, you must exclude its contents also, or s3cmd will grab its contents. However, this is easy, you can just add a line like <foldername>/* inside the .gitignore and everything will be ok.
EDIT:
Well, better than this. Set up a .s3ignore file and just refer to it from the sync command:
s3cmd sync --exclude-from .s3ignore <local_dir> s3://<bucket>/
.s3ignore example:
.git
.git/*
.gitignore
node_modules
node_modules/*
*.swo
*.swp
*.pyo
*.pyc
I've do something similar. The key is to use --exclude before each pattern you want to match:
s3cmd -v --recursive --exclude ".ts" --exclude ".aac" --exclude "/thumbnails" put /var/www/folder s3://bucket/
Also I managed to use .ts without the wildcard symbol and it worked in my case!
Other answers mention passing --exclude <pattern> for each pattern, and packing all patterns into a file to pass with --exclude-from <file>
Using regex:
You can also pack all patterns into a regular expression and pass it with the --rexclude option:
The Regex pattern for the question above: ".^\.*|._*|.~$*|.*.lock"
s3cmd sync --dry-run --verbose --delete-removed --rexclude ".^\.*|._*|.~$*|.*.lock" /home/user-folder s3://bucket-name/