I'd like to use tmutil recursively to list all of the files currently excluded from Time Machine backups. I know that I can determine this for a single file with
tmutil isexcluded /path/to/file
but I can't seem to get this to run recursively. I have tried grepping for the excluded files and outputting to a file like this:
tmutil isexcluded * | grep -i excluded >> ~/Desktop/TM-excluded.txt
but this only outputs data for the top level of the current directory. Can I use find or a similar command to feed every file/directory on the machine to tmutil isexcluded and pull out a list of the excluded files? What is the best way to structure the command?
I'm aware that most of the exclusions can be found in
/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
and that some app-specific exclusions are searchable via
sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'"
but I am looking for a way to compare the actual flags on the files to these lists.
This should do it:
find /starting/place -exec tmutil isexcluded {} + | grep -F "[Excluded]" | sed -E 's/^\[Excluded\][[:space:]]*//'
This takes advantage of the fact that tmutil allows you to pass multiple filenames, so I use + at the end of the find instead of ; then I don't have to execute a new process for every single file on your machine - which could be slow. The grep looks for the fixed (F) string [Excluded] and the sed removes the [Excluded] and following 4 spaces.
You can get all the files in any subdirectory of /path/to/master/dir with
find /path/to/master/dir -type f
Now you cannot pipe the output to tmutil, so what you can do is
find /path/to/master/dir -type f -exec tmutil isexcluded {} \;
What this does is:
We know what find /path/to/master/dir -type f does.
-exec will execute anything after it.
{} will use each file from find's output separately with tmutil isexcluded.
\ ends the -exec so the -exec knows what command should execute (in case you would want to do something else after it).
Related
So, I have this simple script which converts videos in a folder into a format which the R4DS can play.
#!/bin/bash
scr='/home/user/dpgv4/dpgv4.py';mkdir -p 'DPG_DS'
find '../Exports' -name "*1080pnornmain.mp4" -exec python3 "$scr" {} \;
The problem is, some of the videos are invalid and won't play, and I've moved those videos to a different directory inside the Exports folder. What I want to do is check to make sure the files are in a folder called new before running the python script on them, preferably within the find command. The path should look something like this:
../Exports/(anything here)/new/*1080pnornmain.mp4
Please note that (anything here) text does not indicate a single directory, it could be something like foo/bar, foo/b/ar, f/o/o/b/a/r, etc.
You cannot use -name because the search is on the path now. My first solution was:
find ./Exports -path '**/new/*1080pnornmain.mp4' -exec python3 "$scr" {} \;
But, as #dan pointed out in the comments, it is wrong because it uses the globstar wildcard (**) unnecessarily:
This checks if /new/ is somewhere in the preceding path, it doesn't have to be a direct parent.
So, the star is not enough here. Another possibility, using find only, could be this one:
find ./Exports -regex '.*/new/[^\/]*1080pnornmain.mp4' -exec python3 "$scr" {} \;
This regex matches:
any number of nested folders before new with .*/new
any character (except / to leave out further subpaths) + your filename with [^\/]*1080pnornmain.mp4
Performances could degrade given that it uses regular expressions.
Generally, instead of using the -exec option of the find command, you should opt to passing each line of find output to xargs because of the more efficient thread spawning, like:
find ./Exports -regex '.*/new/[^\/]*1080pnornmain.mp4' | xargs -0 -I '{}' python3 "$scr" '{}'
First I made a question here: Unzip a file and then display it in the console in one step
It works and helped me a lot. (please read)
Now I have a second issue. I do not have a single zipped log file but I have a lot of them in defferent folders, which I need to find first. The files have the same names. For example:
/somedir/server1/log.gz
/somedir/server2/log.gz
/somedir/server3/log.gz
and so on...
What I need is a way to:
find all the files like: find /somedir/server* -type f -name log.gz
unzip the files like: gunzip -c log.gz
use grep on the content of the files
Important! The whole should be done in one step.
I cannot first store the extracted files in the filesystem because it is a readonly filesystem. I need somehow to connect, with pipes, the output from one command to the input of the next.
Before, the log files were in text format (.txt), therefore I had not to unzip them first. In this case it was easy:
ex.
find /somedir/server* -type f -name log.txt | xargs grep "term"
Now I have to deal with zipped files. That means, after I find the files, I need first somehow do unzip them and then send the contents to grep.
With one file I do:
gunzip -p /somedir/server1/log.gz | grep term
But for multiple files I don't know how to do it. For example how to pass the output of find to gunzip and the to grep?!
Also if there is another way / "best practise" how to do that, it is welcome :)
find lets you invoke a command on the files it finds:
find /somedir/server* -type f -name log.gz -exec gunzip -c '{}' + | grep ...
From the man page:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of
invocations of the command will be much less than the number
of matched files. The command line is built in much the same
way that xargs builds its command lines. Only one instance of
{} is allowed within the command, and (when find is being
invoked from a shell) it should be quoted (for example, '{}')
to protect it from interpretation by shells. The command is
executed in the starting directory. If any invocation with
the + form returns a non-zero value as exit status, then
find returns a non-zero exit status. If find encounters an
error, this can sometimes cause an immediate exit, so some
pending commands may not be run at all. This variant of -exec
always returns true.
I wish to create a program that zips whatever file is created in the directory the find parameters specify, and run it as a background process. I heavily comment it to give a better idea of what I'm trying to achieve. I'm running this from my MacBook Pro terminal, OS X version 10.9
#!/bin/sh
#find file in directory listed below
#type f to omit directories or special files
#mtime/ctime is modified/created -0 days or less
#name is with the name given in double quotes
#asterik meaning any file name with any file extension
#use xargs to convert find sequence to a command for the line after pipe
find /Users/name/thisdirectory type f -ctime -0 -name "'*'.'*'" | xargs zip -
Maybe you're looking for this:
find /path/to/dir -type f -ctime -0 -name "*.*" | zip -# file.zip
If you read zip -h, it explains that -# is to read the filenames from standard input.
You don't need xargs here, the function to work with a list of files received from standard input is built into zip itself, similar to most compression tools like tar.
Btw, I think you want to change -ctime -0, because I don't think it can match anything this way...
I have a folder called "week1", and in that folder there are about ten other folders that all contain multiple files, including one called "submit.pdf". I would like to be able to copy all of the "submit.pdf" files into one folder, ideally using Terminal to expedite the process. I've tried cp week1/*/submit.pdf week1/ as well as cp week1/*/*.pdf week1/, but it had only been ending up copying one file. I just realized that it has been writing over each file every time which is why I'm stuck with one...is there anyway I can prevent that from happening?
You don't indicate your OS, but if you're using Gnu cp, you can use cp week1/*/submit.pdf --backup=t week/ to have it (arbitrarily) number files that already exist; but, that won't give you any real way to identify which-is-which.
You could, perhaps, do something like this:
for file in week1/*/submit.pdf; do cp "$file" "${file//\//-}"; done
… which will produce files named something like "week1-subdir-submit.pdf"
For what it's worth, the "${var/s/r}" notation means to take var, but before inserting its value, search for s (\/, meaning /, escaped because of the other special / in that expression), and replace it with r (-), to make the unique filenames.
Edit: There's actually one more / in there, to make it match multiple times, making the syntax:
"${ var / / \/ / - }"
take "var" replace every instance of / with -
find to the rescue! Rule of thumb: If you can list the files you want with find, you can copy them. So try first this:
$ cd your_folder
$ find . -type f -iname 'submit.pdf'
Some notes:
find . means "start finding from the current directory"
-type -f means "only find regular files" (i.e., not directories)
-iname 'submit.pdf' "... with case-insensitive name 'submit.dpf'". You don't need to use 'quotation', but if you want to search using wildcards, you need to. E.g.:
~ foo$ find /usr/lib -iname '*.So*'
/usr/lib/pam/pam_deny.so.2
/usr/lib/pam/pam_env.so.2
/usr/lib/pam/pam_group.so.2
...
If you want to search case-sensitive, just use -name instead of -iname.
When this works, you can copy each file by using the -exec command. exec works by letting you specify a command to use on hits. It will run the command for each file find finds, and put the name of the file in {}. You end the sequence of commands by specifying \;.
So to echo all the files, do this:
$ find . -type f -iname submit.pdf -exec echo Found file {} \;
To copy them one by one:
$ find . -type f -iname submit.pdf -exec cp {} /destination/folder \;
Hope this helps!
Would like to create a shell script that will read the contents of a folder that contains many folders on a server- it should output a list of these folders with their size and date modified if possible.
If you want to do it recursively (it's not clear to me from the question whether or not you do), you can do:
$ find /path/to/dir -type d -exec stat -c '%n: %s: %y' {} \;
(If you have a find which supports the feature, you can replace '\;' with '+')
Note that the %s gives the size of the directory, which is not the number of files in the directory, nor is it the disk usage of the files in the directory.
ls -l /path/to/folder | grep ^d
Try this find command to list sub-directories and their size (since stat command doesn't run same way on mac and Linux):
#!/bin/bash
find /your/base/dir -type d -exec du -k {} \; > sizes.txt