If a file has new line symbol in it, and I use find or ls, it shows as "?" symbol in a file name. But when I use | to do anything more with it, it splits the string and messes up everything. How do I deal with it?
Don't pipe the results of ls at all. It is, as you see, unpredictable. Use find instead:
find -maxdepth 1 -exec command {} \;
{} represents the file name.
Alternatively you can also use glob expressions. The results of a glob expression are not subject of word splitting, meaning it is safe if they contain newlines or spaces:
for file in ./* ; do
command "$file"
done
The results of that commands can then be used in another pipe.
Related
My problem is pretty straightforward to understand.
I have images.txt which is a list of line separated URLs pointing to .jpg files separated as follows:
https://region.URL.com/files/2/2f/dir/2533x1946_IMG.jpg?Tag=2&Policy=BLAH__&Signature=BLAH7-BLAH-BLAH__&Key-Pair-Id=BLAH
I'm able to successfully download with wget -i but they are formatted like 2533x1946_IMG.jpg?BLAH_BLAH_BLAH_BLAH when I need them named like this instead: 2533x1946_IMG.jpg
Note that I've already tried the popular solutions to no avail (see below), so I'm thinking more along the lines of a solution that would involved sed, grep and awk
wget --content-disposition-i images.txt
wget --trust-server-names -i images.txt
wget --metalink-over-http --trust-server-names --content-disposition -i images.txt
wget --trust-server-names --content-disposition -i images.txt
and more iterations like this based on those three flags....
I'd ideally like to do it with one command, but even if it's a matter of downloading the files as-is and later doing a recursive command that renames them to the 2533x1946_IMG.jpg format is acceptable too.
1) you can use rename in ONE liner to rename all files
rename -n 's/[?].*//' *_BLAH
rename uses the next sintax 's/selectedString/whatYouChange/'
rename uses regex to find all your files and also to rename using a loop. Because your name is very specific, you can select it very easy. you're going to select the char ? and because in regex it has a special meaning youre going to put that in brackets [ ]. end result [?].
-n argument it's to show you what is going to change and not make the changes until you remove it. delete -n and changes will be applied.
.* is for selecting everything after the char ?, so BLAH_BLAH_BLAH_BLAH
// is for remove what you select, because there are NOT words OR anything in here.
*_BLAH is for selecting all files that end with _BLAH, you could use * but maybe you have other files, folders in that same place, so it's safer this way.
output
find . \
-name '*[?]*' \
-exec bash -c $'for f; do mv -- "$f" "${f%%\'?\'*}"; done' _ {} +
Why *[?]*? That prevents the ? from being treated as a single-character wildcard, and instead ensures that it only matches itself.
Why $'...\'?\'...'? The $'...' ANSI-C-style string quoting form allows backslash escapes to be able to specify literal ' characters even inside a single-quoted string.
Why bash -c '...' _ {} +? Unlike approaches that substitute the filenames that were found into code to be executed, this keeps those names out-of-band from the code, preventing shell injection attacks via hostile filenames. The _ placeholder fills in $0, so subsequent arguments become $1 and onword; and the for loop iterates over them (for f; do is the same as for f in "$#"; do).
What does ${f%%'?'*} do? This paramater expansion expands $f with the longest possible string matching the glob-style/fnmatch pattern '?'* removed from the end.
I am trying to copy a .nii file (Gabor3.nii) path to a variable but even though the file is found by the find command, I can't copy the path to the variable.
find . -type f -name "*.nii"
Data= '/$PWD/"*.nii"'
output:
./Gabor3.nii
./hello.sh: line 21: /$PWD/"*.nii": No such file or directory
What went wrong
You show that you're using:
Data= '/$PWD/"*.nii"'
The space means that the Data= parts sets an environment variable $Data to an empty string, and then attempts to run '/$PWD/"*.nii"'. The single quotes mean that what is between them is not expanded, and you don't have a directory /$PWD (that's a directory name of $, P, W, D in the root directory), so the script "*.nii" isn't found in it, hence the error message.
Using arrays
OK; that's what's wrong. What's right?
You have a couple of options. The most reliable is to use an array assignment and shell expansion:
Data=( "$PWD"/*.nii )
The parentheses (note the absence of spaces before the ( — that's crucial) makes it an array assignment. Using shell globbing gives a list of names, preserving spaces etc in the names correctly. Using double quotes around "$PWD" ensures that the expansion is correct even if there are spaces in the current directory name.
You can find out how many files there are in the list with:
echo "${#Data[#]}"
You can iterate over the list of file names with:
for file in "${Data[#]}"
do
echo "File is [$file]"
ls -l "$file"
done
Note that variable references must be in double quotes for names with spaces to work correctly. The "${Data[#]}" notation has parallels with "$#", which also preserves spaces in the arguments to the command. There is a "${Data[*]}" variant which behaves analogously to "$*", and is of similarly limited value.
If you're worried that there might not be any files with the extension, then use shopt -s nullglob to expand the globbing expression into an empty list rather than the unexpanded expression which is the historical default. You can unset the option with shopt -u nullglob if necessary.
Alternatives
Alternatives involve things like using command substitution Data=$(ls "$PWD"/*.nii), but this is vastly inferior to using an array unless neither the path in $PWD nor the file names contain any spaces, tabs, newlines. If there is no white space in the names, it works OK; you can iterate over:
for file in $Data
do
echo "No white space [$file]"
ls -l "$file"
done
but this is altogether less satisfactory if there are (or might be) any white space characters around.
You can use command substitution:
Data=$(find . -type f -name "*.nii" -print -quit)
To prevent multiline output, the -quit option stop searching after the first file was found(unless you're sure only one file will be found or you want to process multiple files).
The syntax to do what you seem to be trying to do with:
Data= '/$PWD/"*.nii"'
would be:
Data="$(ls "$PWD"/*.nii)"
Not saying it's the best approach for whatever you want to do next of course, it's probably not...
I'm running macOS and looking for a way to quickly sort thousands of jpg files. I need to create folders based on part of filenames and then move those files into it.
Simply, I want to put these files:
x_not_relevant_part_of_name.jpg
x_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
Into these folders:
x
y
Keep in mind that length of "x" and "y" part of name may be different.
Is there an automatic solution for that in maxOS?
I've tried using Automator and Terminal but i'm not a programmer so I haven't done well.
I would back up the files first to somewhere safe in case it all goes wrong. Then I would install homebrew and then install rename with:
brew install rename
Then you can do what you want with this:
rename --dry-run -p 's|(^[^_]*)|$1/$1|' *.jpg
If that looks correct, remove the --dry-run and run it again.
Let's look at that command.
--dry-run means just say what the command would do without actually doing anything
-p means create any intermediate paths (i.e. directories) as necessary
's|...|' I will explain in a moment
*.jpg means to run the command on all JPG files.
The funny bit in single quotes is actually a substitution, in its simplest form it is s|a|b| which means substitute thing a with b. In this particular case, the a is caret (^) which means start of filename and then [^_]* means any number of things that are not underscores. As I have surrounded that with parentheses, I can refer back to it in the b part as $1 since it is the first thing in parentheses in a. The b part means "whatever was before the underscore" followed by a slash and "whatever was before the underscore again".
Using find with bash Parameter Substitution in Terminal would likely work:
find . -type f -name "*jpg" -maxdepth 1 -exec bash -c 'mkdir -p "${0%%_*}"' {} \; \
-exec bash -c 'mv "$0" "${0%%_*}"' {} \;
This uses bash Parameter Substitution with find to recursively create directories (if they don't already exist) using the prefix of any filenames matching jpg. It takes the characters before the first underscore (_), then moves the matching files into the appropriate directory. To use the command simply cd into the directory you would like to organize. Keep in mind that without using the maxdepth option running the command multiple times can produce more folders; limit the "depth" at which the command can operate using the maxdepth option.
${parameter%word}
${parameter%%word}
The word is expanded to produce a pattern just as in filename expansion. If the pattern matches a trailing portion of the expanded
value of parameter, then the result of the expansion is the value of
parameter with the shortest matching pattern (the ‘%’ case) or the
longest matching pattern (the ‘%%’ case) deleted.
↳ GNU Bash : Shell Parameter Expansion
I have searched looking for the right solution. I have found some close examples.Bash script to replace spaces in file names
But what I'm looking for is how to replace multiple .dots in current DIRECTORY/SUBDIRECTORY names, then replace multiple .dots in FILENAMES excluding the *.extention "recursively".
This example is close but not right:
find . -maxdepth 2 -name "*.*" -execdir rename 's/./-/g' "{}" \;
another example but not right either:
for f in *; do mv "$f" "${f//./-}"; done
So
dir.dir.dir/dir.dir.dir/file.file.file.ext
Would become
dir-dir-dir/dir-dir-dir/file-file-file.ext
You can assign to a variable and pipe like this:
x="dir.dir.dir/dir.dir.dir/file.file.file.ext"
echo "$(echo "Result: ${x%.*}" | sed 's/\./_/g').${x##*.}"
Result: dir_dir_dir/dir_dir_dir/file_file_file.ext
You have to escape . in regular expressions (such as the ones used for rename, because by default it has the special meaning of "any single character". So the replacement statement at least should be s/\./-/g.
You should not quote {} in find commands.
You will need two find commands, since you want to replace all dots in directory names, but keep the last dot in filenames.
You are searching for filenames which contain spaces (* *). Is that intentional?
I have a bit of a problem. I'm trying to write a script that looks for all files of a given type (php) in a given directory. If it doesn't find it, it goes through all the sub-directories in the parent directory. If it finds it then, it performs a given operation and breaks.
Here is what I have so far:
function findPHP(){
declare -a FILES
FILES=$(find ./ -type f -name \*.php)
for f in $FILES
do
echo "Processing $f file..."
# take action on each file.
done
}
Any ideas?
When using $(...) the shell doesn't treat any characters within the parentheses as special (unlike the similar backquote syntax), which suggests that the * does not need to be escaped. The find command is probably literally seeing \*. Try removing the backslash.