Below is a simple bash program. It takes file types as command line arguments and it queries the current directory and prints the files of the type specified.
I would like to be able to query two different file types and therefore need two boolean expressions to represent this.
Below is my code for querying just one file type
#!/bin/bash
for x in $(ls *$1); do
echo $x;
done
Now what I would like to be able to do is (in pseudocode)
command line args fileName .sh .c
for x in (current directory files of *.sh) OR (in current directory files of *.c) do
print .sh files
print.c files
done
I've tried using || and I get syntax errors I can not find any evidence of being able to use || for two expressions in for loop.
I've tried using two nested for loops but they do not work and yield errors.
Is there any way I can accomplish this using the same for loop system.
Thank you.
Sounds like you want something like:
for extension in "$#"; do
printf 'Files ending in %s:\n' "$extension"
printf '%s\n' *"$extension"
done
Loop through all arguments passed to the script and print all files ending in each extension + a newline character.
Note that printf is a much more useful tool than echo, as it allows you to control the format of each thing is prints.
ls doesn't do anything useful either here; it is the shell which expands the * to the list of files matching the pattern.
Related
I try to use automator for renaming multiple files
I got this far:
the first variable must be inside the Exiftool command line
in this case I selected 2 files, but that could be 1 or 100 files
how do I make this happen? is it possible to start from array key 1 instead of array key 0 for the filenames?
The standard way to do this is to store $1 in a variable, then use shift to remove it from the argument list, and then use "$#" to get all of the remaining arguments (i.e. the original "$2" "$3" "$4" ...) Something like this:
RenameTo="$1"
shift
echo "New name: $RenameTo"
echo "files:" "$#"
I'm not sure exactly what you're trying to to with exiftool, so I won't try to give that full command.
Note that the double-quotes aren't required in zsh, but they make this portable to POSIX-compliant shells. Also, echo isn't a very good way to see what a command would do, because it looses the distinction between spaces within an argument (e.g. spaces in the new name, or within a filename) and spaces between arguments (e.g. between the filenamess in a list of them).
I have 40 csv files that I need to edit. 20 have matching format and the names only differ by one character, e.g., docA.csv, docB.csv, etc. The other 20 also match and are named pair_docA.csv, pair_docB.csv, etc.
I have the code written to edit and combine docA.csv and pair_docA.csv, but I'm struggling writing a loop that calls both the above files, edits them, and combines them under the name combinedA.csv, then goes on the the next pair.
Can anyone help my rudimentary bash scripting? Here's what I have thus far. I've tried in a single for loop, and now I'm trying in 2 (probably 3) for loops. I'd prefer to keep it in a single loop.
set -x
DIR=/path/to/file/location
for file in `ls $DIR/doc?.csv`
do
#code to edit the doc*.csv files ie $file
done
for pairdoc in `ls $DIR/pair_doc?.csv`
do
#code to edit the piar_doc*.csv files ie $pairdoc
done
#still need to combine the files. I have the join written for a single iteration,
#but how do I loop the code to save each join as a different file corresponding
#to combined*.csv
Something along these lines:
#!/bin/bash
dir=/path/to/file/location
cd "$dir" || exit
for file in doc?.csv; do
pair=pair_$file
# "${file#doc}" deletes the prefix "doc"
combined=combined_${file#doc}
cat "$file" "$pair" >> "$combined"
done
ls, on principle, shouldn't be used in a shell script in order to iterate over the files. It is intended to be used interactively and nearly never needed within a script. Also, all-capitalized variable names shouldn't be used as ordinary variables, since they may collide with internal shell variables or environment variables.
Below is a version without changing the directory.
#!/bin/bash
dir=/path/to/file/location
for file in "$dir/"doc?.csv; do
basename=${file#"$dir/"}
pair=$dir/pair_$basename
combined=$dir/combined_${basename#doc}
cat "$file" "$pair" >> "$combined"
done
This might work for you (GNU parallel):
parallel cat {1} {2} \> join_{1}_{2} ::: doc{A..T}.csv :::+ pair_doc{A..T}.csv
Change the cat commands to your chosen commands where {1} represents the docX.csv files and {2} represents the pair_docX.csv file.
N.B. X represents the letters A thru T
I am working with a program that combines individuals files, and I am incorporating this program into a BASH pipeline that I'm putting together. The program requires a flag for each file, like so:
program -V file_1.g.vcf -V file_2.g.vcf -V file_3.g.vcf -O combined_output.g.vcf
In order to allow the script to work with any number of samples, I would like to read the individual files names within a directory, and expand the path for each file after a '-V' flag.
I have tried adding the file paths to a variable with the following, but have not had success with proper expansion:
GVCFS=('-V' `ls gvcfs/*.g.vcf`)
Any help is greatly appreciated!
You can do this by using a loop to populate an array with the options:
options=()
for file in gvcfs/*.g.vcf; do # Don't parse ls, just use a direct wildcard expression
options+=(-V "${file##*/}") # If you want the full path, leave off ##*/
done
program "${options[#]}" -O combined_output.g.vcf
printf can help:
options=( $(printf -- "-V %s " gvcfs/*.g.vcf ) )
Though this will not deal gracefully with whitespace in filenames.
Also consider realpath to generate absolute filenames.
I'm writing a small piece of code that checks for .mov files in a specific folder over 4gb and writes it to a log.txt file by name (without an extension). I'm then reading the names into a while loop line by line which signals some archiving and copying commands.
Consider a file named abcdefg.mov (new) and a corresponding folder somewhere else named abcdefg_20180525 (<-*underscore timestamp) that also contains a file named abcedfg.mov (old).
When reading in the filename from the log.txt, I strip the extension to store the variable "abcdefg" ($in1) and i'm using that variable to locate a folder elsewhere that contains that matching string at the beginning.
My problem is with how the mv command seems to support a wild card in the "source" string, but not in the "destination" string.
For example i can write;
mv -f /Volumes/Myshare/SourceVideo/$in1*/$in1.mov /Volumes/Myshare/Archive
However a wildcard on the destination doesn't work in the same way. For example;
mv -f /Volumes/Myshare/Processed/$in1.mov Volumes/Myshare/SourceVideo/$in1*/$in1.mov
Is there an easy fix here that doesn't involve using another method?
Cheers for any help.
mv accepts a single destination path. Suppose that $in1 is abcdfg, and that $in1* expands to abcdefg_20180525 and abcdefg_20180526. Then the command
mv -f /dir1/$in1 /dir2/$in1*/$in1.mov
will be equivalent to:
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir2/abcdefg_20180525/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
Moreover, because the destination file is the same in all three cases, the first two files will be overwritten by the third.
You should create a precise list and do a precise copy instead of using wild cards.
This is what I would probably do, generate a list of results in a file with FULL path information, then read those results in another function. I could have used arrays but I wanted to keep it simple. At the bottom of this script is a function call to scan for files of EXT mp4 (case insensitive) then writes the results to a file in tmp. then the script reads the results from that file in another function and performs some operation (mv etc.). Note, if functions are confusing , you can just remove the function name { } and name calls and it becomes a normal script again. functions are really handy, learn to love them!
#!/usr/bin/env bash
readonly SIZE_CHECK_LIMIT_MB="10M"
readonly FOLDER="/tmp"
readonly DESTINATION_FOLDER="/tmp/archive"
readonly SAVE_LIST_FILE="/tmp/$(basename $0)-save-list.txt"
readonly EXT="mp4"
readonly CASE="-iname" #change to -name for exact ext type upper/lower
function find_files_too_large() {
> ${SAVE_LIST_FILE}
find "${FOLDER}" -maxdepth 1 -type f "${CASE}" "*.${EXT}" -size +${SIZE_CHECK_LIMIT_MB} -print0 | while IFS= read -r -d $'\0' line ; do
echo "FOUND => $line"
echo "$line" >> ${SAVE_LIST_FILE}
done
}
function archive_large_files() {
local read_file="${SAVE_LIST_FILE}"
local write_folder="$DESTINATION_FOLDER"
if [ ! -s "${read_file}" ] || [ ! -f "${read_file}" ] ;then
echo "No work to be done ... "
return
fi
while IFS= read -r line ;do
echo "mv $line $write_folder" ;sleep 1
done < "${read_file}"
}
# MAIN (this is where the script starts) We just call two functions.
find_files_too_large
archive_large_files
it might be easier, i think, to change the filenames to the folder name initially. So abcdefg.mov would be abcdefg_timestamp.mov. I can always strip the timestamp from the filename easy enough after its copied to the right location. I was hoping i had a small syntax issue but i think there is no easy way of doing what i thought i could...
I think you have a basic misunderstanding of how wildcards work here. The mv command doesn't support wildcards at all; the shell expands all wildcards into lists of matching files before they get passed to the mv command as wildcards. Furthermore, the mv command doesn't know if the list of arguments it got came from wildcards or not, and the shell doesn't know anything about what the command is going to do with them. For instance, if you run the command grep *, the grep command just gets a list of names of files in the current directory as arguments, and will treat the first of them as a regex pattern ('cause that's what the first argument to grep is) to search the rest of the files for. If you ran mv * (note: don't do this!), it will interpret all but the last filename as sources, and the last one as a destination.
I think there's another source of confusion as well: when the shell expands a string containing a wildcard, it tries to match the entire thing to existing files and/or directories. So when you use Volumes/Myshare/SourceVideo/$in1*/$in1.mov, it looks for an already-existing file in a matching directory; AIUI the file isn't there yet, there's no match. What it does in that case is pass the raw (unexpanded) wildcard-containing string to mv as an argument, which looks for that exact name, doesn't find it, and gives you an error.
(BTW, should there be a "/" at the front of that pattern? I assume so below.)
If I understand the situation correctly, you might be able to use this:
mv -f /Volumes/Myshare/Processed/$in1.mov /Volumes/Myshare/SourceVideo/$in1*/
Since the filename isn't supplied in the second string, it doesn't look for existing files by that name, just directories with the right prefix; mv will automatically retain the filename from the source.
However, I'll echo #Sergio's warning about chaos from multiple matches. In this case, it won't overwrite files (well, it might, but for other reasons), but if it gets multiple matching target directories it'll move all but the last one into the last one (along with the file you meant to move). You say you're 100% certain this won't be a problem, but in my experience that means that there's at least a 50% chance that something you'd never have thought of will go ahead and make it happen anyway. For instance, is it possible that $in1 could wind up empty, or contain a space, or...?
Speaking of spaces, I'd also recommend double-quoting all variable references. You want the variables inside double-quotes, but the wildcards outside them (or they won't be expanded), like this:
mv -f "/Volumes/Myshare/Processed/$in1.mov" "/Volumes/Myshare/SourceVideo/$in1"*/
I don't really know that much about bash scripts OR imagemagick, but I am attempting to create a script in which you can give some sort of regexp matching pattern for a list of images and then process those into new files that have a given filename prefix.
for example given the following dir listing:
allfiles01.jpg allfiles02.jpg
allfiles03.jpg
i would like to call the script like so:
./resisemany.sh allfiles*.jpg 30 newnames*.jpg
the end result of this would be that you get a bunch of new files with newnames, the numbers match up,
so far what i have is:
IMAGELIST=$1
RESIEZFACTOR=$2
NUMIMGS=length($IMAGELIST)
for(i=0; i<NUMIMGS; i++)
convert $IMAGELIST[i] -filter bessel -resize . RESIZEFACTOR . % myfile.JPG
Thanks for any help...
The parts that I obviously need help with are
1. how to give a bash script matching criteria that it understands
2. how to use the $2 without having it match the 2nd item in the image list
3. how to get the length of the image list
4. how to create a proper for loop in such a case
5. how to do proper text replacement for a shell command whereby you are appending items as i allude to.
jml
Probably the way a standard program would work would be to take an "in" filename pattern and an "out" filename pattern and perform the operation on each file in the current directory that matches the "in" pattern, substituting appropriate parts into the "out" pattern. This is pretty easy if you have a hard-coded pattern, when you can write one-off commands like
for infile in *.jpg; do convert $infile -filter bessel -resize 30% ${infile//allfiles/newnames}; done
In order to make a script that will do this with any pattern, though, you need something more complicated because your filename transformation might be something more complicated than just replacing one part with another. Unfortunately Bash doesn't really give you a way to identify what part of the filename matched a specific part of the pattern, so you'd have to use a more capable regular expression engine, like sed for example:
#!/bin/bash
inpattern=$1
factor=$2
outpattern=$3
for infile in *; do
outfile=$(echo $infile | sed -n "s/$inpattern/$outpattern/p")
test -z $outfile && continue
convert $infile -filter bessel -resize $factor% $outfile
done
That could be invoked as
./resizemany.sh 'allfiles\(.*\).jpg' 30 'newnames\1.jpg'
(note the single quotes!) and it would resize allfiles1.jpg to newnames1.jpg, etc. But then you'd wind up basically having to learn sed's regular expression syntax to specify your in and out patterns. (It's not that bad, really)
You could eliminate the regex problem if you make a folder of all the files to be processed, and then run something like:
for img in `ls *.jpg`
do
convert $img -filter bessel -resize 30% processed-$img
done
Then, if you need to rename them all later, you could do something like:
ls | nl -nrz -w2 | while read a b; do mv "$b" newfilename.$a.jpg; done;
Also, If you are doing a batch process of the same operation, you might see if using mogrify might help (imagemagik's method for converting multiple files). Like the above example, it's always good to make a copy of the folder, and then run any processing so you don't destroy your original files.
Your script should be called using a syntax such as:
./resizemany.sh -r 30 -n newnames -o allfiles allfiles*.jpg
and use getopts to process the options. What you may not be aware of is that the shell expands the file glob before the script gets it so the way you had your arguments your script would never be able to distinguish the filenames from the other parameters.
Output files will be named using the rename script often found on systems with Perl installed. A file named "allfiles03.jpg" will be output as "newname03.jpg".
#!/bin/bash
options=":r:n:o:"
while getopts $options option
do
case $option in
n)
newnamepattern=$OPTARG
;;
o)
oldnamepattern=$OPTARG
;;
r)
resizefacor=$OPTARG
;;
\?)
echo "Invalid option"
exit 1
esac
done
# a check to see if any options are missing should be performed (not implemented)
shift $((OPTIND - 1))
# now all that's left will be treated as filenames
for file
do
convert (input options) "$file" -resize $resizefactor (output options) "${file}.out"
rename "s/$old/$new/;s/\.out$//" "${file}.out"
done
This is untested (obviously since most of the arguments to convert are missing).
Parameter validation such as range checks, missing required options and others are left as exercises for further development. Also absent are checks for successful completion of one step before continuing to the next one. Also issues such as locations of files and name collisions and others are not addressed.