I'm attempting to write a program that moves zipped files that arrive in a directory, unzips them and then outputs the contents.
#!/bin/bash
shopt -s extglob
echo "Press [CTRL+C] to stop.."
#begin loop
while :
do
#search folder test_source for files and append names to array
queue+=($(ls /home/ec2-user/glutton/test_source | egrep 'test[0-9]{1}.gz'))
for i in $queue; do
#move file in test_source to temp folder
mv /home/ec2-user/glutton/test_source/${queue[i]} /home/ec2-user/glutton/temp
#unzip file
gunzip /home/ec2-user/glutton/temp/${queue[i]}
#add new file name to variable unzipped
unzipped=($(ls /home/ec2-user/glutton/temp | egrep 'test[0-9]{1}'))
cat temp/$unzipped
#Test for successful run
exit_status=$?
if [ $exit_status -eq 1 ]; then
#If fail move file to bank and output
mv /home/ec2-user/glutton/temp/$unzipped /home/ec2-user/glutton/logs/bank
echo "Error! $unzipped moved to /home/ec2-user/glutton/logs/bank"
#write to error.log
echo "moved ${queue[i]} to bank due to error" >> /home/ec2-user/glutton/logs/error.log
else
#If success delete file
rm /home/ec2-user/glutton/temp/$unzipped
fi
#wipe variable
unset unzipped
i=$i+1
done
#wipe array
unset queue
i=0
#go to top of loop
done
This has worked pretty well up until I added the unzipping feature and now my program outputs this error when attempting to move the .gz file:
./glutton.sh: line 11: test0.gz: syntax error: invalid arithmetic operator (error token is ".gz")
When I run the first part of my script in the command line it seems to work perfectly, but doesn't when I run it on its own, I'm pretty confused.
Your main issue is that when you iterate an array like you are doing, you get the first item of the array, not the index. So in your case, $i is not a number, it is the filename (i.e. test1.gz) and it will only see the first file. The cleanest way I have seen to iterate the items in an array would be for i in "${arrayName[#]}".
Also, using '{1}' in your regex is redundant, the character class will already match only 1 character if you don't specify a modifier.
It shouldn't matter depending on the contents of your 'temp' folder, but I would be more specific on your egreps too, if you add -x then it will have to match the whole string. As it is currently, a file called 'not_a_test1' would match.
Related
Hello I am trying to get all files with Jane's name to a separate file called oldFiles.txt. In a directory called "data" I am reading from a list of file names from a file called list.txt, from which I put all the file names containing the name Jane into the files variable. Then I'm trying to test the files variable with the files in list.txt to ensure they are in the file system, then append the all the files containing jane to the oldFiles.txt file(which will be in the scripts directory), after it tests to make sure the item within the files variable passes.
#!/bin/bash
> oldFiles.txt
files= grep " jane " ../data/list.txt | cut -d' ' -f 3
if test -e ~data/$files; then
for file in $files; do
if test -e ~/scripts/$file; then
echo $file>> oldFiles.txt
else
echo "no files"
fi
done
fi
The above code gets the desired files and displays them correctly, as well as creates the oldFiles.txt file, but when I open the file after running the script I find that nothing was appended to the file. I tried changing the file assignment to a pointer instead files= grep " jane " ../data/list.txt | cut -d' ' -f 3 ---> files=$(grep " jane " ../data/list.txt) to see if that would help by just capturing raw data to write to file, but then the error comes up "too many arguments on line 5" which is the 1st if test statement. The only way I get the script to work semi-properly is when I do ./findJane.sh > oldFiles.txt on the shell command line, which is me essentially manually creating the file. How would I go about this so that I create oldFiles.txt and append to the oldFiles.txt all within the script?
The biggest problem you have is matching names like "jane" or "Jane's", etc. while not matching "Janes". grep provides the options -i (case insensitive match) and -w (whole-word match) which can tailor your search to what you appear to want without having to use the kludge (" jane ") of appending spaces before an after your search term. (to properly do that you would use [[:space:]]jane[[:space:]])
You also have the problem of what is your "script dir" if you call your script from a directory other than the one containing your script, such as calling your script from your $HOME directory with bash script/findJane.sh. In that case your script will attempt to append to $HOME/oldFiles.txt. The positional parameter $0 always contains the full pathname to the current script being run, so you can capture the script directory no matter where you call the script from with:
dirname "$0"
You are using bash, so store all the filenames resulting from your grep command in an array, not some general variable (especially since your use of " jane " suggests that your filenames contain whitespace)
You can make your script much more flexible if you take the information of your input file (e.g list.txt), the term to search for (e.g. "jane"), the location where to check for existence of the files (e.g. $HOME/data) and the output filename to append the names to (e.g. "oldFile.txt") as command line [positonal] parameters. You can give each default values so it behaves as you currently desire without providing any arguments.
Even with the additional scripting flexibility of taking the command line arguments, the script actually has fewer lines simply filling an array using mapfile (synonymous with readarray) and then looping over the contents of the array. You also avoid the additional subshell for dirname with a simple parameter expansion and test whether the path component is empty -- to replace with '.', up to you.
If I've understood your goal correctly, you can put all the pieces together with:
#!/bin/bash
# positional parameters
src="${1:-../data/list.txt}" # 1st param - input (default: ../data/list.txt)
term="${2:-jane}" # 2nd param - search term (default: jane)
data="${3:-$HOME/data}" # 3rd param - file location (defaut: ../data)
outfn="${4:-oldFiles.txt}" # 4th param - output (default: oldFiles.txt)
# save the path to the current script in script
script="$(dirname "$0")"
# if outfn not given, prepend path to script to outfn to output
# in script directory (if script called from elsewhere)
[ -z "$4" ] && outfn="$script/$outfn"
# split names w/term into array
# using the -iw option for case-insensitive whole-word match
mapfile -t files < <(grep -iw "$term" "$src" | cut -d' ' -f 3)
# loop over files array
for ((i=0; i<${#files[#]}; i++)); do
# test existence of file in data directory, redirect name to outfn
[ -e "$data/${files[i]}" ] && printf "%s\n" "${files[i]}" >> "$outfn"
done
(note: test expression and [ expression ] are synonymous, use what you like, though you may find [ expression ] a bit more readable)
(further note: "Janes" being plural is not considered the same as the singular -- adjust the grep expression as desired)
Example Use/Output
As was pointed out in the comment, without a sample of your input file, we cannot provide an exact test to confirm your desired behavior.
Let me know if you have questions.
As far as I can tell, this is what you're going for. This is totally a community effort based on the comments, catching your bugs. Obviously credit to Mark and Jetchisel for finding most of the issues. Notable changes:
Fixed $files to use command substitution
Fixed path to data/$file, assuming you have a directory at ~/data full of files
Fixed the test to not test for a string of files, but just the single file (also using -f to make sure it's a regular file)
Using double brackets — you could also use double quotes instead, but you explicitly have a Bash shebang so there's no harm in using Bash syntax
Adding a second message about not matching files, because there are two possible cases there; you may need to adapt depending on the output you're looking for
Removed the initial empty redirection — if you need to ensure that the file is clear before the rest of the script, then it should be added back, but if not, it's not doing any useful work
Changed the shebang to make sure you're using the user's preferred Bash, and added set -e because you should always add set -e
#!/usr/bin/env bash
set -e
files=$(grep " jane " ../data/list.txt | cut -d' ' -f 3)
for file in $files; do
if [[ -f $HOME/data/$file ]]; then
if [[ -f $HOME/scripts/$file ]]; then
echo "$file" >> oldFiles.txt
else
echo "no matching file"
fi
else
echo "no files"
fi
done
Trying to write a bash script to copy a large number of files from an external drive into separate directories based on a subject id.
I've included the script I've written below.
I get the following error:
cat: /Volumes/Seagate: No such file or directory
cat: Backup: No such file or directory
cat: Plus: No such file or directory
cat: Drive/Subject_List.txt: No such file or directory
When I try to copy a single file at a time using the terminal, it copies using the exact command I've put in this script. I'm not sure why it's not recognizing the directory when I try and use it in the script below. Any help is greatly appreciated!
#!/bin/bash
#A bash script to copy the structural and functional files into the HCP_Entropy folder
#subject list
SUBJECT_LIST="/Volumes/Seagate/Backup/Plus/Drive/Subject_List.txt
for j in $(cat ${SUBJECT_LIST}); do
echo ${j}
cp /Volumes/Seagate\ Backup\ Plus\ Drive/HCP_DATA/Structural/{j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz /Users/myname/Box/HCP_Entropy/BEN/${j}/anat
done
the line
$SUBJECT_LIST=/Volumes/Seagate\ Backup\ Plus\ Drive/Subject_List.txt
is bogus.
to assign values to a variable, you must not add the $ specifier.
a token starting with $ will be expanded, so $SUBJECT_LIST=... will first be expanded to =... (since you haven't assigned anything to the SUBJECT_LIST variable yet it is empty).
the proper way would be:
SUBJECT_LIST="/Volumes/Seagate Backup Plus Drive/Subject_List.txt"
(this uses quotes instead of escaping each space, which i find much more readable)
you also need to quote variables in case they contain spaces, else they might be interpreted by the command (cp) as multiple arguments.
for j in $(cat "${SUBJECT_LIST}"); do
# ...
done
and of course, you should check whether the source file actually exists, just like the destination directory.
indir="/Volumes/Seagate Backup Plus Drive"
SUBJECT_LIST="${indir}/Subject_List.txt"
cat "${SUBJECT_LIST}" | while read j; do
infile="${indir}/HCP_DATA/Structural/${j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz"
outdir="/Users/myname/Box/HCP_Entropy/BEN/${j}/anat"
mkdir -p "${outdir}"
if [ -e "${infile}" ]; then
cp -v "${infile}" "${outdir}"
else
echo "missing file ${infile}" 1>&2
fi
done
I have bash unix script to move all files from folder to another folder.
But, it's always give me binary operation expected when there's more than 1 file in source folder.
My Code goes like this
#Variable
NAS_MYTH=/cd/myth_extfile/shyn/
NAS_SHYN=/cd/shyn/MYTH/*
#Script
echo " Move file SHYN to NAS MYTH : $NAS_MYTH"
if [ -f $NAS_SHYN ]; then
mv $NAS_SHYN $NAS_MYTH
echo "Success: moving files to $NAS_MYTH"
else
echo "Success : No files need to be moved"
fi
I'm using if method to check if there is data there, so it'll success error shown
But, this code i use is fine when in folder MYTH there is just one data, but when it's more than one data i'll get error output like this
Move file SHYN to NAS MYTH : /cd/myth_extfile/shyn/
./mv_shyn_files.sh: line 52: [: /cd/shyn/MYTH/FAQ - MYTH.pdf: binary operator expected
Sucess : No file need to be move
Can anyone help me? Thanks.
UPDATE
It's all solved, im using this code now
shopt -s nullglob
NAS_MYTH=/cd/myth_extfile/shyn/
NAS_SHYN=(/cd/shyn/MYTH/*)
echo "Checking if there is a file, please wait"
if [ "${#NAS_SHYN[#]}" != 0 ]; then
echo "There is ${#NAS_SHYN[#]} file"
mv -f "${NAS_SHYN[#]}" "$NAS_MYTH"
echo "Success : ${#NAS_SHYN[#]} file moved"
else
echo "Success : There is no file need to be moved"
fi
Thanks guys for helping me :D
NAS_SHYN=/cd/shyn/MYTH/* assigns a literal * to NASH_SHYN. But when you write $NAS_SHYN without quotes then the * is expanded into a list of files which are then expanended into a list of words by splitting the file names at whitespace. In your case, it seems like there is only one file, namely FAQ - MYTH.pdf. However, that file has spaces in it, therefore bash splits your filename into multiple words causing the error.
Instead, use an array and quote your variables.
shopt -s nullglob
NAS_SHYN=(/cd/shyn/MYTH/*)
if [ "${#NAS_SHYN[#]}" != 1 ]; then
echo "expected one match but found ${#NAS_SHYN[#]}"
exit 1
fi
echo "The only match is ${#NAS_SHYN[0]}"
if [ -f "${NAS_SHYN[0]}" ]; then
…
By the way ALLCAPS variables are by convention special or environment variables. To avoid accidental name collisions use lowercase variables instead.
So I have a conversion program (works through command line) that takes a data file and converts it into a new format while also putting it in a folder with various subfolders. I wanted to create a script that would check for duplicates before executing this conversion program.
So far, I have
#!/bin/bash
for subj in `ls <directory to data files>`
do
subj_name=$subj
subj_path=<directory to data files>/$subj_name #I need this for my program, can ignore
cd <directory with output folders>
if [ -e “$subj” ]; then
echo “This file already exists” #This will restart the loop and move to the next file
else
echo “This folder does not exist”
My_Program #I can handle this one
fi
done
The program works fine with files of the same format (ie .txt and .txt) but cannot check for a folder and .txt for the same name. Are there any changes I can make to check for the same name regardless of file format?
Edit: I did a little experimenting, and I put a duplicate data file into the directory with the output folders and it still didn't recognize it. I think the cd line or the if line is wrong then.. anyone have any tips on how I could fix this?
Use the syntax bellow to remove ".txt" from the end of value of $subj, returning the resulting string . (more info on "Bash String Manipulation")
${subj%.txt}
Then check the existence of files/directories with or without .txt:
if [ -e "$subj" ] || [ -e "${subj%.txt}" ]; then
....
If you want to remove any suffix (.txt, .tgz, ...) use ${subj%.*} to delete all characters after (and including) the last '.' Example:
[bash]$ subj=file.txt
[bash]$ echo ${subj%.*}
[bash]$ file
Or use ${subj%%.*} to delete all characters after (and including) the first '.':
[bash]$ subj=file.txt.tgz
[bash]$ echo ${subj%%.*}
[bash]$ file
#!/bin/sh
ls|cat > temp #WRITE CONTENTS OF CURRENT DIRECTORY INTO temp FILE
LENGTH=$(wc -l temp) #TO MONITOR EOF
echo "there are ${LENGTH% *} files/directories present"
COUNT=1 #INITIALIZE COUNTER BY ONE
CONTENT=$(head -$COUNT temp|tail -1) #STORE COUNTth NAME INTO CONTENT
echo $CONTENT|sed s/' '/'\\'/g
mv `echo $CONTENT|sed s/' '/'\\'/g` virtual/
The intention is to write the output of the ls command to a temp file, pick a file name one by one from the temp file, and copy them to respective location.
The problem is, when I use "CONTENT" as an argument to the cp command, suppose $CONTENT is "hello world", cp treat it as two separate files "hello" and "world".
Need help.
Always wrap shell variables in quotes when you use them as parameters to a command. This solves the space problem you encountered and helps protect against malicious input.
echo "$CONTENT"