Thank you in advance for any help, this is coursework so further reading/ pointers is greatly appreciated.
I asked a question the other day relating to my own delete/trash/restore scripts and I have completed delete and trash as well as giving delete a backup text file for Restore to use later on.
However, instead of giving me errors, the Restore script just kinda stops in the console. Like when I type # ~/Restore -n the cursor skips to the next line without the usual # and I have to close it manually. Likewise without the -n option. The -n option should ask for a new location to restore to, and without it should restore to the files original location.
I'll post my script, see what y'all think.
#!/bin/bash
if [ "$1" == "-n" ]
then cd ~/rubbish
restore= grep $2 ~/store
filename= basename "$restore"
echo "Type the files new location"
read location
location1 = "readlink -f $location"
mv -i $filename "$location1" /$filename
else cd ~/rubbish
restore= grep $2 ~/store
filename= basename "$restore"
mv -i $filename "$location1" $location
fi
so, ~/rubbish is my own created directory to act as a recycle bin and ~/store is my text file which appends the deleted files readlink details on deletion. I can post the whole 3 scripts if necessary?
Many thanks!
If you call ~/Restore -n it will go to the if part and do a grep $2 ~/store. Since there is no parameter $2 it will result in grep ~/store, which tells grep to search for "~/store" in the input coming from standard input.
That's why your script stops and waits for input.
You can either test for a second parameter or enclose $2 in double quotes to make sure grep gets the correct number of parameters. Better yet, do both: 1. test for a second parameter and 2. enclose $2 in double quotes.
Some more points:
Don't put spaces around =
enclose commands in backticks `, if you want to capture the output
And no spaces between directory and filename
So, you should presumably write
restore=`grep "$2" ~/store`
filename=`basename "$restore"`
echo "Type the files new location"
read location
location1=`readlink -f "$location"`
mv -i $filename "$location1/$filename"
I suggest you look at bash info and follow the "Books and Resources".
I wrote one of these quite some time ago which I still use today. I don't have a restore script because I wrote it so that you could open your desktop trash can, right click and select "Restore". In other words it follows the Linux "trash info" standard.
http://wiki.linuxquestions.org/wiki/Scripting#KDE4_Command_Line_Trash_Can
Related
Hello I am trying to get all files with Jane's name to a separate file called oldFiles.txt. In a directory called "data" I am reading from a list of file names from a file called list.txt, from which I put all the file names containing the name Jane into the files variable. Then I'm trying to test the files variable with the files in list.txt to ensure they are in the file system, then append the all the files containing jane to the oldFiles.txt file(which will be in the scripts directory), after it tests to make sure the item within the files variable passes.
#!/bin/bash
> oldFiles.txt
files= grep " jane " ../data/list.txt | cut -d' ' -f 3
if test -e ~data/$files; then
for file in $files; do
if test -e ~/scripts/$file; then
echo $file>> oldFiles.txt
else
echo "no files"
fi
done
fi
The above code gets the desired files and displays them correctly, as well as creates the oldFiles.txt file, but when I open the file after running the script I find that nothing was appended to the file. I tried changing the file assignment to a pointer instead files= grep " jane " ../data/list.txt | cut -d' ' -f 3 ---> files=$(grep " jane " ../data/list.txt) to see if that would help by just capturing raw data to write to file, but then the error comes up "too many arguments on line 5" which is the 1st if test statement. The only way I get the script to work semi-properly is when I do ./findJane.sh > oldFiles.txt on the shell command line, which is me essentially manually creating the file. How would I go about this so that I create oldFiles.txt and append to the oldFiles.txt all within the script?
The biggest problem you have is matching names like "jane" or "Jane's", etc. while not matching "Janes". grep provides the options -i (case insensitive match) and -w (whole-word match) which can tailor your search to what you appear to want without having to use the kludge (" jane ") of appending spaces before an after your search term. (to properly do that you would use [[:space:]]jane[[:space:]])
You also have the problem of what is your "script dir" if you call your script from a directory other than the one containing your script, such as calling your script from your $HOME directory with bash script/findJane.sh. In that case your script will attempt to append to $HOME/oldFiles.txt. The positional parameter $0 always contains the full pathname to the current script being run, so you can capture the script directory no matter where you call the script from with:
dirname "$0"
You are using bash, so store all the filenames resulting from your grep command in an array, not some general variable (especially since your use of " jane " suggests that your filenames contain whitespace)
You can make your script much more flexible if you take the information of your input file (e.g list.txt), the term to search for (e.g. "jane"), the location where to check for existence of the files (e.g. $HOME/data) and the output filename to append the names to (e.g. "oldFile.txt") as command line [positonal] parameters. You can give each default values so it behaves as you currently desire without providing any arguments.
Even with the additional scripting flexibility of taking the command line arguments, the script actually has fewer lines simply filling an array using mapfile (synonymous with readarray) and then looping over the contents of the array. You also avoid the additional subshell for dirname with a simple parameter expansion and test whether the path component is empty -- to replace with '.', up to you.
If I've understood your goal correctly, you can put all the pieces together with:
#!/bin/bash
# positional parameters
src="${1:-../data/list.txt}" # 1st param - input (default: ../data/list.txt)
term="${2:-jane}" # 2nd param - search term (default: jane)
data="${3:-$HOME/data}" # 3rd param - file location (defaut: ../data)
outfn="${4:-oldFiles.txt}" # 4th param - output (default: oldFiles.txt)
# save the path to the current script in script
script="$(dirname "$0")"
# if outfn not given, prepend path to script to outfn to output
# in script directory (if script called from elsewhere)
[ -z "$4" ] && outfn="$script/$outfn"
# split names w/term into array
# using the -iw option for case-insensitive whole-word match
mapfile -t files < <(grep -iw "$term" "$src" | cut -d' ' -f 3)
# loop over files array
for ((i=0; i<${#files[#]}; i++)); do
# test existence of file in data directory, redirect name to outfn
[ -e "$data/${files[i]}" ] && printf "%s\n" "${files[i]}" >> "$outfn"
done
(note: test expression and [ expression ] are synonymous, use what you like, though you may find [ expression ] a bit more readable)
(further note: "Janes" being plural is not considered the same as the singular -- adjust the grep expression as desired)
Example Use/Output
As was pointed out in the comment, without a sample of your input file, we cannot provide an exact test to confirm your desired behavior.
Let me know if you have questions.
As far as I can tell, this is what you're going for. This is totally a community effort based on the comments, catching your bugs. Obviously credit to Mark and Jetchisel for finding most of the issues. Notable changes:
Fixed $files to use command substitution
Fixed path to data/$file, assuming you have a directory at ~/data full of files
Fixed the test to not test for a string of files, but just the single file (also using -f to make sure it's a regular file)
Using double brackets — you could also use double quotes instead, but you explicitly have a Bash shebang so there's no harm in using Bash syntax
Adding a second message about not matching files, because there are two possible cases there; you may need to adapt depending on the output you're looking for
Removed the initial empty redirection — if you need to ensure that the file is clear before the rest of the script, then it should be added back, but if not, it's not doing any useful work
Changed the shebang to make sure you're using the user's preferred Bash, and added set -e because you should always add set -e
#!/usr/bin/env bash
set -e
files=$(grep " jane " ../data/list.txt | cut -d' ' -f 3)
for file in $files; do
if [[ -f $HOME/data/$file ]]; then
if [[ -f $HOME/scripts/$file ]]; then
echo "$file" >> oldFiles.txt
else
echo "no matching file"
fi
else
echo "no files"
fi
done
Objective: The moment multiple.csv files are uploaded to the folder, code should check each filename, if appropriate filename, file should be further used by sqlloader to get data uploaded in the database. Once file is uploaded, code should delete the file processed. Next time, same process repeats.
I have some parts of the code working but some are creating problem, especially related to inotifywait. Please help.
In first loop, I am trying to monitor the /uploads folder, the moment it finds the .csv file, it checks if the filename has space. If yes, it wants to change the space to underscore in the filename. I have been trying to find a way to find "space, () or ," in the filename but only could do the 'space' part change. This is giving me an error that file cannot be moved, no such file or directory.
Second loop works separately but not when incorporated with first loop as there are errors which I have not been able to debug. If I run second loop separately, it is working correctly. But if there is a way to optimize the code better in one loop, I would be happy to know. Thanks!
Example: folder name: /../../upload
filenames: abc_123.csv (code should not make any change) , pqr(12 Apr).csv (code should change it to pqr_12_Apr.csv), May 12.csv (code should change it to May_12.csv) etc.
Once these 3 files have proper naming, it should be ready to be uploaded through sql loader and once files are processed, they get deleted.
My code is:
#!bin/bash
inotifywait -mqe create /../../upload | while read file; do
if [[ $file = '* *'.csv]]; then
mv "$file" ${file// /_}
fi
done
for file in /../..upload/*.csv
do
sqlcommand="sqlldr user/pwd control="/../xxx.ctl" data=$file silent=feedback, header"
$sqlcommand
rm $file
done
Thank you!
I have modified your script to this,
#!/usr/bin/env bash
while IFS= read -r file; do
filename=${file#* CREATE }
pathname=${file%/*}
if [[ $pathname/$filename = *\ *.csv ]]; then
echo mv -v "$pathname/$filename" "$pathname/${filename// /_}"
fi
done < <(inotifywait -mqe create /../../upload)
Remove the echo if you think the output is correct.
I just don't know how you can integrate the other parts of your script with that, probably create a separate script or remove the -m (which you don't want to do most probably). Well you could use a named pipe if mkfifo is available.
EDIT: as per OP's message add another parameter expansion for another string removal.
Add the code below the if [[ ... ]]; then
newfilename=${filename//\(\)}
Then change "${filename// /_}" to "${newfilename// /_}"
Disclaimer: I'm very new to bash and for some reason I'm having a very hard time learning this one. The syntax seems very different depending on the website I visit.
I have a simple wrapper script that I want to test if a file is gzipped or not, and if so, to zcat the file to a new temporary file and open it in an editor. Here's part of the script:
if file $FILE | grep -q gzip
then
timestamp=$(date +"%D_%T")
$( zcat $FILE > tmp-$timestamp )
fi
I'm getting an error: "tmp-10/19/15_15:16:41: No such file or directory"
I tried removing the command substitution syntax or putting tmp-$timestamp in double quotes and I get the same error. If I remove the -$timestamp part, then it seems to work fine. Can someone tell me what's going on here? I'm clearing missing something very simple.
tmp-10/19/15_15:16:41 refers to a file named 15_15:16:41 in directory 19 which is a subdirectory of tmp-10. If those directories and subdirectories do not exist, you cannot write to them.
Replace:
timestamp=$(date +"%D_%T")
With:
timestamp=$(date +"%F_%T")
This gives the date without the /.
As an example of this format:
$ date +"%F_%T"
2015-10-19_12:37:05
With %F, the year comes before the month which comes before the day. This means that your files will sort properly. For most people, that is an important advantage over %D.
Revised script
Your script can be simplified to:
if file "$file" | grep -q gzip
then
zcat "$file" > "tmp-$(date +"%F_%T")"
fi
Notes:
It is best practices not to use all caps for your shell variable. The system uses all caps for its variables and you don't want to accidentally overwrite one. Use lower case or mixed case and you'll be safe.
File names, such as $file, should always be in double-quotes. Some day, someone will give you a file name with a space in it and you don't want that to cause your script to fail.
The command substitution $(...) does not belong here. It has been removed.
I'm very new to SED and I'm having a hard time trying to append to an end of a directory. What I'm doing involves 2 basic things with sed but for some reason, no changes are made after the script runs. I will show segments of my script
I have a bash script that pulls my home directory from the host and I define the ID variable.
USERNAME="test"
#pull the home directory
dir=$(ssh -n -t $SERVERNAME "echo \$HOME";)
the above example will store /export/home/ID in the dir variable
echo $dir | sed 's/\([/export/home]*\).*/\1/' > olddir
the sed command above stores /export/home/ in the file olddir (takes off the ending)
sed -i 's_/home/$ _\$USERNAME_' olddir
i am now trying to change /export/home/ to /export/home/test using the defined variable with the escaped $.
after the script runs, it still has /export/home/ as the entry in the olddir file.
I'm using the -i to modify the file and I think I'm using the deliminators correctly? what could I be doing wrong? i even took off the $ from the USERNAME variable which didn't do anything. I know I'm missing something small but i just can't figure it out. I really appreciate your time to answer my question.
I think your command line can be modified then redirect output to olddir as follows:
echo $dir | sed 's#\(/export/home\).*#\1#' > olddir
to add the USERNAME VARIABLE
sed -i "s#/export/home#&/$USERNAME#" olddir
After using the provided information and playing around with the code, I found a solution that will work for both /export/home/ID and /home/ID situations.
I used both Xorg and also gniourf_gniourf's suggestions and below is my result.
echo $dir | sed 's_\([/export/home\]*\).*_\1_' > olddir
The above code is the first part of my solution.
I used what Xorg provided for the above code but it looks like i need those [] so i can use the *\ if i have it as echo $dir | sed 's_\(/export/home\).*_\1_' > olddir then the olddir file will contain /export/home/ID/test after the following sed command is used.
Here is the following sed command that I used:
sed -i 's_/home_&/'"$USERNAME"'_' olddir
The above code is the second part of my solution. I figured that I really only need to put focus on /home/ since i'm appending after it. I looked at gniourf_gniourf's comment and used the example where the quotes to isolate $USERNAME with "" that seems to be the way you can tell the script to use $USERNAME as a variable and not just put in the characters $USERNAME.
so after the script is run, depending on the host, I either have export/home/test/ or /home/test/ I can now put either of these into a new variable on the script to use to specify the home directory when creating ID's on remote hosts!
newdir="$(cat olddir)"
Thank you all so much for your help. I wouldn't have been able to figure this out with out your help.
Update: it turns out that the traling / at the end of directory is problem so i found a much easier way to replace the ID in the home directory
newdir="${dir/ID/${USERNAME}}"
i used these if statments for both home directory situations
`if [[ $dir == "/home/dhabinsk" ]]; then
newdir="${dir/dhabinsk/${USERNAME}}"
fi
if [[ $dir == "/export/home/dhabinsk" ]]; then
newdir="${dir/dhabinsk/${USERNAME}}"
fi`
cheers
Sometimes I need to rename some amount of files, such as add a prefix or remove something.
At first I wrote a python script. It works well, and I want a shell version. Therefore I wrote something like that:
$1 - which directory to list,
$2 - what pattern will be replacement,
$3 - replacement.
echo "usage: dir pattern replacement"
for fname in `ls $1`
do
newName=$(echo $fname | sed "s/^$2/$3/")
echo 'mv' "$1/$fname" "$1/$newName&&"
mv "$1/$fname" "$1/$newName"
done
It works but very slowly, probably because it needs to create a process (here sed and mv) and destroy it and create same process again just to have a different argument. Is that true? If so, how to avoid it, how can I get a faster version?
I thought to offer all processed files a name (using sed to process them at once), but it still needs mv in the loop.
Please tell me, how you guys do it? Thanks. If you find my question hard to understand please be patient, my English is not very good, sorry.
--- update ---
I am sorry for my description. My core question is: "IF we should use some command in loop, will that lower performance?" Because in for i in {1..100000}; do ls 1>/dev/null; done creating and destroying a process will take most of the time. So what I want is "Is there any way to reduce that cost?".
Thanks to kev and S.R.I for giving me a rename solution to rename files.
Every time you call an external binary (ls, sed, mv), bash has to fork itself to exec the command and that takes a big performance hit.
You can do everything you want to do in pure bash 4.X and only need to call mv
pat_rename(){
if [[ ! -d "$1" ]]; then
echo "Error: '$1' is not a valid directory"
return
fi
shopt -s globstar
cd "$1"
for file in **; do
echo "mv $file ${file//$2/$3}"
done
}
Simplest first. What's wrong with rename?
mkdir tstbin
for i in `seq 1 20`
do
touch tstbin/filename$i.txt
done
rename .txt .html tstbin/*.txt
Or are you using an older *nix machine?
To avoid re-executing sed on each file, you could instead setup two name streams, one original, and one transformed, then sip from the ends:
exec 3< <(ls)
exec 4< <(ls | sed 's/from/to/')
IFS=`echo`
while read -u3 orig && read -u4 to; do
mv "${orig}" "${to}";
done;
I think you can store all of file names into a file or string, and use awk and sed do it once instead of one by one.