I need to use this command
/usr/local/bin/mcl find -f .bz2
which returns me this
:???????? '/Cloud Drive/test1.bz2'
:???????? '/Cloud Drive/test2.bz2'
into a BASH script. The problem is that I need the last parameter (.bz2) to be a variable.
I've tried with this
FILENAME=".bz2"
UPLOADED=$(/usr/local/bin/mcl find -f $FILENAME)
# do something with $UPLOADED
But obviously it is not working. After some research on StackOverflow and on the web I have found several ways to do something like that (even using backticks), but still I can't manage to make it work.
What is the correct way to do that?
You mean like this?
uploaded=$(mcl find -f "$FILENAME" | cut -d"'" -f2)
for u in $uploaded; do
echo "$u"
# process "$u"
done
You can try save the following as e.g. ./script.sh
filename="${1:-.bz2}" #<-- your variable as 1st argument, defaults to .bz2
do_my_work() {
local uploaded="$1"
#do whatever you want with the "uploaded"
printf "got:==%s==\n" "$uploaded"
}
while IFS= read -r __mc1path
do
do_my_work "$__mc1path"
done < <(mc1 find -f "$filename" | sed "s/.*'\(.*\)'.*/\1/")
# variable----^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^- keep only stuff inside of quotes
and use it as
./script.sh .bz2 #or anything, defaults to ".bz2"
and will print
got:==/Cloud Drive/test1.bz2==
got:==/Cloud Drive/test2.bz2==
I think you want that :
UPLOADED=`/usr/local/bin/mcl find -f $FILENAME`
Related
I'm trying to run the following code on files that I choose and put into a variable source file. This is for ease of the user when I export it out. This is the code before trying to add a source file:
for file in ~/Twitter/Users/New/*; do
[ -f "$file" ] && sed '1,7d' "$file" | head -n -9 > ~/Twitter/Users/TBA/"${file##*/}"
done
So I tried adding a source file like so:
#!/bin/bash
source ~/MYBASHSCRIPTS/Tests/scriptsettings.in
for file in $loctorem/*; do
[ -f "$file" ] && sed '1,7d' "$file" | head -n -9 > $locdone
done
echo $loctorem
echo $locdone
with the scriptsettings.in configured as such:
loctorem="~/Twitter/Users/New"
locdone='~/Twitter/Users/TBA/"${file##*/}"'
I have tried both half old/half new code but neither work. does it really need to be hard coded in order to run? This will throw my whole "noob friendly" idea in the trash if so...
EDIT--- I only echo it at the end so that I can verify that it is calling the correct locations.
EDIT2--- Here is the exact script I original ran.
#!/bin/bash
for file in ~/Anon/Twitter/OpISIS/New/*; do
[ -f "$file" ] && sed '1,7d' "$file" | head -n -9 > ~/Anon/Twitter/OpISIS/TBA/"${file##*/}"
done
And the new variant:
source ~/MYBASHSCRIPTS/Tests/scriptsettings.in
for file in $loctorem/*; do
[ -f "$file" ] && sed '1,7d' "$file" | head -n -9 > "$(locdone_path "$file")"
done
with the source file being:
loctorem=/home/matrix/Anon/Twitter/OpISIS/New
locdone_path() { printf '%s\n' ~/Twitter/Users/TBA/"${1##*/}
as I said before, I'm still pretty new so sorry ifI'm doing an insanely stupid thing in here..
I'm trying to make the input and output folder/file set to a variable that the user can change. in the end this script will be ~80 lines and I want anyone to be able to run it instead of forcing everyone to have directories/files set up like mine. Then I'll have a setup script that makes the file with the variables stored in them so there is a one time setup, or the user can later change locations but they dont have to go into the entire code and change everything just to fit their system.
You've got two problems here. The first are the quotes, which prevent tilde expansion:
# this stores a path with a literal ~ character
loctorem='~/Twitter/Users/New'
# this works
loctorem=~/Twitter/Users/New
# this works too
loctorem="$HOME/Twitter/Users/New"
The second issue is that you're depending on $file before it's available. If you want to store code (an algorithm on how to calculate something, for instance), in your configuration, define a function:
# this can be put in your sourced file
locdone_path() { printf '%s\n' ~/Twitter/Users/TBA/"${1##*/}"; }
...and, later, to use that code, invoke the function:
... | head -n 9 >"$(locdone_path "$file")"
However, if you only want to make the directory customizable, you might do something much simpler:
loctorem=~/Twitter/Users/New
locdone=~/Twitter/Users/TBA
and:
... | head -n 9 >"$locdone/${file##*/}"
I'm very new to SED and I'm having a hard time trying to append to an end of a directory. What I'm doing involves 2 basic things with sed but for some reason, no changes are made after the script runs. I will show segments of my script
I have a bash script that pulls my home directory from the host and I define the ID variable.
USERNAME="test"
#pull the home directory
dir=$(ssh -n -t $SERVERNAME "echo \$HOME";)
the above example will store /export/home/ID in the dir variable
echo $dir | sed 's/\([/export/home]*\).*/\1/' > olddir
the sed command above stores /export/home/ in the file olddir (takes off the ending)
sed -i 's_/home/$ _\$USERNAME_' olddir
i am now trying to change /export/home/ to /export/home/test using the defined variable with the escaped $.
after the script runs, it still has /export/home/ as the entry in the olddir file.
I'm using the -i to modify the file and I think I'm using the deliminators correctly? what could I be doing wrong? i even took off the $ from the USERNAME variable which didn't do anything. I know I'm missing something small but i just can't figure it out. I really appreciate your time to answer my question.
I think your command line can be modified then redirect output to olddir as follows:
echo $dir | sed 's#\(/export/home\).*#\1#' > olddir
to add the USERNAME VARIABLE
sed -i "s#/export/home#&/$USERNAME#" olddir
After using the provided information and playing around with the code, I found a solution that will work for both /export/home/ID and /home/ID situations.
I used both Xorg and also gniourf_gniourf's suggestions and below is my result.
echo $dir | sed 's_\([/export/home\]*\).*_\1_' > olddir
The above code is the first part of my solution.
I used what Xorg provided for the above code but it looks like i need those [] so i can use the *\ if i have it as echo $dir | sed 's_\(/export/home\).*_\1_' > olddir then the olddir file will contain /export/home/ID/test after the following sed command is used.
Here is the following sed command that I used:
sed -i 's_/home_&/'"$USERNAME"'_' olddir
The above code is the second part of my solution. I figured that I really only need to put focus on /home/ since i'm appending after it. I looked at gniourf_gniourf's comment and used the example where the quotes to isolate $USERNAME with "" that seems to be the way you can tell the script to use $USERNAME as a variable and not just put in the characters $USERNAME.
so after the script is run, depending on the host, I either have export/home/test/ or /home/test/ I can now put either of these into a new variable on the script to use to specify the home directory when creating ID's on remote hosts!
newdir="$(cat olddir)"
Thank you all so much for your help. I wouldn't have been able to figure this out with out your help.
Update: it turns out that the traling / at the end of directory is problem so i found a much easier way to replace the ID in the home directory
newdir="${dir/ID/${USERNAME}}"
i used these if statments for both home directory situations
`if [[ $dir == "/home/dhabinsk" ]]; then
newdir="${dir/dhabinsk/${USERNAME}}"
fi
if [[ $dir == "/export/home/dhabinsk" ]]; then
newdir="${dir/dhabinsk/${USERNAME}}"
fi`
cheers
Sometimes I need to rename some amount of files, such as add a prefix or remove something.
At first I wrote a python script. It works well, and I want a shell version. Therefore I wrote something like that:
$1 - which directory to list,
$2 - what pattern will be replacement,
$3 - replacement.
echo "usage: dir pattern replacement"
for fname in `ls $1`
do
newName=$(echo $fname | sed "s/^$2/$3/")
echo 'mv' "$1/$fname" "$1/$newName&&"
mv "$1/$fname" "$1/$newName"
done
It works but very slowly, probably because it needs to create a process (here sed and mv) and destroy it and create same process again just to have a different argument. Is that true? If so, how to avoid it, how can I get a faster version?
I thought to offer all processed files a name (using sed to process them at once), but it still needs mv in the loop.
Please tell me, how you guys do it? Thanks. If you find my question hard to understand please be patient, my English is not very good, sorry.
--- update ---
I am sorry for my description. My core question is: "IF we should use some command in loop, will that lower performance?" Because in for i in {1..100000}; do ls 1>/dev/null; done creating and destroying a process will take most of the time. So what I want is "Is there any way to reduce that cost?".
Thanks to kev and S.R.I for giving me a rename solution to rename files.
Every time you call an external binary (ls, sed, mv), bash has to fork itself to exec the command and that takes a big performance hit.
You can do everything you want to do in pure bash 4.X and only need to call mv
pat_rename(){
if [[ ! -d "$1" ]]; then
echo "Error: '$1' is not a valid directory"
return
fi
shopt -s globstar
cd "$1"
for file in **; do
echo "mv $file ${file//$2/$3}"
done
}
Simplest first. What's wrong with rename?
mkdir tstbin
for i in `seq 1 20`
do
touch tstbin/filename$i.txt
done
rename .txt .html tstbin/*.txt
Or are you using an older *nix machine?
To avoid re-executing sed on each file, you could instead setup two name streams, one original, and one transformed, then sip from the ends:
exec 3< <(ls)
exec 4< <(ls | sed 's/from/to/')
IFS=`echo`
while read -u3 orig && read -u4 to; do
mv "${orig}" "${to}";
done;
I think you can store all of file names into a file or string, and use awk and sed do it once instead of one by one.
I'm using this script to monitor the downloads folder for new .bin files being created. However, it doesn't seem to be working. If I remove the grep, I can make it copy any file created in the Downloads folder, but with the grep it's not working. I suspect the problem is how I'm trying to compare the two values, but I'm really not sure what to do.
#!/bin/sh
downloadDir="$HOME/Downloads/"
mbedDir="/media/mbed"
inotifywait -m --format %f -e create $downloadDir -q | \
while read line; do
if [ $(ls $downloadDir -a1 | grep '[^.].*bin' | head -1) == $line ]; then
cp "$downloadDir/$line" "$mbedDir/$line"
fi
done
The ls $downloadDir -a1 | grep '[^.].*bin' | head -1 is the wrong way to go about this. To see why, suppose you had files named a.txt and b.bin in the download directory, and then c.bin was added. inotifywait would print c.bin, ls would print a.txt\nb.bin\nc.bin (with actual newlines, not \n), grep would thin that to b.bin\nc.bin, head would remove all but the first line leaving b.bin, which would not match c.bin. You need to be checking $line to see if it ends in .bin, not scanning a directory listing. I'll give you three ways to do this:
First option, use grep to check $line, not the listing:
if echo "$line" | grep -q '[.]bin$'; then
Note that I'm using the -q option to supress grep's output, and instead simply letting the if command check its exit status (success if it found a match, failure if not). Also, the RE is anchored to the end of the line, and the period is in brackets so it'll only match an actual period (normally, . in a regular expression matches any single character). \.bin$ would also work here.
Second option, use the shell's ability to edit variable contents to see if $line ends in .bin:
if [ "${line%.bin}" != "$line" ]; then
the "${line%.bin}" part gives the value of $line with .bin trimmed from the end if it's there. If that's not the same as $line itself, then $line must've ended with .bin.
Third option, use bash's [[ ]] expression to do pattern matching directly:
if [[ "$line" == *.bin ]]; then
This is (IMHO) the simplest and clearest of the bunch, but it only works in bash (i.e. you must start the script with #!/bin/bash).
Other notes: to avoid some possible issues with whitespace and backslashes in filenames, use while IFS= read -r line; do and follow #shellter's recommendation about double-quotes religiously.
Also, I'm not very familiar with inotifywait, but AIUI its -e create option will notify you when the file is created, not when its contents are fully written out. Depending on the timing, you may wind up copying partially-written files.
Finally, you don't have any checking for duplicate filenames. What should happen if you download a file named foo.bin, it gets copied, you delete the original, then download a different file named foo.bin. As the script is now, it'll silently overwrite the first foo.bin. If this isn't what you want, you should add something like:
if [ ! -e "$mbedDir/$line" ]; then
cp "$downloadDir/$line" "$mbedDir/$line"
elif ! cmp -s "$downloadDir/$line" "$mbedDir/$line"; then
echo "Eeek, a duplicate filename!" >&2
# or possibly something more constructive than that...
fi
This question already has answers here:
Rename multiple files based on pattern in Unix
(24 answers)
Closed 5 years ago.
I have loads of files which look like this:
DET01-ABC-5_50-001.dat
...
DET01-ABC-5_50-0025.dat
and I want them to look like this:
DET01-XYZ-5_50-001.dat
...
DET01-XYZ-5_50-0025.dat
How can I do this?
There are a couple of variants of a rename command, in your case, it may be as simple as
rename ABC XYZ *.dat
You may have a version which takes a Perl regex;
rename 's/ABC/XYZ/' *.dat
for file in *.dat ; do mv $file ${file//ABC/XYZ} ; done
No rename or sed needed. Just bash parameter expansion.
Something like this will do it. The for loop may need to be modified depending on which filenames you wish to capture.
for fspec1 in DET01-ABC-5_50-*.dat ; do
fspec2=$(echo ${fspec1} | sed 's/-ABC-/-XYZ-/')
mv ${fspec1} ${fspec2}
done
You should always test these scripts on copies of your data, by the way, and in totally different directories.
You'll need to learn how to use sed http://unixhelp.ed.ac.uk/CGI/man-cgi?sed
And also to use for so you can loop through your file entries http://www.cyberciti.biz/faq/bash-for-loop/
Your command will look something like this, I don't have a term beside me so I can't check
for i in `dir` do mv $i `echo $i | sed '/orig/new/g'`
I like to do this with sed. In you case:
for x in DET01-*.dat; do
echo $x | sed -r 's/DET01-ABC-(.+)\.dat/mv -v "\0" "DET01-XYZ-\1.dat"/'
done | sh -e
It is best to omit the "sh -e" part first to see what will be executed.
All of these answers are simple and good. However, I always like to add an interactive mode to these scripts so that I can find false positives.
if [[ -n $inInteractiveMode ]]
then
echo -e -n "$oldFileName => $newFileName\nDo you want to do this change? [Y/n]: "
read run
[[ -z $run || "$run" == "y" || "$run" == "Y" ]] && mv "$oldFileName" "$newFileName"
fi
Or make interactive mode the default and add a force flag (-f | --force) for automated scripts or if you're feeling daring. And this doesn't slow you down too much: the default response is "yes, I do want to rename" so you can just hit the enter key at each prompt (because of the -z $run test.