This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I wrote a basic script which changes the directory to a specific path and shows the list of folders, but my script shows the list of files of the current folder where my script lies instead of which I specify in script.
Here is my script:
#!/bin/bash
v1="$(ls -l | awk '/^-/{ print $NF }' | rev | cut -d "_" -f2 | rev)"
v2=/home/PS212-28695/logs/
cd $v2 && echo $v1
Does any one knows what I am doing wrong?
Your current script makes no sense really. v1 variable is NOT a command to execute as you expect, but due to $() syntax it is in fact output of ls -t at the moment of assignment and that's why you have files from current directory there as this is your working directory at that particular moment. So you should rather be doing ordinary
ls -t /home/PS212-28695/logs/
EDIT
it runs but what if i need to store the ls -t output to variable
Then this is same syntax you already had, but with proper arguments:
v1=$(ls -t /home/PS212-28695/logs/)
echo ${v1}
If for any reason you want to cd then you have to do that prior setting v1 for the same reason I explained above.
I'm a terrible beginner at bash scripting, and am hoping someone can help me out with this issue.
Having a problem with the Prey project standalone scripts. There's a line that's supposed to send an email, and apparently its not formatted correctly.
response=`mailsender -f "$mail_from" -t "$mail_to" -u "$complete_subject" \
-s $smtp_server -a $file_list -o message-file="$trace_file.msg" \
tls=auto username=$smtp_username \
password=\`decrypt \"$smtp_password\"\``
Where mailsender is an alias to Brandon Zehm's PERL sendEmail script, $smtp_password is a pointless base64 encoding of the password, and decrypt is:
decrypt() {
echo "$1" | openssl enc -base64 -d
}
So can anyone tell me what's wrong with the script? For reference, if I just replace the entire decrypt part with the plaintext password, it works fine. i.e.:
response=`mailsender -f "$mail_from" -t "$mail_to" -u "$complete_subject" \
-s $smtp_server -a $file_list -o message-file="$trace_file.msg" \
tls=auto username=$smtp_username password=actual_password`
The simplest thing to do is avoid backticks, and use $() instead -- they nest cleanly, with no special escaping needed:
response=$(Documents/Projects/Shell\ Scripting/printargs -f "$mail_from" \
-t "$mail_to" -u "$complete_subject" -s $smtp_server -a $file_list \
-o message-file="$trace_file.msg" tls=auto username=$smtp_username \
password="$(decrypt "$smtp_password")")
I think this script is isomorphic with yours:
decrypt()
{
echo "$1" | tr 'a-z' 'A-Z'
}
xxx=`echo xxx=yyy pass=\`decrypt \"xyz abc\"\``
echo "$xxx"
When run with 'sh -x xxx' (where 'sh' is 'bash' in disguise):
$ sh -x xxx
+++ decrypt '"xyz' 'abc"'
+++ echo '"xyz'
+++ tr a-z A-Z
++ echo xxx=yyy 'pass="XYZ'
+ xxx='xxx=yyy pass="XYZ'
+ echo 'xxx=yyy pass="XYZ'
xxx=yyy pass="XYZ
$
You can see where there are problems - if you know how to look. The decrypt command line has two arguments where the intention was to have just one, and the arguments include a double quote before the first and another at the end of the second.
So, in your script, the argument passed to decrypt includes a pair of double quotes, which probably isn't what you wanted.
If we rewrite the script using the '$(...)' notation, which nests much more neatly, then we get:
decrypt()
{
echo "$1" | tr 'a-z' 'A-Z'
}
yyy=$(echo zzz=yyy pass=$(decrypt "xyz abc"))
echo "$yyy"
The trace from this looks like:
$ sh -x xxx
+++ decrypt 'xyz abc'
+++ echo 'xyz abc'
+++ tr a-z A-Z
++ echo zzz=yyy pass=XYZ ABC
+ yyy='zzz=yyy pass=XYZ ABC'
+ echo 'zzz=yyy pass=XYZ ABC'
zzz=yyy pass=XYZ ABC
$
I'm one of the guys from Prey. This bug was confimed yesterday and a fix has already been commited.
I do agree that $() is much easier to read than backticking -- and specially back-backticking --, and actually that's one of the things we're working on (big code refactoring).
Lately I've been working on a Bash framework called Skull which provides a much nicer interface for writing shell scripts. Hopefully Prey 0.6 will be based completely on it, and excessive backticking will be replaced with $() to make it easier for everyone to read.
How could I retrieve the current working directory/folder name in a bash script, or even better, just a terminal command.
pwd gives the full path of the current working directory, e.g. /opt/local/bin but I only want bin.
No need for basename, and especially no need for a subshell running pwd (which adds an extra, and expensive, fork operation); the shell can do this internally using parameter expansion:
result=${PWD##*/} # to assign to a variable
result=${result:-/} # to correct for the case where PWD=/
printf '%s\n' "${PWD##*/}" # to print to stdout
# ...more robust than echo for unusual names
# (consider a directory named -e or -n)
printf '%q\n' "${PWD##*/}" # to print to stdout, quoted for use as shell input
# ...useful to make hidden characters readable.
Note that if you're applying this technique in other circumstances (not PWD, but some other variable holding a directory name), you might need to trim any trailing slashes. The below uses bash's extglob support to work even with multiple trailing slashes:
dirname=/path/to/somewhere//
shopt -s extglob # enable +(...) glob syntax
result=${dirname%%+(/)} # trim however many trailing slashes exist
result=${result##*/} # remove everything before the last / that still remains
result=${result:-/} # correct for dirname=/ case
printf '%s\n' "$result"
Alternatively, without extglob:
dirname="/path/to/somewhere//"
result="${dirname%"${dirname##*[!/]}"}" # extglob-free multi-trailing-/ trim
result="${result##*/}" # remove everything before the last /
result=${result:-/} # correct for dirname=/ case
Use the basename program. For your case:
% basename "$PWD"
bin
$ echo "${PWD##*/}"
Use:
basename "$PWD"
OR
IFS=/
var=($PWD)
echo ${var[-1]}
Turn the Internal Filename Separator (IFS) back to space.
IFS=
There is one space after the IFS.
You can use a combination of pwd and basename. E.g.
#!/bin/bash
CURRENT=`pwd`
BASENAME=`basename "$CURRENT"`
echo "$BASENAME"
exit;
How about grep:
pwd | grep -o '[^/]*$'
This thread is great! Here is one more flavor:
pwd | awk -F / '{print $NF}'
basename $(pwd)
or
echo "$(basename $(pwd))"
I like the selected answer (Charles Duffy), but be careful if you are in a symlinked dir and you want the name of the target dir. Unfortunately I don't think it can be done in a single parameter expansion expression, perhaps I'm mistaken. This should work:
target_PWD=$(readlink -f .)
echo ${target_PWD##*/}
To see this, an experiment:
cd foo
ln -s . bar
echo ${PWD##*/}
reports "bar"
DIRNAME
To show the leading directories of a path (without incurring a fork-exec of /usr/bin/dirname):
echo ${target_PWD%/*}
This will e.g. transform foo/bar/baz -> foo/bar
echo "$PWD" | sed 's!.*/!!'
If you are using Bourne shell or ${PWD##*/} is not available.
Surprisingly, no one mentioned this alternative that uses only built-in bash commands:
i="$IFS";IFS='/';set -f;p=($PWD);set +f;IFS="$i";echo "${p[-1]}"
As an added bonus you can easily obtain the name of the parent directory with:
[ "${#p[#]}" -gt 1 ] && echo "${p[-2]}"
These will work on Bash 4.3-alpha or newer.
There are a lots way of doing that I particularly liked Charles way because it avoid a new process, but before know this I solved it with awk
pwd | awk -F/ '{print $NF}'
For the find jockeys out there like me:
find $PWD -maxdepth 0 -printf "%f\n"
i usually use this in sh scripts
SCRIPTSRC=`readlink -f "$0" || echo "$0"`
RUN_PATH=`dirname "${SCRIPTSRC}" || echo .`
echo "Running from ${RUN_PATH}"
...
cd ${RUN_PATH}/subfolder
you can use this to automate things ...
Just use:
pwd | xargs basename
or
basename "`pwd`"
Below grep with regex is also working,
>pwd | grep -o "\w*-*$"
If you want to see only the current directory in the bash prompt region, you can edit .bashrc file in ~. Change \w to \W in the line:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u#\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
Run source ~/.bashrc and it will only display the directory name in the prompt region.
Ref: https://superuser.com/questions/60555/show-only-current-directory-name-not-full-path-on-bash-prompt
I strongly prefer using gbasename, which is part of GNU coreutils.
Just run the following command line:
basename $(pwd)
If you want to copy that name:
basename $(pwd) | xclip -selection clipboard
An alternative to basname examples
pwd | grep -o "[^/]*$"
OR
pwd | ack -o "[^/]+$"
My shell did not come with the basename package and I tend to avoid downloading packages if there are ways around it.
You can use the basename utility which deletes any prefix ending in / and the suffix (if present in string) from string, and prints the
result on the standard output.
$basename <path-of-directory>
Just remove any character until a / (or \, if you're on Windows). As the match is gonna be made greedy it will remove everything until the last /:
pwd | sed 's/.*\///g'
In your case the result is as expected:
λ a='/opt/local/bin'
λ echo $a | sed 's/.*\///g'
bin
Here's a simple alias for it:
alias name='basename $( pwd )'
After putting that in your ~/.zshrc or ~/.bashrc file and sourcing it (ex: source ~/.zshrc), then you can simply run name to print out the current directories name.
The following commands will result in printing your current working directory in a bash script.
pushd .
CURRENT_DIR="`cd $1; pwd`"
popd
echo $CURRENT_DIR
I have a list of URLs which I would like to feed into wget using --input-file.
However I can't work out how to control the --output-document value at the same time,
which is simple if you issue the commands one by one.
I would like to save each document as the MD5 of its URL.
cat url-list.txt | xargs -P 4 wget
And xargs is there because I also want to make use of the max-procs features for parallel downloads.
Don't use cat. You can have xargs read from a file. From the man page:
--arg-file=file
-a file
Read items from file instead of standard input. If you use this
option, stdin remains unchanged when commands are run. Other‐
wise, stdin is redirected from /dev/null.
how about using a loop?
while read -r line
do
md5=$(echo "$line"|md5sum)
wget ... $line ... --output-document $md5 ......
done < url-list.txt
In your question you use -P 4 which suggests you want your solution to run in parallel. GNU Parallel http://www.gnu.org/software/parallel/ may help you:
cat url-list.txt | parallel 'wget {} --output-document "`echo {}|md5sum`"'
You can do that like this :
cat url-list.txt | while read url;
do
wget $url -O $( echo "$url" | md5 );
done
good luck