This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I wrote a shell script that calls the ffmpeg tool but when I run it, it says No such file or directory yet it does!
Here is my script:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
FFMPEG_DIR="/media/sf_data/livraison_transcripts/ffmpeg-git-20180208-64bit-static"
for file in MAIN_DIR/audio_mp3/*.mp3;
do
cp -p file FFMPEG_DIR;
done
for file in FFMPEG_DIR/*.mp3;
do
./ffmpeg -i ${file%.mp3}.ogg
sox $file -t raw --channels=1 --bits=16 --rate=16000 --encoding=signed-
integer --endian=little ${file%.ogg}.raw;
done
for file in FFMPEG_DIR/*.raw;
do
cp -p file MAIN_DIR/pipeline/audio_raw/;
done
and here is the debug response:
cp: cannot stat ‘file’: No such file or directory
./essai.sh: line 14: ./ffmpeg: No such file or directory
sox FAIL formats: can't open input file `FFMPEG_DIR/*.mp3': No such file or
directory
cp: cannot stat ‘file’: No such file or directory
FYI I'm running CentOS7 on VirtualBox
Thank you
Here's a Minimal, Complete, and Verifiable example (MCVE), a version of your script that removes everything not required to show the problem:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
echo MAIN_DIR
Expected output:
/media/sf_data/pipeline
Actual output:
MAIN_DIR
This is because bash requires a $ when expanding variables:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
echo "$MAIN_DIR"
The quotes are not required to fix the issue, but prevent issues with whitespaces.
Hi You need couple of correction in your shell script see below. To get the actual value assigned to a variable you need to add $ at the front of the variable in shell script.
for file in $"MAIN_DIR"/audio_mp3/*.mp3;
do
cp -p "$file" "$FFMPEG_DIR";
done
for file in "$FFMPEG_DIR"/*.mp3;
./ffmpeg -i ${file%.mp3}.ogg
#provide full path like /usr/bin/ffmpeg
for file in "$FFMPEG_DIR"/*.raw;
do
cp -p "$file" "$MAIN_DIR"/pipeline/audio_raw/;
done
I have a .txt file which has sed replace commands on each line and has 1000+ entries. The file is on my server.
Can any one please help me how to run the commands in file line by line and execute till the end of file.
My txt file looks like
sed -i 's|http://www.myoldsite/url/category/another/blah|http://www.mynewsite.com|g' ./db.sql
sed -i 's|http://www.myoldsite/url/category/blah/blah|http://www.mynewsite.com|g' ./db.sql
sed -i 's|http://www.myoldsite/url/category/blah|http://www.mynewsite.com|g' ./db.sql
You can convert your file to a list of substitution commands by removing all occurrences of sed -i ' and ' ./db.sql.
Using process substitution, the list can then be processed as a file passed to the sed -f option.
sed -i -f <(sed "s/[^']*'//;s/'.*//" file) ./db.sql
Maybe this one helps:
for i in $(cat yourfile.txt)
do
sudo $i
done
EDIT:
Why downvote?
There are multiple solutions to your problem.
SOLUTION 1
Running each command in your .txt file:
Make your .txt file executable by giving command:
chmod +x yourfile.txt
And then executing it by giving a command:
./yourfile.txt
SOLUTION 2
Creating a script
#!/bin/sh
FILE=$1
while read line; do
$line
done < $FILE
making the script executable:
chmod +x yourscriptfile.sh
And then executing your script by providing your .txt file as argument:
./yourscriptfile.sh yourfilewithcommands.txt
Hope this helps.
You can turn the file into a script by adding
#!/bin/sh
as the first line, than making the file executable with
chomd u+x file
and then just run it with
./file
BUT, this will be very slow. It's much faster to run sed only once with all the expressions, i.e. changing it into
sed -i~ 's|http://www.myoldsite/url/category/another/blah|http://www.mynewsite.com|g;
s|http://www.myoldsite/url/category/blah/blah|http://www.mynewsite.com|g;
s|http://www.myoldsite/url/category/blah|http://www.mynewsite.com|g
' ./db.sql
I've successfully used the following sed command to search/replace text in Linux:
sed -i 's/old_link/new_link/g' *
However, when I try it on my Mac OS X, I get:
"command c expects \ followed by text"
I thought my Mac runs a normal BASH shell. What's up?
EDIT:
According to #High Performance, this is due to Mac sed being of a different (BSD) flavor, so my question would therefore be how do I replicate this command in BSD sed?
EDIT:
Here is an actual example that causes this:
sed -i 's/hello/gbye/g' *
If you use the -i option you need to provide an extension for your backups.
If you have:
File1.txt
File2.cfg
The command (note the lack of space between -i and '' and the -e to make it work on new versions of Mac and on GNU):
sed -i'.original' -e 's/old_link/new_link/g' *
Create 2 backup files like:
File1.txt.original
File2.cfg.original
There is no portable way to avoid making backup files because it is impossible to find a mix of sed commands that works on all cases:
sed -i -e ... - does not work on OS X as it creates -e backups
sed -i'' -e ... - does not work on OS X 10.6 but works on 10.9+
sed -i '' -e ... - not working on GNU
Note Given that there isn't a sed command working on all platforms, you can try to use another command to achieve the same result.
E.g., perl -i -pe's/old_link/new_link/g' *
I believe on OS X when you use -i an extension for the backup files is required. Try:
sed -i .bak 's/hello/gbye/g' *
Using GNU sed the extension is optional.
This works with both GNU and BSD versions of sed:
sed -i'' -e 's/old_link/new_link/g' *
or with backup:
sed -i'.bak' -e 's/old_link/new_link/g' *
Note missing space after -i option! (Necessary for GNU sed)
Had the same problem in Mac and solved it with brew:
brew install gnu-sed
and use as
gsed SED_COMMAND
you can set as well set sed as alias to gsed (if you want):
alias sed=gsed
Or, you can install the GNU version of sed in your Mac, called gsed, and use it using the standard Linux syntax.
For that, install gsed using ports (if you don't have it, get it at http://www.macports.org/) by running sudo port install gsed. Then, you can run sed -i 's/old_link/new_link/g' *
Your Mac does indeed run a BASH shell, but this is more a question of which implementation of sed you are dealing with. On a Mac sed comes from BSD and is subtly different from the sed you might find on a typical Linux box. I suggest you man sed.
Insead of calling sed with sed, I do ./bin/sed
And this is the wrapper script in my ~/project/bin/sed
#!/bin/bash
if [[ "$OSTYPE" == "darwin"* ]]; then
exec "gsed" "$#"
else
exec "sed" "$#"
fi
Don't forget to chmod 755 the wrapper script.
Sinetris' answer is right, but I use this with find command to be more specific about what files I want to change. In general this should work (tested on osx /bin/bash):
find . -name "*.smth" -exec sed -i '' 's/text1/text2/g' {} \;
In general when using sed without find in complex projects is less efficient.
I've created a function to handle sed difference between MacOS (tested on MacOS 10.12) and other OS:
OS=`uname`
# $(replace_in_file pattern file)
function replace_in_file() {
if [ "$OS" = 'Darwin' ]; then
# for MacOS
sed -i '' -e "$1" "$2"
else
# for Linux and Windows
sed -i'' -e "$1" "$2"
fi
}
Usage:
$(replace_in_file 's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' "./mysql/.env")
Where:
, is a delimeter
's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' is pattern
"./mysql/.env" is path to file
As the other answers indicate, there is not a way to use sed portably across OS X and Linux without making backup files. So, I instead used this Ruby one-liner to do so:
ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml
In my case, I needed to call it from a rake task (i.e., inside a Ruby script), so I used this additional level of quoting:
sh %q{ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml}
Here's how to apply environment variables to template file (no backup need).
1. Create template with {{FOO}} for later replace.
echo "Hello {{FOO}}" > foo.conf.tmpl
2. Replace {{FOO}} with FOO variable and output to new foo.conf file
FOO="world" && sed -e "s/{{FOO}}/$FOO/g" foo.conf.tmpl > foo.conf
Working both macOS 10.12.4 and Ubuntu 14.04.5
Here is an option in bash scripts:
#!/bin/bash
GO_OS=${GO_OS:-"linux"}
function detect_os {
# Detect the OS name
case "$(uname -s)" in
Darwin)
host_os=darwin
;;
Linux)
host_os=linux
;;
*)
echo "Unsupported host OS. Must be Linux or Mac OS X." >&2
exit 1
;;
esac
GO_OS="${host_os}"
}
detect_os
if [ "${GO_OS}" == "darwin" ]; then
sed -i '' -e ...
else
sed -i -e ...
fi
sed -ie 's/old_link/new_link/g' *
Works on both BSD & Linux with gnu sed
I want to take a local file and zip it at some other temp location.
Following is my code which is giving error
path="/home/tft/Downloads/ie.js"
temp="/home/tft/Downloads/Temp"
name=$(basename "$path")
echo "$path"
echo "$name"
echo "$temp"
echo "$temp/$name.zip" #Its output is also weird!
zip -r -j "$temp/$name.zip" $path
Getting below output:
/bin/bash test.sh
/home/tft/Downloads/ie.js
ie.js
/home/tft/Downloads/Temp
.zipjstft/Downloads/Temp
zip warning: name not matched: /home/tft/Downloads/ie.js
)zip . -i /home/tft/Downloads/ie.js -r -j /home/tft/Downloads/Temp
Your script is in DOS format. Convert it first to UNIX format:
sed -i 's|\r||' yourscript.sh
Or use dos2unix:
dos2unix yourscript.sh
The error messages you see is caused by having an extra character (carriage return \r) at the end of your values. This happens when your file's format is not UNIX but DOS since DOS' line endings is \r\n where UNIX only has \n.