I have a .txt file which has sed replace commands on each line and has 1000+ entries. The file is on my server.
Can any one please help me how to run the commands in file line by line and execute till the end of file.
My txt file looks like
sed -i 's|http://www.myoldsite/url/category/another/blah|http://www.mynewsite.com|g' ./db.sql
sed -i 's|http://www.myoldsite/url/category/blah/blah|http://www.mynewsite.com|g' ./db.sql
sed -i 's|http://www.myoldsite/url/category/blah|http://www.mynewsite.com|g' ./db.sql
You can convert your file to a list of substitution commands by removing all occurrences of sed -i ' and ' ./db.sql.
Using process substitution, the list can then be processed as a file passed to the sed -f option.
sed -i -f <(sed "s/[^']*'//;s/'.*//" file) ./db.sql
Maybe this one helps:
for i in $(cat yourfile.txt)
do
sudo $i
done
EDIT:
Why downvote?
There are multiple solutions to your problem.
SOLUTION 1
Running each command in your .txt file:
Make your .txt file executable by giving command:
chmod +x yourfile.txt
And then executing it by giving a command:
./yourfile.txt
SOLUTION 2
Creating a script
#!/bin/sh
FILE=$1
while read line; do
$line
done < $FILE
making the script executable:
chmod +x yourscriptfile.sh
And then executing your script by providing your .txt file as argument:
./yourscriptfile.sh yourfilewithcommands.txt
Hope this helps.
You can turn the file into a script by adding
#!/bin/sh
as the first line, than making the file executable with
chomd u+x file
and then just run it with
./file
BUT, this will be very slow. It's much faster to run sed only once with all the expressions, i.e. changing it into
sed -i~ 's|http://www.myoldsite/url/category/another/blah|http://www.mynewsite.com|g;
s|http://www.myoldsite/url/category/blah/blah|http://www.mynewsite.com|g;
s|http://www.myoldsite/url/category/blah|http://www.mynewsite.com|g
' ./db.sql
Related
This question already has answers here:
Run script on multiple files
(3 answers)
Closed 3 years ago.
I'm very new to bash. I have ten text files that I want to edit with the same line of code.
#!/bin/bash
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g'| grep -o "here.*there" | sed -r '/^.{,100}$/d'
< files 1-10
I know I could use sed -f sed.sh <file1 >file1 but that only works with sed commands and it only works one file at a time?
Do I have to run a loop?
There's some great existing answers on the Unix stack exchange that help deal with your problem. Specifically, from this post, they use a loop to recursively loop through all the files in a particular directory, as follows:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -- 's/foo/bar/g' "$file"
fi
done
)
Note the line, shopt -s globstar dotglob;, which allows us to use globbing patterns in the for loop. We also enclose the code in brackets, to prevent the shopt -s globstar dotglob; line option from becoming a global setting.
If you would like to apply this example to your file, you can just place your files in the current directory, and the code would probably look something like this:
( shopt -s globstar dotglob;
for file in **; do
if [[ -f $file ]] && [[ -w $file ]]; then
sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$file"
fi
done
)
Note that we have placed a "$file" variable beside each of the seds that you used in your code, this replaces the name of the file for each command.
There is another example given in the code that allows you to pick which files to run on, rather than all the files in a directory, which you can also re-purpose for your code, as given here:
( shopt -s globstar dotglob
sed -i -- 's/foo/bar/g' **baz*
sed -i -- 's/foo/bar/g' **.baz
)
To answer your question of doing a loop on each line, you will need to put a loop for each line inside your for loop, like so:
while read line ; do
: sed -i -e 's/.\{6\}/&\n/g' -e 's/edit/edit2/g' | tr -d "\n" | sed 's/edit2/edit/g' | grep -o "here.*there" | sed -r '/^.{,100}$/d' "$line”
done
)
Although the for loop can be useful for dealing with files in recursive directories, I would recommend against also using another loop to grab lines, since it muddies your code, and it’s possible there is a better way to do it without parsing line by line.
The linked question is a fairly complete guide to many of the cases you may come across, and is also worth a read if you want to learn more.
Hope that helps!
You could use a for loop.
You could use the tool parallel.
Example
Create a set of test files using a for-loop
mkdir -p /tmp/so58333536
cd /tmp/so58333536
for i in 1.txt 2.txt 3.txt 4.txt 5.txt;do echo "The answer is 41" > $i;done
cat /tmp/so58333536/*
Now correct your mistake using parallel [1].
mkdir /tmp/so58333536.new
ls /tmp/so58333536/* |parallel "sed 's/41/42/' {} > /tmp/so58333536.new/{/}"
cat /tmp/so58333536.new/*
{}:: refers to the current file
{/}:: refers to name of the current file (path is removed)
Reads: List all files in so58333536 and apply the following sed command to each file and write the output to so58333536.new.
[1] Another option is to use sed -i for in-place editing.
Be very carefull with this!! Mistakes can cause serious damages!
# !! Do not use -i option regularly !!
ls /tmp/so58333536/* |parallel "sed -i 's/41/42/'"
I'm trying to delete a line from a file and update the file to reflect that. I'm running a bash command inside a python program. The line delete works on the terminal, but the file isn't updated.
subprocess.call("sed -e $d {}".format(self._path).split())
How can I update the file to not have this line anymore.
Should add the -i flag to sed command to edit the file in-place.
On BSD's sed:
subprocess.call("sed -i '' -e $d {}".format(self._path).split())
On GNU's sed:
subprocess.call("sed -i -e $d {}".format(self._path).split())
I want to copy only 20180721 files from Outgoing to Incoming folder. I also want to remove the first numbers from the file name and want to rename from -1 to -3. I want to keep my commands to minimum so I am using pax command below.
Filename:
216118105741_MOM-09330-20180721_102408-1.jar
Output expected:
MOM-09330-20180721_102408-3.jar
I have tried this command and it's doing most of the work apart from removing the number coming in front of the file name. Can anyone help?
Command used:
pax -rw -pe -s/-1/-3/ ./*20180721*.jar ../Incoming/
Try this simple script using just parameter expansion:
for file in *20180721*.jar; do
new=${file#*_}
cp -- "$file" "/path/to/destination/${new%-*}-3.jar"
done
You can try this
In general
for i in `ls files-to-copy-*`; do
cp $i `echo $i | sed "s/rename-from/rename-to/g"`;
done;
In your case
for i in `ls *_MOM*`; do
cp $i `echo $i | sed "s/_MOM/MOM/g" | sed "s/-1/-3/g"`;
done;
pax only applies the first successful substitution even if the -s option is specified more than once. You can pipe the output to a second pax instance, though.
pax -w -s ':^[^_]*_::p' *20180721*.jar | (builtin cd ../Incoming; pax -r -s ':1[.]jar$:3.jar:p')
I wrote a shell script that calls the ffmpeg tool but when I run it, it says No such file or directory yet it does!
Here is my script:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
FFMPEG_DIR="/media/sf_data/livraison_transcripts/ffmpeg-git-20180208-64bit-static"
for file in MAIN_DIR/audio_mp3/*.mp3;
do
cp -p file FFMPEG_DIR;
done
for file in FFMPEG_DIR/*.mp3;
do
./ffmpeg -i ${file%.mp3}.ogg
sox $file -t raw --channels=1 --bits=16 --rate=16000 --encoding=signed-
integer --endian=little ${file%.ogg}.raw;
done
for file in FFMPEG_DIR/*.raw;
do
cp -p file MAIN_DIR/pipeline/audio_raw/;
done
and here is the debug response:
cp: cannot stat ‘file’: No such file or directory
./essai.sh: line 14: ./ffmpeg: No such file or directory
sox FAIL formats: can't open input file `FFMPEG_DIR/*.mp3': No such file or
directory
cp: cannot stat ‘file’: No such file or directory
FYI I'm running CentOS7 on VirtualBox
Thank you
Here's a Minimal, Complete, and Verifiable example (MCVE), a version of your script that removes everything not required to show the problem:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
echo MAIN_DIR
Expected output:
/media/sf_data/pipeline
Actual output:
MAIN_DIR
This is because bash requires a $ when expanding variables:
#!/bin/bash
MAIN_DIR="/media/sf_data/pipeline"
echo "$MAIN_DIR"
The quotes are not required to fix the issue, but prevent issues with whitespaces.
Hi You need couple of correction in your shell script see below. To get the actual value assigned to a variable you need to add $ at the front of the variable in shell script.
for file in $"MAIN_DIR"/audio_mp3/*.mp3;
do
cp -p "$file" "$FFMPEG_DIR";
done
for file in "$FFMPEG_DIR"/*.mp3;
./ffmpeg -i ${file%.mp3}.ogg
#provide full path like /usr/bin/ffmpeg
for file in "$FFMPEG_DIR"/*.raw;
do
cp -p "$file" "$MAIN_DIR"/pipeline/audio_raw/;
done
I have a bunch of .sh files in a directory. I need to find out how many stored procedures each one of them calls. What would be the best way to do that?
I'm REALLY new with bash scripts so this is all very new to me. From what I looked online I hacked up a starting point (I think) but I have no idea how I would open each file, and find "something.sql" in it and then out put number of times that was found in each file.
Here's what I have:
#!/bin/sh
for i in 'ls *.sh'
echo -e "\n **** START****"
do
echo -e " \n Filename: $i"
done
echo -e "\n **** END ****"
done
Thanks for any help!
Try this:
grep -nc sql *.sh
See how that moves you. You can add -i if you name sql files in as file.SQL too. Or if they all have a .sql extension.
grep -nc '\.sql' *.sh
For you comment you added, try this:
for i in *.sh
grep -Hc '\.sql' $i
grep '\.sql' $i
done