I am a new bie to shell scripting, here i am trying to find the text and replace the text using the shell script.
what i am trying to do is, actually i have a text file which has 2 strings separated by ":
"
Like this
lorem:ipsum
dola:meru
etc....
my script will take 2 parameters while running.
now the script should check if first parameter is found or not if not found it should add it to text file.
if the first parameter is found, then it should replace the second parameter.
for example
The text file has data like this
lorem:ipsum
dola:meru
caby:cemu
i am running my script with 2 parameters like this
./script.sh lorem meru
So when i run the script it should check if the first parameter found in the file if found, the script should replace the second string..
i.e
I ran the script like this
./script.sh lorem meru
so in the file
lorem:ipsum
dola:meru
caby:cemu
after running the script, in the line
lorem:ipsum
should get replaced to
lorem:meru
here is what i have tried..
#!/bin/sh
#
FILE_PATH=/home/script
FILE_NAME=$FILE_PATH/new.txt
echo $1
echo $2
if [] then
else
echo $1:$2 >> $FILE_NAME
fi
Using sed might be simpler.
$ cat inputfile
lorem:ipsum
dola:meru
caby:cemu
$ pattern="lorem"
$ word="meru"
$ sed "/^$pattern:/ s/:.*/:$word/" inputfile
lorem:meru
dola:meru
caby:cemu
try this line in your script:
awk -F: -v OFS=":" -v s="$1" -v r="$2" '$1==s{$2=r}7' file > newFile
You can try this as well,
input file : new.txt
lorem:ipsum
dola:meru
caby:cemu
Script file: script.sh
#!/bin/sh
sed -i "s/$1:.*/$1:$2/" new.txt
if you run the script as u wished "./script.sh lorum meru"
Output: new.txt, will be displayed as
lorem:meru
dola:meru
caby:cemu
Explanation:
sed is a powerful text processing tool, where u can manipulate the text as you wish with the options it provides.
Brief explanation of the code,
sed -i > used to replace the contents of the original file with the new changes ( if u dont give -i in sed it will just print the modified contents without affecting the original file)
"s/old:.*/old:new/"
"s" is used for substitution. Syntax is "s/old/new/'.
Here * specifies anything that is present after the ":"
So by executing this you will get the desired output.
Related
I have a file like this in content :
#INCLUDE<~/boulou/billy.txt>
And from a bash/sed script/command, I want to replace this line by the content of the file ~/boulou/billy.txt
This is my current command that can find the file path :
sed -E "s/#INCLUDE\<(.*)\>/\1/g" test.sh
This shows me the file path but when I want to fetch the content with this :
sed -E "s/#INCLUDE\<(.*)\>/$(cat \1)/g" test.sh
I get the error "cat: 1: No such file or directory"
Let me know if you want more information.
Your code doesn't work because it first runs cat \1 (which is the same as cat 1), inserts the output into the sed command, then runs sed -E "s/...//g" test.sh (empty replacement because cat 1 outputs nothing on stdout).
This is because the shell processes $( ... ) first, then runs the resulting command line.
For what you're trying to do, you have to read the file from inside the search/replace command, not up front.
I don't know how to do that with sed, so I'd simply use Perl:
perl -pe 's{#INCLUDE<([^<>]*)>}{ open my $fh, "<", $1 or die "$1: $!"; local $/; readline $fh }eg' test.sh
However, this will not work with your example because ~/boulou/billy.txt does not exist (you most likely do not have a directory literally called ~ in your current working directory). To fix this (and simplify the code considerably), I'd use:
perl -MPath::Tiny -pe 's{#INCLUDE<([^<>]*)>}{ path($1)->slurp }eg' test.sh
However, this requires the Path::Tiny module, which is not part of the core perl distribution.
It's possible to expand ~ manually, but that makes the code even messier (this is the point at which I'd consider putting it in a separate script file):
perl -pe 's{#INCLUDE<([^<>]*)>}{ my $p = $1; $p =~ s{^~/}{$ENV{HOME}/}; open my $fh, "<", $p or die "$p: $!"; local $/; readline $fh }eg' test.sh
(Disclaimer: Somewhat hacky, does not handle ~user notation, only replaces a leading ~/ by the contents of the HOME environment variable.)
You can try something like that :
$ sed -r 's/#INCLUDE<(.*)>/printf "%b" "$(cat \1)"/e' test.sh
It should work pretty smooth :)
This might work for you (GNU sed):
sed 's/^#INCLUDE<\(.*\)>$/cat \1/e' file
This evaluates the expression on the right side of the substitution command. In this case it will replace the matched line by the contents of the file named between < and >.
The problem here is that the shell is replacing $(cat \1) way early. That is, even before the sed command is executed and passed that parameters, the shell is doing the interpolation.
If you are just doing #INCLUDE substitutions, you may consider using m4. Awk may be a reasonable way of approaching this if you need to write your own.
This will work robustly with any input file contents running any awk in any shell on any UNIX system:
INPUT:
$ cat file1
before
foo [#INCLUDE<file2>] bar
after
$ cat file2
stuff
& nonsense
here
SCRIPT:
$ cat tst.awk
match($0,/#INCLUDE<[^<>]*>/) {
file = substr($0,RSTART+9,RLENGTH-10)
rep = sep = ""
while ( (getline line < file) > 0 ) {
rep = rep sep line
sep = ORS
}
$0 = substr($0,1,RSTART-1) rep substr($0,RSTART+RLENGTH)
}
{ print }
OUTPUT:
$ awk -f tst.awk file1
before
foo [stuff
& nonsense
here] bar
after
Here's what the currently accepted answer will do with that same input:
$ sed -r 's/#INCLUDE<(.*)>/printf "%s" $(cat \1)/e' file1
before
/bin/sh: foo: command not found
after
and here's what it'll do if the INCLUDE is on a line of it's own:
$ cat file1
before
#INCLUDE<file2>
after
$ sed -r 's/#INCLUDE<(.*)>/printf "%s" $(cat \1)/e' file1
before
stuff&nonsensehere
after
This simpler script would be closer to correct if INCLUDE was on a line of its own:
$ sed -r 's/#INCLUDE<(.*)>/cat \1/e' file1
before
stuff
& nonsense
here
after
but would also fail if any other text was on the line as it'd still try to execute it!
I want to replace a string in linux file with a variable value using Shell Script:
Tried doing this in shell script, but not working.
sed -i "s/$StrVal1/$StrVal2/" "$TargetFile"
Please help.
I think you forgot the "g" in the sed command. Works for me on my linux shell. You can also cat the file and then run the sed command.
First I define variables as shown below, on the prompt:
a='class'; b='CLASS'
Then I cat the file with a sed command:
cat a.txt | sed "s/$a/$b/g"
So my input file is like
class="pending">Count
class="completed">Count
My output with that command is like
CLASS="pending">Count:
CLASS="completed">Count:
So I have a file containing millions of lines.
and now within the file I have occurrences such as
=Continent
=Country
=State
=City
=Street
Now I have an excel file in which I have the text that should replace these occurrences - as an example :
=Continent should be replaced with =Asia
Similarly for other text
Now I was thinking of writing a java program to read my input file , read the mapping file and for each occurrence search and replace.
I am being lazy here - was wondering if I could do the same using editors like VIM ?
would that be possible ?
NOTE - I dont want to do a single text replace - I have multiple text that need to be found and replaced and I dont want to do the search and replace manually for each.
EDIT1:
Contents of my file that I want to replace: "1.txt"
continent=cont_text
country=country_text
The file that contains the values I want to replace with : "to_replace.txt"
=cont_text~Asia
=country_text~India
and finally using 'sed' here is my .sh file - but I am doing something wrong - it does not replace the contents of "1.txt"
while IFS="~" read foo bar;
do
echo $foo
echo $bar
for filename in 1.txt; do
sed -i.backup 's/$foo/$bar/g;' $filename
done
done < to_replace.txt
You can't put $foo and $bar in single quotes because the shell won't expand them. You don't need the for $filename in 1.txt loop because sed will loop through the lines of 1.txt. And you can't use -i.backup inside the loop because it will change the backup file each time and not preserve the original. So your script should be:
#!/bin/bash
cp 1.txt 1.txt.backup
while IFS="~" read foo bar;
do
echo $foo
echo $bar
sed -i "s/$foo/=$bar/g;" 1.txt
done < to_replace.txt
Output:
$ cat 1.txt
continent=Asia
country=India
sed is for simple substitutions on individual lines and shell is an environment from which to call tools not a tool to manipulate text so any time you write a shell loop to manipulate text you are doing it wrong.
Just use the tool that the same guys who invented sed and shell also invented to do general text processing jobs like this, awk:
$ awk -F'[=~]' -v OFS="=" 'NR==FNR{map[$2]=$3;next} {$2=map[$2]} 1' to_replace.txt 1.txt
continent=Asia
country=India
This sed command will do it without any loop:
sed -n 's#\(^=[^~]*\)~\(.*\)#s/\1/=\2/g#p' to_replace.txt |sed -i -f- 1.txt
Or sed with extended regex:
sed -nr 's#(^=[^~]*)~(.*)#s/\1/=\2/g#p' to_replace.txt | sed -i -f- 1.txt
Explanation:
The sed command:
sed -n 's#\(^=[^~]*\)~\(.*\)#s/\1/=\2/g#p' to_replace.txt
generates an output:
s/=cont_text/=Asia/g
s/=country_text/=India/g
which is then used as a sed script for the next sed after the pipe.
$ cat 1.txt
continent=Asia
country=India
I recently wrote a script that does a sed command, to replace all the occurrences of "string1" with "string2" in a file named "test.txt".
It looks like this:
sed -i 's/string1/string2/g' test.txt
The catch is, "string1" does not necessarily exist in test.txt.
I notice after executing a bunch of these sed commands, I get a number of empty files, left behind in the directory, with names that look like this:
"sed4l4DpD"
Does anyone know why this might be, and how I can correct it?
-i is the suffix given to the new/output file. Also, you need -e for the command.
Here's how you use it:
sed -i '2' -e 's/string1/string2/g' test.txt
This will create a file called test.txt2 that is the backup of test.txt
To replace the file (instead of creating a new copy - called an "in-place" substitution), change the -i value to '' (ie blank):
sed -i '' -e 's/string1/string2/g' test.txt
EDIT II
Here's actual command line output from a Mac (Snow Leopard) that show that my modified answer (removed space from between the -i and the suffix) is correct.
NOTE: On a linux server, there must be no space between it -i and the suffix.
> echo "this is a test" > test.txt
> cat test.txt
this is a test
> sed -i '2' -e 's/a/a good/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a good test
> cat test.txt2
this is a test
> sed -i '' -e 's/a/a really/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a really good test
I wasn't able to reproduce this with a quick test (using GNU sed 4.2.1) -- but strace did show sed creating a file called sedJd9Cuy and then renaming it to tmp (the file named on the command line).
It looks like something is going wrong after sed creates the temporary file and before it's able to rename it.
My best guess is that you've run out of room in the filesystem; you're able to create a new empty file, but unable to write to it.
What does df . say?
EDIT:
I still don't know what's causing the problem, but it shouldn't be too difficult to work around it.
Rather than
sed -i 's/string1/string2/g' test.txt
try something like this:
sed 's/string1/string2/g' test.txt > test.txt.$$ && mv -f test.txt.$$ test.txt
Something is going wrong with the way sed creates and then renames a text file to replace your original file. The above command uses sed as a simple input-output filter and creates and renames the temporary file separately.
So after much testing last night, it turns out that sed was creating these files when trying to operate on an empty string. The way i was getting the array of "$string1" arguments was through a grep command, which seems to be malformed. What I wanted from the grep was all lines containing something of the type "Text here '.'".
For example the string, "Text here 'ABC.DEF'" in a file, should have been caught by grep, then the ABC.DEF portion of the string, would be substituted by ABC_DEF. Unfortunately the grep I was using would catch lines of the type "Text here ''" (that is, nothing between the ''). When later on, the script attempted to perform a sed replacement using this empty string, the random file was created (probably because sed died).
Thanks for all your help in understanding how sed works.
Its better if you do it in this way:
cat large_file | sed 's/string1/string2/g' > file_filtred
I am facing a problem in a bash shell script. This script is supposed to execute another shell script (./script here) and the output of the script is redirected to a file(tmp). Then the file should be read line by line and for each of the lines the same script(./script) should be executed giving the line as argument and the result should be stored in a file(tmp1). Eventually these results should be appended to the first file(tmp).
I am pasting my script below:
./script $1 $2 > tmp
cat tmp | while read a
do
./script $a $2 >> tmp1
done
cat tmp1 | while read line
do
./script $line $2 >> tmp
done
I get the following error when I execute the script "./script: line 11: syntax error: unexpected end of file"
Can anyone please help me out in this??
Thanks a lot in advance.
The script file has DOS/Windows line endings. Try this command, then run your script:
dos2unix ./script
You're probably editing the file using a Windows editor and that's adding \r (0x0d) to the end of each line. This can be removed by using dos2unix.
The shell splits on whitespace normally. You can make it split on newlines by writing, towards the top,
IFS='
'
However, I think what you're trying to do might not be appropriate for shell, and might be better served by Perl or Ruby.
lose all the cats! they are unnecessary. And i suppose summ_tmp is an existing file?
#!/bin/bash
set -x
./wiki $1 $2 > tmp
while read -r a
do
./wiki $a $2 >> tmp1
done < summ_tmp
while read -r line
do
./wiki $line $2 >> tmp
done < tmp1
With what you are doing, you might want to refactor your "./script" to eliminate unnecessary steps. If its not too long, show what your "./script" does. Show your desired output and show examples of relevant input files where possible
Put set -x in your script (wiki and ./script) to help you debug.
Alternatively you could use xargs - this executes a command on every line in a file, which is exactly what you want.
You can replace
cat tmp1 | while read line
do
./script $line $2 >> tmp
done
with
xargs <tmp1 -n1 -IXXX ./script XXX $2 >>tmp
-n1 means read one line at a time from the input,
-IXXX means substitute XXX with the line that was read in - the default is to append it to the end of the command line.