How to replace a word in a specific line where the replace pattern contains a variable? - shell
I have the following files:
[root#f9044b5d9d1e aws-marketing-and-sales]# grep -HRe ".*\/common\/.*\${local.parent_folder_name}" *
ap-northeast-1/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
ap-northeast-2/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
ap-south-1/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
ap-southeast-1/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
I'm trying to replace the occurrences of "${local.parent_folder_name}" where the line contains "common" in all files with the parent folder name, like this:
for file in $(grep -HRe ".*\/common\/.*\${local.parent_folder_name}" *); do
filename=$(echo $file | cut -d: -f1)
parent=$(echo $file | rev | cut -d/ -f2 | rev)
sed -i "/common/\${local.parent_folder_name}/$parent/g" $filename
done
This is the error that I get when running the above script:
sed: -e expression #1, char 9: unknown command: `$'
I've found some SO questions regarding this but none of them have examples with using a variable as the replace pattern.
I've also tried different separators (| , !) but to no avail.
Edit:
#moshe, it didn't work, that's the output:
grep -Re "/common\/.*\${local.parent_folder_name}" . | while read -r grep_line; do
if [[ $grep_line == *"$0"* ]]; then
continue
fi
echo $grep_line
filename=$(echo $grep_line | cut -d: -f1)
parent=$(echo $grep_line | rev | cut -d/ -f2 | rev)
echo "parent: $parent"
sed -i "/common/s?\${local.parent_folder_name}?$parent?g" $filename
done
./ca-central-1/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
parent: ${local.parent_folder_name}
sed: 1: "./ca-central-1/config/c ...": invalid command code .
./us-west-2/config/config/terragrunt.hcl: inline_policy = templatefile("${get_parent_terragrunt_dir()}/common/${local.environment}/config/${local.parent_folder_name}/inline-policy-s3.tpl", {
I tried replacing the "." to "8" in the first grep and it worked only on some of the files but not on all. Any idea?
What am I doing wrong?
not sure what the parent var should be, probably should fix it, but the script could look like that
#!/bin/bash
grep -re "/common\/.*\${local.parent_folder_name}" . | while read -r grep_line; do
if [[ $grep_line == *"$0"* ]]; then
continue
fi
echo $grep_line
filename=$(echo $grep_line | cut -d: -f1)
parent=$(echo $grep_line | rev | cut -d/ -f2 | rev)
echo "parent: $parent"
sed -i "/common/s?\${local.parent_folder_name}?$parent?g" $filename
done
Note that sed takes a pattern (/common/) and then a command (a to append, d to delete, s to replace)
We want to change only lines with the pattern /common/ in them,
so after the pattern, we perform a regular search and replace - s/\${local.parent_folder_name}/$parent/g
To make it more readable, I changed the separator / to ?
So the sed is:
sed -i "/common/s?\${local.parent_folder_name}?$parent?g" $filename
Related
Bash Script: Filter large files for value
I have several config files with around 20k lines each and I need to get some values from them.
I know that each of the values I need starts with a specific word "CONFNET" so I tried to get the values with a while loop, which reads every line.
But unfortunately this is extremely inefficient and slow.
Is there a better solution to this?
for filename in ~/configs/*; do
ip=$(cat $filename | strings | grep -i -A 7 "addnet_outside" | head -7 | grep "IP" | sed "s/IP//" | sed "s/=//" | sed -e 's/^[ \t]*//')
hostname=$(cat $filename | strings | grep -a "Inst:" | head -1 | sed "s/Inst://" | sed -e 's/^[ \t]*//')
while IFS= read -r line; do
object_name=$(echo $line | strings | grep "CONFNET" | sed "s/CONFNET//" | awk '{print $1}')
object_value=$(echo $line | strings | grep "CONFNET" | sed "s/CONFNET//" | awk '{print $3}' | sed -e 's/^[ \t]*//')
if [ ! -z $object_name ] && [ ! -z $object_value ]
then
echo $hostname "->" $object_name ":" $object_value
done < "$filename"
done
Text substitution in file while expanding variables, using them as string literals, and adding a newline
I want to replace anything that looks like
if (Thing *thing1 = Stuff) {
with
Thing *thing1 = Stuff;
if (thing1) {
Essentially take declarations out of if statements.
I have all the parts and I see that they're correct, I'm just having trouble with the replacement
grep ' if (.* = .*) {' $file | while read -r line ; do
inside="$( echo "$line" | cut -d "(" -f2 | cut -d ")" -f1 );"
object="$( echo $( echo "$inside" | cut -d "*" -f2 | cut -d "=" -f1 ) | xargs )"
newIF=" if ($object) {"
replacement="$inside\n$newIF"
line_regexp="$(echo "$line" | sed -e 's/[]\/$*.^|[]/\\&/g')"
replacement_regexp="$(echo "$replacement" | sed -e 's/[\/&]/\\&/g')"
sed -i.bak "s/$line_regexp/$replacement_regexp/g" $file
done
Edit thanks to: https://superuser.com/questions/422459/substitution-in-text-file-without-regular-expressions
Now I just have to figure out how to turn "\n" into an actual newline
This sed should work for you:
$ sed -n 's/if (\([^ ]* \*\([^ ]*\) = [^)]*\)) {/\1;\nif (\2) {/p' <<< "if (Thing *thing1 = Stuff) {"
Thing *thing1 = Stuff;
if (thing1) {
exiting an IF statement after initial match bash scripting
I have a script which iterates through a file and finds matches in another file. How to I get the process to stop once I've found a match.
For example:
I take the first line in name.txt, and then try to find a match for it in file.txt.
name.txt:
7,7,FRESH,98,135,
65,10,OLD,56,45,
file.txt:
7,7,Dave,S
8,10,Frank,S
31,7,Gregg
45,5,Jake,S
Script:
while read line
do
name_id=`echo $line | cut -f1,2 -d ','`
identiferOne=`echo $name_id | cut -f1 -d ','`
identiferTwo=`echo $name_id | cut -f2 -d ','`
while IFS= read line
do
CHECK=`echo $line | cut -f4 -d','`
if [ $CHECK = "S" ]
then
symbolName=`echo $line | cut -f3 -d ','`
numberOne=`echo $line | awk -F',' '{print $1}'`
numberTwo=`echo $line | cut -f2 -d ','`
if [ "$numberOne" == $identiferOne ] && [ "$numberTwo" == $identifierTwo ]
then
echo "WE HAVE A MATCH with $symbolName"
break
fi
fi
done < /tmp/file.txt
done < /tmp/name.txt
My question is - how do I stop the script from iterating through file.txt once it has found an initial match, and then set that matched record into a variable, stop the if statement, then do some other stuff within the loop using that variable. I tried using break; but that exits the loop, which is not what I want.
You can tell grep different things: Stop searching after the first match (option -m 1). Read the searchkeys from a file (option -f file). Pretend that the output of a command is a file (not really grep, bash helps here) with <(cmmnd). Combining these will give you grep -m1 -f <(cut -d"," -f1-2 name.txt) file.txt Close, but not what you want. The substrings given by cut -d"," -f1-2 name.txt will match everywhere in the line, and you want to match the first two fields. Matching at the start of the line is done with ^, so we use sed to make strings like ^field1,field2 : grep -m1 -f <(sed 's/\([^,]*,[^,]*,\).*/^\1/' name.txt) file.txt
bash scripting for mysqldump
i use the following code to download all mysql database in a different file and not in one file (like --all-databases) and put them in the /backup/mysql folder #!/bin/bash mysqldump=`which mysqldump` echo $mysqldump mkdir /backup/mysql/$(date '+%d-%b-%Y') echo "creating folder for current date done" for line in "$(mysqlshow |cut -f1 -d"-" | cut -c 3- | cut -f1 -d" ")" do $mysqldump $line > /backup/mysql/$(date '+%d-%b-%Y')/"$line" echo "$line\n" done I used the cut pipes to remove dashes and empty space before and at the end of the database name and it gave me what I want. The problem is at line 13 according to bash but with no more details. Any ideas what I'm doing wrong?
mysqlshow output format +---------------------+ | Databases | +---------------------+ | information_schema | | gitlabhq_production | | mysql | | performance_schema | | phpmyadmin | | test | +--------------------- Your script doesn't manage the first 3 lines nor the last one, so you $line variable is invalid. Solution mysql | tail -n +4 | head -n -1 | tr -d '| ' tail -n +4: skip first four lines (may need adjustement); head -n -1: ignore last line ; tr -d '| ': remove pipe and space. Advices quotes your variables ; use $() instead of backtick ; don't use for i in $(ls *.mp3). read How can I read a file (data stream, variable) line-by-line (and/or field-by-field)? Better solution Instead of a for loop you should use a while with a Process Substitution: while read -r db; do echo "[$db]"; done < <(mysqlshow -u root -p | tail -n +3 | head -n -1 | tr -d ' |' )
how to list file names into a function
I have a folder with a bunch of files, the files only have a url in it i.e
http://itunes.apple.com/us/app/keynote/id361285480?mt=8
Here is my code. How can I get it to do this for each url in each file?
var='{"object":"App","action":"scrape","args":{"itunes_url":"!!!!HERE!!!!"}}'
string=$(echo "$var" | sed -e 's/"/\\"/g')
string='{"request":"'"$string"'"}'
api="http://api.lewis.com"
output=$(curl -s -d "request=$string" "$api")
code=$(echo "$output" | tr '{', '\n' | sed -n "2p" | sed -e 's/:/ /' | awk '{print $2}')
if [ "${code:0:1}" -ne "2" ]; then
# :(
echo "Error: response code $code was returned, "
else
string=$(echo "$output" | tr '{', '\n' | sed -e '/"signature":\(.*\)/d;/"data":\(.*\)/d;/"signature":\(.*\)/d;/"code":\(.*\)/d' |sed -e 's/\\"//g;s/\\\\\\\//\//g;s/\\//g' | tr '}', '\n' | sed -e 's/"//' | sed '/^$/d')
echo "$string"
fi
use a for loop for filename in folder/*; do -- your code where you do something using $filename -- done og if you prefer to give the filenames as arguments to the script then: for filename do -- your code where you do something using $filename -- done then run your script followed by the files ./script.sh folder/*
You could do: for file in *; do for line in $(cat $file); do # Stuff goes here done done Or even just: for line in $(cat *); do # Stuff goes here done