Bash: using sed to replace block of code, treating indents - bash

There's a file, code looks like that:
{
"name": {
"port": 4466,
"host": "localhost",
"appPort": 3555,
"abc": {
"defg": {
"host": "localhost",
"port": 5500,
"userName": "test",
"password": "test"
}
},
"name2": {
"port": 4321,
"host": "localhost",
"appPort": 1234,
"abc": {
"defg": {
"host": "localhost",
"port": 5500,
"userName": "test",
"password": "test"
}
},
etc.
I need to replace the following block of code in a file using sed (or maybe awk):
"name": {
"port": 4466,
"host": "localhost",
"appPort": 3555
Here's what I have, did not succeed:
confSearch='"name": {\n "port": 4466,\n "host": "localhost",\n "appPort": 3555'
confReplacement='"name": {\n "port": 4466,\n "host": "10.20.30.40",\n "appPort": 3555'
sed -i "s|$confSearch|$confReplacement|g" "$configFile"
Then I tried the following (to search from "name" to "3555" and then replace it):
sed -i "/name.*/ {N; s/name.*3555\./$confReplacement/g}" "$configFile"
I don't receive any error, the search text is simply not found. I suppose, that's because of indents. How to treat indents correctly? Or should I prefer something else? Taking into account only bash, not perl.
Would be grateful for the help.

I'd advise against using sed for this task; JSON is not a line-based format, and sed is ill-equipped to handle it unless you are very certain that the file will always be formatted just the right way. There are tools out there that parse JSON properly and work on the object it encodes, so indentation or swapped lines or having several nodes on the same line does not faze them.
In this particular case, I'd suggest jq:
jq 'if .name.port == 4466 and .name.appPort == 3555 and .name.host == "localhost" then .name.host="10.20.30.40" else . end' "$configFile"

sed -i '/"port": 4466,/,/"host": "localhost",/ s/localhost/110.20.30.40/' "$configFile"
assuming there is only 1 host with port 4466

it seems you just want to replace the host, won't you? use something like
sed -i 's/"host":.*$/"host": "10.20.30.40"\n/'.

Related

Assigning the values to array from jq and iterate over the values using cb commands

I have the below json in a file from which I wanted to take the "ids" from all price object and put into an array variable.
{
"documentType": "Prices",
"fullCharges": [
{
"ResourceId": null,
"price": {
"href": null,
"id": "8ddaaabc92bc"
},
"product": {
"href": null,
"id": "123"
}
},
{
"price": {
"href": null,
"id": "326f0f273258"
},
"product": {
"href": null,
"id": "123"
}
}
],
"createdBy": "test",
"createdOn": "2021-10-05T00:00:55Z",
"currentSeqNum": 2
}
I am using the below query but the result is coming in the proper way.
priceIds=$(jq -r .fullCharges[].price.id ${file})
Using above command, it is behaving like a single value, not like array. If I print the priceId value it is only showing the last value.
326f0f273258 instead of 8ddaaabc92bc 326f0f273258
And when I am looping over it, again it is behaving as a single value.
for price in "${priceIds[#]}"
do
printf "$price"
cbq -u Administrator -p Administrator -e "http://localhost:8093" --script="select * FROM \`com.src.test.price\` where docId==\"$price\";"
done
Output command of above loop: select * FROM `com.src.test.price` where documentId=="8ddaaabc92bc 326f0f273258";
There should be 2 command like these
select * FROM `com.src.test.price` where documentId=="8ddaaabc92bc"
select * FROM `com.src.test.price` where documentId=="326f0f273258"
Using extra parens, it helped me to create array.
Using the below sed command over array item variable, i was able to remove extra line from it.
price=$(echo "$price" | sed 's/^[ \n]*//;s/[ \n]*$//')

How to get first one "id" in a JSON using sed in shell script?

'''[ {
"id": "**49b18e99-7516-4c9a-8e1a-9d28f470512**",
"name": "Name",
"appId": "",
"publishTime": "2020-05-17",
"published": true,
"stream": {
"id": "aaec8d41-5201-43ab809f-3063750dffff",
"name": "name",
"privileges": null
},
"savedInProductVersion": "12.0",
"migrationHash": "qewjhjkw-djqwhndj",
"availabilityStatus": 1,
"privileges": null
}, {
"id": "**59b18e99-7516-4c9a-8e1a-9d28f4705123**",
"name": "Name",
"appId": "",
"publishTime": "2020-05-17",
"published": true,
"stream": {
"id": "abec8d41-5201-43ab809f-3063750deeee",
"name": "name",
"privileges": null
},
"savedInProductVersion": "12.0",
"migrationHash": "qewjhjkw-djqwhndj",
"availabilityStatus": 1,
"privileges": null
}
]'''
I want this 49b18e99-7516-4c9a-8e1a-9d28f470512 and 59b18e99-7516-4c9a-8e1a-9d28f4705123 , but using "sed -n 's|.*"id":"\([^"]*\)".*|\1|p'"
I am getting
49b18e99-7516-4c9a-8e1a-9d28f470512,
aaec8d41-5201-43ab809f-3063750dffff,
59b18e99-7516-4c9a-8e1a-9d28f4705123,
abec8d41-5201-43ab809f-3063750deeee
if file have multiple json objects like above given.
Then please help me to find the best command for the above-using sed only.
Using any POSIX awk:
awk '
/{ *$/ { depth++ }
/^ *}/ { depth-- }
(depth == 1) && sub(/^[[:space:]]*"id": *"/,"") && sub(/", *$/,"")
' file
**49b18e99-7516-4c9a-8e1a-9d28f470512**
**59b18e99-7516-4c9a-8e1a-9d28f4705123**
With GNU awk and with your shown samples, please try following awk code. Written and tested in GNU awk. Simple explanation would be, setting RS as "id":[ till 1st occurrence of }. Then using match function to match exact value needed by OP by using regex "id":[[:space:]]+"[^"]* and printing the values then.
awk -v RS='"id":[^}]*' '
match(RT,/"id":[[:space:]]+"[^"]*/){
val=substr(RT,RSTART,RLENGTH)
sub(/.*"/,"",val)
print val
}
' Input_file

change varible inside json file using bash [duplicate]

This question already has answers here:
How do I use sed to change my configuration files, with flexible keys and values?
(8 answers)
Closed 4 years ago.
I have total 3 environment dev, stag and Prod all environment have config.json like this
{
"braintree": {
"merchantid": "MERCHANTID",
"publickey": "PUBLICKEY",
"privatekey": "PRIVATEKEY"
},
"karix": {
"url": "URL",
"pass": "PASS",
"user": "USER"
},
"fikarix": {
"source": "SOURCE"
},
"mailgun": {
"api_key": "API_KEY",
"domain": "DOMAIN",
"apikey": "APIKEY"
},
"paymentrails": {
"key": "KEY",
"environment": "ENVIRONMENT",
"secret": "SECRET"
}
}
Now I want to convert it into like this for all environment using shell script
dev environment config.json
{
"braintree": {
"merchantid": "dev_MERCHANTID",
"publickey": "dev_PUBLICKEY",
"privatekey": "dev_PRIVATEKEY"
},
"karix": {
"url": "dev_URL",
"pass": "dev_PASS",
"user": "dev_USER"
},
"fikarix": {
"source": "dev_SOURCE"
},
"mailgun": {
"api_key": "dev_API_KEY",
"domain": "dev_DOMAIN",
"apikey": "dev_APIKEY"
},
"paymentrails": {
"key": "dev_KEY",
"environment": "dev_ENVIRONMENT",
"secret": "dev_SECRET"
}
}
How I can get this using sed or any other solution?
sed 's/: "/: "dev_/g' config.json
using sed u can do
Edit:
To insert pass -i
sed -i 's/: "/: "dev_/g' config.json

Create a json file using jq and multiple variables

I have a json file like this:
{
"name": "Job",
"type": "xdb",
"typeLogoUrl": "public/app/plugins/logo.svg",
"access": "proxy",
"url": "http://xdb:80",
"password": {},
"user": "xx",
"database": "Job",
"basicAuth": true,
"basicAuthUser": "xx",
"basicAuthPassword": {},
"withCredentials": true,
"isDefault": false,
"jsonData": {},
"secureJsonFields": null
}
Now all I want is to pass environment variables to key password and basicAuthPassword and to generate new json file as below:
{
"name": "Job",
"type": "xdb",
"typeLogoUrl": "public/app/plugins/logo.svg",
"access": "proxy",
"url": "http://xdb:80",
"password": "password" ,
"user": "xx",
"database": "Job",
"basicAuth": true,
"basicAuthUser": "xx",
"basicAuthPassword": "password",
"withCredentials": true,
"isDefault": false,
"jsonData": {},
"secureJsonFields": null
}
I have tried like below to replace at least one key and it's giving me null.
/usr/local/bin/jq -n --arg "password" '.password = $arg' < input.json
Can anyone suggest me how to achieve this?
Security wise, it's a bad idea to pass password in command line, but I hope you know what you're doing.
Assuming that, you need to modify your jq command like this:
jq --arg p "password" '.password = $p | .basicAuthPassword = $p' < input.json
If the password is already an environment variable (which may not be such a good idea though), then to avoid exposing it on the command-line, you could use the env builtin, along the lines of:
.password = env.password
etc.
An alternative to consider would be putting the password into a temporary file, and then using the --argfile command-line option, for example. Or if your shell supports it, --argfile pw <(echo "\"$password\"")

Sed command failing because of / in replacement pattern

I have this ECS task definition as follow:
{
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/foo:1.0",
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/bar:latest",
....
}
I need to replace only the first "image" value, for instance:
{
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/foo:2.0",
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/bar:latest",
....
}
Here's my command sed -e "s/.*foo:.*/\"image\":\"${REPO}:${VERSION}\",/" taskdef.json
Where REPO=123.dkr.ecr.us-east-1.amazonaws.com/foo and VERSION=2.0
This is the error I got:
sed: -e expression #1, char 70: unknown option to `s'
This happens because the slash / from REPO variable.
You can use any character as the delimiter for `s' commands in sed, the first character after s will be the delimiter. For example - #
sed -e "s#foo:.*#\"image\":\"${REPO}:${VERSION}\",#" taskdef.json
Will resolve this particular issue (assuming no # in $REPO or $VERSION) as the / will no longer break the pattern.
To replace the value for the first image would be:
$ awk -v repo="$REPO" -v vers="$VERSION" '
!f && ($1~/"image"/) { f=1; sub(/:.*/,""); $0=$0 ": \"" repo ":" vers "\"," } 1
' file
{
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/foo:2.0",
...
"image": "123.dkr.ecr.us-east-1.amazonaws.com/bar:latest",
....
}
The above would convert escape sequences to their literal characters (e.g. \t to a literal tab character) if they appeared in REPO or VERSION. It's a trivial workaround if that's a possible issue (just set them on the command line or export them then access with ENVIRON[]) and it'll work no matter what other characters appear in the strings since it's using literal string functionality.
The right way with json processor called jq (v1.5):
Sample ECS task definition task.json:
{
"containerDefinitions": [
{
"name": "wordpress",
"links": [
"mysql"
],
"image": "123.dkr.ecr.us-east-1.amazonaws.com/foo:1.0",
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"memory": 500,
"cpu": 10
},
{
"environment": [
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "password"
}
],
"name": "mysql",
"image": "123.dkr.ecr.us-east-1.amazonaws.com/bar:latest",
"cpu": 10,
"memory": 500,
"essential": true
}
],
"family": "hello_world"
}
The job:
jq '.containerDefinitions[0].image = (.containerDefinitions[0].image | sub("1.0$";"2.0"))' task.json
The output:
{
"containerDefinitions": [
{
"name": "wordpress",
"links": [
"mysql"
],
"image": "123.dkr.ecr.us-east-1.amazonaws.com/foo:2.0",
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"memory": 500,
"cpu": 10
},
{
"environment": [
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "password"
}
],
"name": "mysql",
"image": "123.dkr.ecr.us-east-1.amazonaws.com/bar:latest",
"cpu": 10,
"memory": 500,
"essential": true
}
],
"family": "hello_world"
}

Resources