Related
I have a YAML file which I am converting into JSON files using yq.
It generates the following output,
{
"a" : 1,
"b" : 2,
"c" : {
"id": "9ee ...",
"parent": "abc..."
}
}
Then I am creating another JSON based on the above JSON key & value. Please find the below code snippet,
# Extract the properties from the YAML file
json=$(yq -j "$file_name")
# Iterate over the properties
parameters=""
for key in $(echo "${json}" | jq -r 'keys[]'); do
# Extract the key and value of the property
value=$(echo "${json}" | jq -r ".$key")
echo "Adding parameter $key with value $value to SSM"
# Add the property to the list of parameters
parameters+="{\"Name\": \"$key\", \"Value\": \"$value\", \"Type\": \"String\", \"Overwrite\": true}"
done
Since the value in the 1st JSON is already JSON so we couldn't able to generate the 2nd JSON which is generating an invalid-JSON error.
Is there any way to escape/stringify the JSON characters in bash script or jq, so that we can able to generate the 2nd JSON?
Any help would be really appreciated.
Actual output:
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "c",
"Value": "{
"id": "9ee ...",
"parent": "abc..."
}",
"Type": "String",
"Overwrite": "true"
}
]
The above one is not a valid JSON.
Expected output:
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "c",
"Value": "{\r\n \"id\": \"9ee ...\",\r\n \"parent\": \"abc...\"\r\n }",
"Type": "String",
"Overwrite": "true"
}
]
Why try to emulate jq's behavior with a shell loop (which should generally be avoided) instead of using jq directly?
yq -j "$file_name" | jq 'to_entries | map({
Name: .key,
Value: .value,
Type: "String",
Overwrite: true
})'
Or directly transform using yq only:
yq 'to_entries | map({
Name: .key,
Value: .value,
Type: "String",
Overwrite: true
})' -j "$file_name"
Update after clarifying edit of the question: It has become clear that you want to transform the value into a string. jq has the tostring filter for that, the program thus becomes:
to_entries | map({
Name: .key,
Value: (.value | tostring),
Type: "String",
Overwrite: true
})
Note that this will not keep the line breaks and indents, but formats the JSON object in a "compact" way. Let us know if that's a problem.
$ jq 'to_entries | map({
Name: .key,
Value: (.value | tostring),
Type: "String",
Overwrite: true
})' <<JSON
{
"a": 1,
"b": 2,
"c": {
"id": "9ee ...",
"parent": "abc..."
}
}
JSON
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": true
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": true
},
{
"Name": "c",
"Value": "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}",
"Type": "String",
"Overwrite": true
}
]
This should achieve what's expected :
jq 'to_entries | map({
Name: .key,
Value: (.value|if (type == "object")
then tojson
else tostring
end),
Type: "String",
Overwrite: true})
' input.json
To JSON-encode data from within jq you can use the tojson (or #json) builtins.
Also, to get the actual type of some data, there is the type function.
Maybe you are trying to accomplish something like this:
# Convert the the YAML file into JSON
json="$(yq -j "$file_name")"
# Transform the JSON data into desired format
jq 'to_entries | map({Name: .key} + (.value | {
EncodedValue: tojson,
OriginalType: type,
Overwrite: true
}))' <<< "$json" > "$new_file_name"
Checking the contents of the new file would now give you something like:
[
{
"Name": "a",
"EncodedValue": "1",
"OriginalType": "number",
"Overwrite": true
},
{
"Name": "b",
"EncodedValue": "2",
"OriginalType": "number",
"Overwrite": true
},
{
"Name": "c",
"EncodedValue": "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}",
"OriginalType": "object",
"Overwrite": true
}
]
I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty
I have a JSON file that I want to convert into a CSV file using the jq in a shell script. I want to create a single row from this entire JSON file. I have to extract value from values. The row output should be something like
null,642,642,412,0,null,null
Here is my JSON file
{
"data": [
{
"name": "exits",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Exits",
"description": "Number of times someone exited the carousel"
},
{
"name": "impressions",
"period": "lifetime",
"values": [
{
"value": 642
}
],
"title": "Impressions",
"description": "Total number of times the media object has been seen"
},
{
"name": "reach",
"period": "lifetime",
"values": [
{
"value": 412
}
],
"title": "Reach",
"description": "Total number of unique accounts that have seen the media object"
},
{
"name": "replies",
"period": "lifetime",
"values": [
{
"value": 0
}
],
"title": "Replies",
"description": "Total number of replies to the carousel"
},
{
"name": "taps_forward",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Taps Forward",
"description": "Total number of taps to see this story's next photo or video"
},
{
"name": "taps_back",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Taps Back",
"description": "Total number of taps to see this story's previous photo or video"
}
]
}
Hi tried using this jq command :
.data | map(.values[].value) | #csv
This is giving the following output:
jq: error (at :70): object ({}) is not valid in a csv row
exit status 5
So when I am getting this empty JSON object it is reflecting an error.
Please Help!!
The row output should be something like
null,642,642,412,0,null,null
Using length==0 here is dubious at best. To check for {} one could write:
jq '.data | map(.values[].value | if . == {} then "null" else . end) | #csv'
Similarly for [].
If you run the command without the #csv part you will see that the output is:
[
{},
642,
412,
0,
{},
{}
]
By replacing the empty objects with "null": (length == 0)
jq '.data | map(.values[].value) | map(if (type == "object" and length == 0 ) then "null" else . end) | #csv'
Output:
"\"null\",642,412,0,\"null\",\"null\""
Per suggestion from #aaron (see comment). The following can produce the requested output without extra post-processing. Disclaimer: this is not working with my jq 1.5, but working on jqplay with jq 1.6.
jq --raw-output '.data | map(.values[].value) | map(if (type == "object" and length == 0 ) then "null" else . end) | join(",")'
Output:
null,642,412,0,null,null
Have a big json like this
"envConfig": {
"environmentName": {
"versions": [
{
"name": "version1",
"value": "Dev"
},
{
"name": "version2",
"host": "qa"
}
],
"userRoles": [
{
"name": "Roles",
"entry": [
{
"name": "employees",
"value": "rwx"
},
{
"name": "customers",
"value": "rx"
}
]
}
]
}
},
I wanted to change the JSON attribute from "environmentName" to "prod". Below is the output i am expecting
"envConfig": {
"prod": {
"versions": [
...
],
"userRoles": [
...
]
}
}
Tried with sed command as below
sed "s/\('environmentName':\)/\1\"prod\"\,/g" version.json
Tried with jq as below but not working
cat version.json | jq ' with_entries(.value |= {"prod" : .environmentName} ) '
Any help here to replace the attribute/key of an json with desired value
You weren't too far off with the jq, how about this?
jq '.envConfig |= with_entries(.key |= sub("^environmentName$"; "prod"))'
Two differences: first off, we want to drill down to envConfig before doing a with_entries, and second, when we get there, the thing we want will be a key, not a value. In case there are any other keys besides environmentName they'll be preserved.
TL,TR
You can use the following command:
jq '(.envConfig |= (. + {"prod":.environmentName}|del(.environmentName)))' foo.json
Let's say you have the following json:
{
"foo": {
"hello" : "world"
}
}
You can rename the node foo to bar by first duplicating it and then remove the original node:
jq '. + {"bar":.foo}|del(.foo)' foo.json
Output:
{
"bar": {
"hello" : "world"
}
}
It get's a bit more complicated if you want to replace a child key somewhere in the tree. Let's say you have the following json:
{
"test": {
"foo": {
"hello": "world"
}
}
}
You can use the following jq command for that:
jq '(.test |= (. + {"bar":.foo}|del(.foo)))' foo.json
Note the additional parentheses and the use of the assignment operator |=.
Output:
{
"test": {
"bar": {
"hello": "world"
}
}
}
Using sed:
sed -i '/^ \"environmentName\":/ s/environmentName/prod/' <yourfile>
Keep in mind that -i will overwrite the file. You may want to make a backup first.
This question already has an answer here:
How to convert a JSON object stream into an array with jq
(1 answer)
Closed 6 years ago.
Say I have the input:
{
"name": "John",
"email": "john#company.com"
}
{
"name": "Brad",
"email": "brad#company.com"
}
How do I get the output:
[
{
"name": "John",
"email": "john#company.com"
},
{
"name": "Brad",
"email": "brad#company.com"
}
]
I tried both:
jq '[. | {name, email}]'
and
jq '. | [{name, email}]'
which both gave me the output
[
{
"name": "John",
"email": "john#company.com"
}
]
[
{
"name": "Brad",
"email": "brad#company.com"
}
]
I also saw no options for an array output in the documentations, any help appreciated
Use slurp mode:
o --slurp/-s:
Instead of running the filter for each JSON object
in the input, read the entire input stream into a large
array and run the filter just once.
$ jq -s '.' < tmp.json
[
{
"name": "John",
"email": "john#company.com"
},
{
"name": "Brad",
"email": "brad#company.com"
}
]