The content is
{
"properties" : {
"CloudSanityPassed" : [ "true" ],
"GITCOMMIT" : [ "test1" ],
"buildNumber" : [ "54" ],
"jobName" : [ "InveergDB-UI" ]
},
"uri" : "http://ergctory:8081/aergergory/api/storage/test-reergerglease-reergpo/cergom/cloergud/waf/ergregBUI/1ergerggregSHOT/ergregerg-34.zip"
}
I use this command
.[] | ."CloudSanityPassed" | .[]
And I get this message
jq: error (at <stdin>:8): Cannot index string with string "CloudSanityPassed"
"true"
exit status 5
I get, what I want ("true" value), but there is a error in output. Could you explain me, how to avoid it and why does it happen?
According to the jq manual, .[] gets the values of the object when applied to object.
So you get two objects, one for value of "properties" and another for value of "uri":
{
"CloudSanityPassed": [
"true"
],
"GITCOMMIT": [
"test1"
],
"buildNumber": [
"54"
],
"jobName": [
"InveergDB-UI"
]
}
"http://ergctory:8081/aergergory/api/storage/test-reergerglease-reergpo/cergom/cloergud/waf/ergregBUI/1ergerggregSHOT/ergregerg-34.zip"
jq tries to apply ."CloudSanityPassed" operator to each object.
Since former object is dictionary (aka hash), you can apply ."CloudSanityPassed" and get the value ["true"], however, latter is a simple string against which you cannot apply ."CloudSanityPassed", so jq outputs an error at that point.
Maybe the command you want is just .properties.CloudSanityPassed.
In my case jq '[.[] | group_by(.foo)]' gave the error but
jq '[.[]] | group_by(.foo)' worked
Related
my requirement is to generate the terraform variables file based on the environment, I'm trying to generate them using bash/shell script, but having difficulty converting the output to terraform HCL language (I can't use JSON because I'm manipulating them further in terraform module)
Current API output (which needs to be converted into HCL):
'cluster_name1' 'REGION1' 'Volume_Size1' 'Instance_Size1'
'cluster_name2' 'REGION2' 'Volume_Size2' 'Instance_Size2'
'cluster_name3' 'REGION3' 'Volume_Size3' 'Instance_Size3'
{...}
Output in CSV format:
"cluster_name1","REGION1","Volume_Size1","Instance_Size1"
"cluster_name2","REGION2","Volume_Size2","Instance_Size2"
"cluster_name3","REGION3","Volume_Size3","Instance_Size3"
{...}
Required format:
variable "cluster_configuration" {
default = [
{
"cluster_name" : "cluster_name1"
"cluster_region" : "REGION1"
"instance_db_size" : "Volume_Size1"
"instance_size" : "Instance_Size1"
},
{
"cluster_name" : "cluster_name2"
"cluster_region" : "REGION2"
"instance_db_size" : "Volume_Size2"
"instance_size" : "Instance_Size2"
},
{
"cluster_name" : "cluster_name3"
"cluster_region" : "REGION3"
"instance_db_size" : "Volume_Size3"
"instance_size" : "Instance_Size3"
},
{....}
]
}
My Terraform code, just for reference:
locals {
dbconfig = [
for db in var.cluster_configuration : [{
instance_name = db.cluster_name
db_size = db.instance_db_size
instance_size = db.instance_size
cluster_region = db.cluster_region
}
]
]
}
I have tried with AWK and SED but have had no luck so far.
Terraform can process JSON data, if you define the objects correctly. You might be able to use jq to format the given CSV data into appropriate JSON for Terraform to consume. If I recall correctly, the filter would be something like
# This *very* much assumes that the quotes aren't actually
# quoting fields that contain a real comma. jq is not suitable
# for robustly parsing CSV data.
[
inputs |
split(",") |
map(ltrimstr("\"")) |
map(rtrimstr("\"")) |
{
cluster_name: .[0],
cluster_region: .[1],
instance_db_size: .[2],
instance_size: .[3]
}
] | {variable: {cluster_configuration: {default: .}}}
(There's probably room for improvement.) Assuming you save that to a file like api.jq and your API output is in output.csv, then
$ jq -nRf api.jq output.csv
{
"variable": {
"cluster_configuration": {
"default": [
{
"cluster_name": "cluster_name1",
"cluster_region": "REGION1",
"instance_db_size": "Volume_Size1",
"instance_size": "Instance_Size1"
},
{
"cluster_name": "cluster_name2",
"cluster_region": "REGION2",
"instance_db_size": "Volume_Size2",
"instance_size": "Instance_Size2"
},
{
"cluster_name": "cluster_name3",
"cluster_region": "REGION3",
"instance_db_size": "Volume_Size3",
"instance_size": "Instance_Size3"
}
]
}
}
}
It might be simpler to pick the language of your choice with a proper CSV parser, though, to generate the JSON.
Solution using jq.
The content is as requested (but the specified formatting is disregarded)
INPUT="
'cluster_name1' 'REGION1' 'Volume_Size1' 'Instance_Size1'
'cluster_name2' 'REGION2' 'Volume_Size2' 'Instance_Size2'
'cluster_name3' 'REGION3' 'Volume_Size3' 'Instance_Size3'
"
jq -srR '
split("\n") | # split lines
map(split(" ") | # split fields
select(any) | # remove emty lines
map(.[1:-1]) | # remove enclosing quotes
{
cluster_name: .[0],
cluster_region: .[1],
instance_db_size: .[2],
instance_size: .[3]
}) |
"variable \"cluster_configuration\" {",
" default = ",
.,
"}"
' <<< "$INPUT"
Output
variable "cluster_configuration" {
default =
[
{
"cluster_name": "cluster_name1",
"cluster_region": "REGION1",
"instance_db_size": "Volume_Size1",
"instance_size": "Instance_Size1"
},
{
"cluster_name": "cluster_name2",
"cluster_region": "REGION2",
"instance_db_size": "Volume_Size2",
"instance_size": "Instance_Size2"
},
{
"cluster_name": "cluster_name3",
"cluster_region": "REGION3",
"instance_db_size": "Volume_Size3",
"instance_size": "Instance_Size3"
}
]
}
I have JSONPath:
$.endpointAgents[?(#.clients.userName=~ 'a')].agentId
How it will look in jq format on Linux??
jq '.endpointAgents [] | select(.clients.userName=~"a") | {agentId}')"
does not work.
Code:
{
"endpointAgents": [
{
"agentId": "MyId",
"agentName": "MYNAME",
"location": {
"locationName": "location"
},
"clients": [
{
"userProfile": {
"userName": "Name"
},
"browserExtensions": [
{
"active": false
}
]
}
],
"totalMemory": "16222 MB",
"agentType": "enterprise"
}
]
}
I want to get userName value by specifying agentId
jq '.endpointAgents[] | select(.agentId == "MyId") | .clients[].userProfile.userName'
Will output "Name"
.endpointAgents[]
Loop over each endpointAgent
select(.agentId == "MyId")
Select the objects where .agentId == "MyId"
.clients[].userProfile.userName
Since clients is an array, loop over it, and show .userProfile.userName for each object.
Try it online!
If I understand correctly, you want to produce the agent id of endpoints with a client whose name matches a.
.endpointAgents[] |
select( any( .clients[].userProfile.userName; test("a") ) ) |
.agentId
To produce the agent id of endpoints with a client whose name is equal to a, use the following instead:
.endpointAgents[] |
select( any( .clients[].userProfile.userName; . == "a" ) ) |
.agentId
I have to get the name of companies in which 'John' worked in the 'sales' department. My JSON
looks like this:
[
{
"name" : "John",
"company" : [{
"name" : "company1",
"department" : "sales"
},
{
"name" : "company2",
"department" : "backend"
},
{
"name" : "company3",
"department" : "sales"
}
],
"phone" : "1234"
}
]
And my jmesquery is like this:
jmesquery: "[? name=='John'].company[? department=='sales'].{Company: name}"
But with this query, I'm getting a null array.
This is because your first filter [?name=='John'] is creating a projection, and more specifically a filter projection, that you will have to reset in order to further filter it.
Resetting a projection can be achieved using pipes.
Projections are an important concept in JMESPath. However, there are times when projection semantics are not what you want. A common scenario is when you want to operate of the result of a projection rather than projecting an expression onto each element in the array.
For example, the expression people[*].first will give you an array containing the first names of everyone in the people array. What if you wanted the first element in that list? If you tried people[*].first[0] that you just evaluate first[0] for each element in the people array, and because indexing is not defined for strings, the final result would be an empty array, []. To accomplish the desired result, you can use a pipe expression, <expression> | <expression>, to indicate that a projection must stop.
Source: https://jmespath.org/tutorial.html#pipe-expressions
So, here would be a first step in your query:
[?name=='John'] | [].company[?department=='sales'].{Company: name}
This said, this still ends in an array of array:
[
[
{
"Company": "company1"
},
{
"Company": "company3"
}
]
]
Because you can end up with multiple people named John in a sales department.
So, one array for the users and another for the companies/departments.
In order to fix this, you can use the flatten operator: [].
So we end with:
[?name=='John'] | [].company[?department=='sales'].{Company: name} []
Which gives:
[
{
"Company": "company1"
},
{
"Company": "company3"
}
]
I'm trying to filter the following json object to get the id which doesn't have a domain value which is either "moderator" or "owner" using jq in bash script.The problem is my third domain value is random and not predetermined. How do I filter it?
{
"data":[
{
"id":1,
"domain":"moderator"
},
{
"id":2,
"domain":"owner"
},
{
"id":3,
"domain":"34b5756175a848f7a1395e1a19e10602"
}
]
}
You can get just the elements of the array with a different domain via:
$ jq '.data[] | select(.domain != "moderator" and .domain != "owner")' input.json
{
"id": 3,
"domain": "34b5756175a848f7a1395e1a19e10602"
}
If you just want the id value and not the entire object, add | .id to the end of the jq filter.
You can use the jq function select
cat text.json | jq '.data[] | select((.domain != "moderator") and (.domain != "owner"))'
Also, you can use the length function to get the size of the array.
jq '.[] | length' test.json
I have a file like below. As you can see there are few lines/contents between curly braces. As there is multiple group of opened and closed curly braces, I want to get the content between the curly brances ({ and } ) for each line separatly.
Sample file:
{
"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml": [
]
}
{
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml": [
".spec.replicas: '2' != '0'",
],
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml": [
"error: the server doesn't have a resource type \"ServiceMonitor\"\n"
]
}
{
"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml": [
"error: the server doesn't have a resource type \"Issuer\"\n",
"error: the server doesn't have a resource type \"Certificate\"\n"
]
}
Expected result: Need 3 seperated data chunks which is between the curly braces.
Could someone help me here?
If you have a sequence of valid JSON objects, you can use jq to easily and robustly process them:
Given file.jsons:
{
"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml": [ ""
]
}
{
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml": [
".spec.replicas: '2' != '0'"
],
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml": [
"error: the server doesn't have a resource type \"ServiceMonitor\"\n"
]
}
{
"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml": [
"error: the server doesn't have a resource type \"Issuer\"\n",
"error: the server doesn't have a resource type \"Certificate\"\n"
]
}
You can for example reformat each object as a single line:
$ jq -s -r 'map(#json) | join("\n")' < file.jsons
{"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml":[""]}
{"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml":[".spec.replicas: '2' != '0'"],"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml":["error: the server doesn't have a resource type \"ServiceMonitor\"\n"]}
{"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml":["error: the server doesn't have a resource type \"Issuer\"\n","error: the server doesn't have a resource type \"Certificate\"\n"]}
Now you can process it line by line without having to worry about matching up curly braces.
Thank you for your suggestion, the above jq would not work for all the json payload . For example for below json payload it is giving an error
{
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-deployment.yaml": [
".spec.replicas: '2' != '3'",
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-metrics-service.yaml": [
".spec.clusterIP: '' != '10.3.24.53'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-service.yaml": [
".spec.clusterIP: '' != '10.3.115.118'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-stats-service.yaml": [
".spec.clusterIP: '' != '10.3.115.30'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/default-backend-deployment.yaml": [
]
}