Extracting json payload in shell script - shell

Question

I have a file like below. As you can see there are few lines/contents between curly braces. As there is multiple group of opened and closed curly braces, I want to get the content between the curly brances ({ and } ) for each line separatly.
Sample file:
{
"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml": [
],
"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml": [
]
}
{
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml": [
".spec.replicas: '2' != '0'",
],
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml": [
"error: the server doesn't have a resource type \"ServiceMonitor\"\n"
]
}
{
"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml": [
"error: the server doesn't have a resource type \"Issuer\"\n",
"error: the server doesn't have a resource type \"Certificate\"\n"
]
}
Expected result: Need 3 seperated data chunks which is between the curly braces.
Could someone help me here?

Answer 1

If you have a sequence of valid JSON objects, you can use jq to easily and robustly process them:
Given file.jsons:
{
"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml": [ ""
],
"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml": [ ""
]
}
{
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml": [
".spec.replicas: '2' != '0'"
],
"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml": [
"error: the server doesn't have a resource type \"ServiceMonitor\"\n"
]
}
{
"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml": [
"error: the server doesn't have a resource type \"Issuer\"\n",
"error: the server doesn't have a resource type \"Certificate\"\n"
]
}
You can for example reformat each object as a single line:
$ jq -s -r 'map(#json) | join("\n")' < file.jsons
{"/tmp/©ƒ-4bf57ed2-velero/velero/templates/crds.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/deployment.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/restic-daemonset.yaml":[""],"/tmp/velero-4bf57ed2-velero/velero/templates/secret.yaml":[""]}
{"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/deployment.yaml":[".spec.replicas: '2' != '0'"],"/tmp/autoscaler-fb12fa7a-cluster-autoscaler/cluster-autoscaler/templates/servicemonitor.yaml":["error: the server doesn't have a resource type \"ServiceMonitor\"\n"]}
{"/tmp/metrics-server-1960953a-metrics-server-certs/raw/templates/resources.yaml":["error: the server doesn't have a resource type \"Issuer\"\n","error: the server doesn't have a resource type \"Certificate\"\n"]}
Now you can process it line by line without having to worry about matching up curly braces.

Answer 2

Thank you for your suggestion, the above jq would not work for all the json payload . For example for below json payload it is giving an error
{
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-deployment.yaml": [
".spec.replicas: '2' != '3'",
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-metrics-service.yaml": [
".spec.clusterIP: '' != '10.3.24.53'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-service.yaml": [
".spec.clusterIP: '' != '10.3.115.118'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/controller-stats-service.yaml": [
".spec.clusterIP: '' != '10.3.115.30'"
],
"/tmp/ingress-dae7bd30-ingress-internet/nginx-ingress/templates/default-backend-deployment.yaml": [
]
}