I need to parse out specific information from the following JSON example bellow using jq. depending on the 'repo' I need to be able to parse from the properties array all 'key' & 'values' and associated with the repository. For example the repo "libs-production-local" would need to parse out any properties key with 'prod' in the string and its associated date value. has to be jq and run in .sh.
{
"results":[
{
"repo":"libs-production-local",
"path":"com/company/version",
"name":"sql.21.tar",
"type":"file",
"size":"40123",
"created":"date",
"created_by":"someone",
"modified":"date",
"modified_by":"someone",
"updated":"date",
"depth":4,
"actual_md5":"asdflsdf23a4324234",
"orginal_sha1":"sadlkfjsdklfjsadf",
"properties":[
{
"key":"deploy.uat",
"value":"2018-09-23"
},
{
"key":"deploy.prod.TLE",
"value":"2018-10-20"
},
{
"key":"deploy.prodXYZ",
"value":"2018-10-20"
},
{
"key":"deploy.prodPDQ",
"value":"2018-10-20"
},
{
"key":"deploy.prod.ABC",
"value":"2018-10-21"
},
{
"key":"businessUnit.name",
"value":"IndivdualName"
},
{
"key":"deploy.qa.ser2",
"value":"2018-10-20"
},
{
"key":"deploy.qa.ser1",
"value":"2018-11-23"
},
{
"key":"build.timestamp",
"value":"1510850899004"
}
],
"virtual_repos":[
"libs-production "
]
},
{
"repo":"libs-production-local",
"path":"com/company/version",
"name":"sql.22.tar",
"type":"file",
"size":"40123",
"created":"date",
"created_by":"someone",
"modified":"date",
"modified_by":"someone",
"updated":"date",
"depth":4,
"actual_md5":"asdflsdf23a4324234",
"orginal_sha1":"sadlkfjsdklfjsadf",
"properties":[
{
"key":"deploy.prodPDQ",
"value":"2018-10-22"
},
{
"key":"deploy.prodABC",
"value":"2018-10-20"
},
{
"key":"businessUnit.name",
"value":"IndivdualName"
},
{
"key":"deploy.qa",
"value":"2018-10-20"
},
{
"key":"deploy.dev",
"value":"2018-11-19"
}
],
"virtual_repos":[
"libs-production "
]
}
],
"range":{
"start_pos":0,
"end_pos":479,
"total":479
}
}
I've tried a number of ways to do this (including this one) and nothing works.
jq -r '.results[] | ( .properties |map(select(.key[] contains ("prod")) '
I solved it like this:
jq -r '[ .results[].properties[] | select(.key | contains("prod")) ]'
This grabs all key-value pairs from each result's properties array. It then selects those that contain "prod" in the key, and returns an array of those keys and values. Given your example input from above, this is the return value:
[
{
"key": "deploy.prod.TLE",
"value": "2018-10-20"
},
{
"key": "deploy.prodXYZ",
"value": "2018-10-20"
},
{
"key": "deploy.prodPDQ",
"value": "2018-10-20"
},
{
"key": "deploy.prod.ABC",
"value": "2018-10-21"
},
{
"key": "deploy.prodPDQ",
"value": "2018-10-22"
},
{
"key": "deploy.prodABC",
"value": "2018-10-20"
}
]
Is that close to what you're looking for?
Related
Using jq I am trying to convert the rawe json below into the desired json outcome.
Objectives:
name renamed to pathParameterName
type renamed to datasetParameter
Raw Json I'm trying to convert
{
"pathOptions": {
"parameters": {
"raw_date": {
"name": "raw_date",
"type": "Datetime",
"datetimeOptions": {
"localeCode": "en-GB"
},
"createColumn": true,
"filter": {
"expression": "(after :date1)",
"valuesMap": {
":date1": "2022-03-08T00:00:00.000Z"
}
}
}
}
}
}
Json desired outcome:
{
"pathOptions": {
"parameters": [
{
"pathParameterName": "raw_date",
"datasetParameter": {
"name": "raw_date",
"type": "Datetime",
"datetimeOptions": {
"localeCode": "en-GB"
},
"createColumn": true,
"filter": {
"expression": "(after :date1)",
"valuesMap": [
{
"valueReference": ":date1",
"value": "2022-03-08T00:00:00.000Z"
}
]
}
}
}
]
}
}
This is what I have so far:
map_values(if type == "object" then to_entries else . end)
This is what my code above currently produces. -I'm struggling with the key renaming.
{
"pathOptions": [
{
"key": "parameters",
"value": [
{
"pathParameterName": "raw_date",
"datasetParameter": {
"name": "raw_date",
"type": "Datetime",
"datetimeOptions": {
"localeCode": "en-GB"
},
"createColumn": true,
"filter": {
"expression": "(after :date1)",
"valuesMap": [
{
"valueReference": ":date1",
"value": "2022-03-08T00:00:00.000Z"
}
]
}
}
}
]
}
]
}
The function to_entries, "converts between an object and an array of key-value pairs" (see the manual). To rename the preset key and value fields, just reassign them to a new name with a new object as in {valueReference: .key, value}.
jq '
.pathOptions.parameters |= (
to_entries | map({
pathParameterName: .key,
datasetParameter: (
.value | .filter.valuesMap |= (
to_entries | map({valueReference: .key, value})
)
)
})
)
'
{
"pathOptions": {
"parameters": [
{
"pathParameterName": "raw_date",
"datasetParameter": {
"name": "raw_date",
"type": "Datetime",
"datetimeOptions": {
"localeCode": "en-GB"
},
"createColumn": true,
"filter": {
"expression": "(after :date1)",
"valuesMap": [
{
"valueReference": ":date1",
"value": "2022-03-08T00:00:00.000Z"
}
]
}
}
}
]
}
}
Demo
I have the following input in Nifi Jolt Specification processor:
[
{
"values": [
{
"id": "paramA",
"value": 1
}
]
},
{
"values": [
{
"id": "paramB",
"value": 3
}
]
}
]
Expected output:
[
{
"id": "paramA",
"value": 1
},
{
"id": "paramB",
"value": 2
}
]
Can you explain how I have to do?
thanks in advance
You want to reach the objects of the values array which are nested within seperate object signs ({}). A "*" notation is needed in order to cross them over per each individual values array, and then use another "*" notation for indexes of those arrays while choosing "" as the counterpart values in order to grab nothing but the sub-objects such as
[
{
"operation": "shift",
"spec": {
"*": {
"values": {
"*": ""
}
}
}
}
]
I use following terraform code to get a list of available db resources:
data "alicloud_db_instance_classes" "resources" {
instance_charge_type = "PostPaid"
engine = "PostgreSQL"
engine_version = "10.0"
category = "HighAvailability"
zone_id = "${data.alicloud_zones.rds_zones.ids.0}"
multi_zone = true
output_file = "./classes.txt"
}
And the output file looks like this:
[
{
"instance_class": "pg.x4.large.2",
"storage_range": {
"max": "500",
"min": "250",
"step": "250"
},
"zone_ids": [
{
"id": "cn-shanghai-MAZ1(b,c)",
"sub_zone_ids": [
"cn-shanghai-b",
"cn-shanghai-c"
]
}
]
},
{
"instance_class": "pg.x8.medium.2",
"storage_range": {
"max": "250",
"min": "250",
"step": "0"
},
"zone_ids": [
{
"id": "cn-shanghai-MAZ1(b,c)",
"sub_zone_ids": [
"cn-shanghai-b",
"cn-shanghai-c"
]
}
]
},
{
"instance_class": "rds.pg.c1.xlarge",
"storage_range": {
"max": "2000",
"min": "5",
"step": "5"
},
"zone_ids": [
{
"id": "cn-shanghai-MAZ1(b,c)",
"sub_zone_ids": [
"cn-shanghai-b",
"cn-shanghai-c"
]
}
]
},
{
"instance_class": "rds.pg.s1.small",
"storage_range": {
"max": "2000",
"min": "5",
"step": "5"
},
"zone_ids": [
{
"id": "cn-shanghai-MAZ1(b,c)",
"sub_zone_ids": [
"cn-shanghai-b",
"cn-shanghai-c"
]
}
]
}
]
And I want to get the one that's cheapest.
One way to do so is by sorting with storage-range.min, but how do I sort this list based on 'storage_range.min'?
Or I can filter by 'instance_class', but "alicloud_db_instance_classes" doesn't seem to like filter as it says: Error: data.alicloud_db_instance_classes.resources: : invalid or unknown key: filter
Any ideas?
The sort() function orders lexicographical and you have no simple key here.
You can use filtering with some code like this (v0.12)
locals {
best_db_instance_class_key = "rds.pg.s1.small"
best_db_instance_class = element( alicloud_db_instance_classes.resources, index(alicloud_db_instance_classes.resources.*.instance_class, best_db_instance_class_key) )
}
(Untested code)
I want edit the structure of json through Terminal using terminal commands or scripts.
If I have a json file Structure like this:
{
"Helloo": [
{
"AlbumTitle": {
"S": "Famous"
},
"SongTitle": {
"S": "Call Me Today"
},
"Artist": {
"S": "No One You Know"
}
},
{
"AlbumTitle": {
"S": "Famous1"
},
"SongTitle": {
"S": "Call Me Today1"
},
"Artist": {
"S": "No One You Know11"
}
}
],
"Music": [
{
"Album": {
"S": "Pop Songs"
},
"Production": {
"S": "X-series"
},
"Song": {
"S": "Once upon
},
"Artist": {
"S": "XYZ"
}
}
]
}
So here i want add "Putrequest" and "Item" attributes to each item of the array.. So i want the output like this:
{
"Helloo": [
{
PutRequest":{
"Item":{
"AlbumTitle": {
"S": "Famous"
},
"SongTitle": {
"S": "Call Me Today"
},
"Artist": {
"S": "No One You Know"
}
}
}
},
{
PutRequest":{
"Item":{
"AlbumTitle": {
"S": "Famous1"
},
"SongTitle": {
"S": "Call Me Today1"
},
"Artist": {
"S": "No One You Know11"
}
}
}
}
],
"Music": [
{
PutRequest":{
"Item":{
"Album": {
"S": "Pop Songs"
},
"Production": {
"S": "X-series"
},
"Song": {
"S": "Once upon
},
"Artist": {
"S": "XYZ"
}
}
}
}
]
}
I tried to use Jq for this but still struggling.. Please help me To add these attributes to json using command prompt or bash/shell scripting.
Thanks
Assuming you actually got valid JSON the following jq expression might work for you:
map_values(map({"PutRequest": { "Item": .}}))
Usage:
jq 'map_values(map({"PutRequest": { "Item": .}}))' file.json
Breakdown:
map_values( # Map values iterate over an object and assign the
# returned value to the property
map( # Map iterate over an array and assign the returned value
# to the index, and creates a new array if an object is
# mapped
{ # Return an object
"PutRequest": { # With PutRequest as a property
"Item": . # And Item, which contains the value (.)
}
}
)
)
I have the following JSON file:
{
"opscenter_adhoc_2014-02-03-12-34-16-UTC": {
"keyspaces": {
"test": {
"nodes": [
"10.242.214.188",
"10.62.77.47",
"10.244.15.39"
],
"cfs": {
"test": "/var/lib/cassandra/data/test/test/snapshots/opscenter_adhoc_2014-02-03-12-34-16-UTC"
}
}
},
"id": "adhoc",
"time": 1391430856
},
"opscenter_adhoc_2014-02-03-13-16-04-UTC": {
"keyspaces": {
"test": {
"nodes": [
"10.242.214.188",
"10.62.77.47",
"10.244.15.39"
],
"cfs": {
"test": "/var/lib/cassandra/data/test/test/snapshots/opscenter_adhoc_2014-02-03-13-16-04-UTC"
}
}
},
"id": "adhoc",
"time": 1391433364
},
"opscenter_adhoc_2014-02-03-11-32-06-UTC": {
"keyspaces": {
"test": {
"nodes": [
"10.242.214.188",
"10.62.77.47",
"10.244.15.39"
],
"cfs": {
"test": "/var/lib/cassandra/data/test/test/snapshots/opscenter_adhoc_2014-02-03-11-32-06-UTC"
}
}
},
"id": "adhoc",
"time": 1391427126
}
I'd like to grep the file by opscenter_adhoc_2014-02-03-11-32-06-UTC only where 2014-02-03-11-32-06-UTC is timestamp of backup and than sort the list by timestamps to get the latest timestamp.
Tried in Ruby by using
File.readlines("/tmp/out.txt").grep(/opscenter_adhoc/)
which returns me all the string but I need just only opscenter_adhoc_TIMESTAMP list
If I understood you correctly, you are expecting to get sorted array of dates. Here you go:
require 'json'
JSON.parse(File.read('/tmp/qq.json')).keys.sort
# ⇒ => [
# [0] "opscenter_adhoc_2014-02-03-11-32-06-UTC",
# [1] "opscenter_adhoc_2014-02-03-12-34-16-UTC",
# [2] "opscenter_adhoc_2014-02-03-13-16-04-UTC"
# ]