I have a JSON doc that looks like:
{
"SecurityGroups": [
{
"GroupName": "database",
"GroupId": "sg-xxxxxx",
"VpcId": "vpc-yyyyyyy",
"IpPermissions": [
{
"FromPort": 22,
"ToPort": 22,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "10.200.0.0/16"
},
{
"CidrIp": "10.200.30.79/32"
},
{
"CidrIp": "10.200.42.0/24"
}
],
"UserIdGroupPairs": []
},
{
"FromPort": 5555,
"ToPort": 5555,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "10.200.0.0/16"
},
{
"CidrIp": "10.200.0.155/32"
}
],
"UserIdGroupPairs": []
},
{
"FromPort": 4506,
"ToPort": 4506,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "10.200.0.0/16"
}
],
"UserIdGroupPairs": []
}
]
}
]
}
The output I need to generate is as follow:
sg-xxxxxx|database|22|22|tcp|10.200.0.0/16
sg-xxxxxx|database|22|22|tcp|10.200.30.79/32
sg-xxxxxx|database|22|22|tcp|10.200.42.0/24
sg-xxxxxx|database|5555|5555|tcp|10.200.0.0/16
sg-xxxxxx|database|5555|5555|tcp|10.200.0.155/32
sg-xxxxxx|database|4506|4506|tcp|10.200.0.0/16
I'm able to achieve that by using using jq first to generate a list of GroupId's and then loop through the list to filter data into jq twice. Here's how I did it:
cat json.in | jq -r '.SecurityGroups[]|"\(.GroupId) \(.GroupName)"' | while read groupid groupname
do
cat json.in | jq ".SecurityGroups[]|{GroupId,IpPermissions,IpPermissionsEgress}|select(.GroupId == \"$groupid\")" | jq -r '.IpPermissions[]|"\(.FromPort)|\(.ToPort)|\(.IpProtocol)|\(.IpRanges[].CidrIp)"' | sed "s/^/$groupid|$groupname|/"
done
My solution is slow and I would like to improve on it, Any pointers?
Here is a more efficient approach. With the -r option, the following filter
.SecurityGroups[]
| .GroupId as $gid
| .GroupName as $gname
| (.IpPermissions[], .IpPermissionsEgress[]?)
| .FromPort as $from
| .ToPort as $to
| .IpProtocol as $pro
| .IpRanges[]
| "\($gid)|\($gname)|\($from)|\($to)|\($pro)|\(.CidrIp)"
with the sample data produces
sg-xxxxxx|database|22|22|tcp|10.200.0.0/16
sg-xxxxxx|database|22|22|tcp|10.200.30.79/32
sg-xxxxxx|database|22|22|tcp|10.200.42.0/24
sg-xxxxxx|database|5555|5555|tcp|10.200.0.0/16
sg-xxxxxx|database|5555|5555|tcp|10.200.0.155/32
sg-xxxxxx|database|4506|4506|tcp|10.200.0.0/16
Note that this includes .IpPermissionsEgress[]? because although it's absent from your sample data and unused in the second part of your script it is nevertheless present in the first part of your sample script so I think you may have intended to include it.
Related
I want to have an output that instead of delimeter space it should be like this ", " using jq
nodes.json
{
"nodes": {
"node1.local": {
":ip": "10.0.0.1",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
},
"node2.local": {
":ip": "10.0.0.2",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
},
"node3.local": {
":ip": "10.0.0.3",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
}
}
}
here is my command use
ips=`cat /vagrant/nodes.json | jq -r '.nodes | to_entries[] | [.value.":ip"] | #tsv'`
echo [\"$ips\"]
where the output is
["10.0.0.1 10.0.0.2 10.0.0.3"]
and i want it to be like this
["10.0.0.1", "10.0.0.2", "10.0.0.3"]
$ jq -c '.nodes | to_entries | map(.value.":ip")' input
["10.0.0.1","10.0.0.2","10.0.0.3"]
I want to have an output with static value using jq with static value :4546
nodes.json
{
"nodes": {
"node1.local": {
":ip": "10.0.0.1",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
},
"node2.local": {
":ip": "10.0.0.2",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
},
"node3.local": {
":ip": "10.0.0.3",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap.sh"
}
}
}
here is my command use
ips=`jq -c '.nodes | to_entries | map(.value.":ip")' nodes.json`
echo $ips
where the output is
["10.0.0.1", "10.0.0.2", "10.0.0.3"]
and i want it to be like this
["10.0.0.1:4546", "10.0.0.2:4546", "10.0.0.3:4546"]
Another try :
jq '[.nodes[][":ip"]+":4546"]' nodes.json
You could use map_values
jq -c '.nodes | to_entries | map(.value.":ip")| map_values(.+":4546")' nodes.jso
You can simply use + operator:
jq '.nodes | to_entries | map(.value.":ip" + ":4546")'
Print data of a file
This it the file content
{
"nodes": {
"server.xyz": {
":ip": "192.168.56.5",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-master.sh"
},
"client1.abc": {
":ip": "192.168.56.10",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-node.sh"
},
"client1.def": {
":ip": "192.168.56.15",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-node.sh"
}
}
}
I only want to print
server.xyz
client1.abc
client2.def
and this IP
192.168.56.5
192.168.56.10
192.168.56.15
You can use jq,
Example
{
"nodes": {
"server.local": {
":ip": "192.168.56.5",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-master.sh"
},
"client1.local": {
":ip": "192.168.56.10",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-node.sh"
},
"client1.local": {
":ip": "192.168.56.15",
"ports": [],
":memory": 1024,
":bootstrap": "bootstrap-node.sh"
}
}}
And run,
cat file | jq -r '.nodes | keys[]'
Output
client1.abc
client1.def
server.xyz
Edit:
If you want ip as well,
cat file | jq -r '.nodes | to_entries[] | [.key, .value.":ip"] | #tsv'
you can execute the file as commands
and have echo in the file so like
"nodes": {
echo"server.xyz": {
echo ":ip": "192.168.56.5",
```
and so on then you do
```eval $(cat file location)```
but i think the has to be in one line
I'm working on generating a new JSON payload to update Consul with a MSSQL database service location.
When I call jq like this:
mssql_svc_ip=$(kubectl get svc/mssql-linux -o 'jsonpath={.spec.clusterIP}')
mssql_svc_port=$(kubectl get svc/mssql-linux -o 'jsonpath={.spec.ports[0].port}')
jq -n -r --arg MSSQL_IP $mssql_svc_ip --arg MSSQL_PORT $mssql_svc_port '{
"Datacenter": "dev",
"Node": "database",
"Address": $MSSQL_IP,
"Service": {
"Service": "mssql-dev",
"Port": $MSSQL_PORT
}
}'
It produces the proper structure:
{
"Datacenter": "dev",
"Node": "database",
"Address": "10.43.192.146",
"Service": {
"Service": "mssql-dev",
"Port": "1433"
}
}
I need to convert the Service.Port field from a string to an integer as that's what the Consul API requires. I can do that with tonumber, like this:
mssql_svc_ip=$(kubectl get svc/mssql-linux -o 'jsonpath={.spec.clusterIP}')
mssql_svc_port=$(kubectl get svc/mssql-linux -o 'jsonpath={.spec.ports[0].port}')
jq -n -r --arg MSSQL_IP $mssql_svc_ip --arg MSSQL_PORT $mssql_svc_port '{
"Datacenter": "dev",
"Node": "database",
"Address": $MSSQL_IP,
"Service": {
"Service": "mssql-dev",
"Port": tonumber($MSSQL_PORT)
}
}'
However, when I try and convert the $MSSQL_PORT variable to a number, I get this error:
jq: error: tonumber/1 is not defined at <top-level>, line 7:
"Port": tonumber($MSSQL_PORT)
jq: 1 compile error
At first I thought it was an assignment error and the variables weren't being passes as arguments properly, but I've tried a couple iterations and I still get the same error. What am I doing incorrectly?
I think you are misusing the tonumber filter. Based on the documentation it looks like the syntax would be something like:
jq -n -r --arg MSSQL_IP "$mssql_svc_ip" --arg MSSQL_PORT "$mssql_svc_port" '{
"Datacenter": "dev",
"Node": "database",
"Address": $MSSQL_IP,
"Service": {
"Service": "mssql-dev",
"Port": ($MSSQL_PORT|tonumber)
}
}'
And indeed, if $msssql_svc_ip is 10.43.192.146 and $mssql_svc_port is
1433, that gets me:
{
"Datacenter": "dev",
"Node": "database",
"Address": "10.43.192.146",
"Service": {
"Service": "mssql-dev",
"Port": 1433
}
}
Looks like you need to pass the number in with --argjson instead of --arg:
$ jq -n -r --argjson foo 12 '{"foo":$foo}'
{
"foo": 12
}
This seems simpler than using tonumber
I have a VPC with RDS available in a private subnet. I can connect to this from an EC2 box from within the subnet. However, my Lambdas cannot connect!
Please could you look at the following configuration and spot my mistake?
Lambda config:
$ aws lambda get-function-configuration --function-name test
{
"FunctionName": "test",
"Role": "arn:aws:iam::xxxx:role/lambda_role",
...
"VpcConfig": {
"SubnetIds": [
"subnet-00f3f0cb6957dbefa",
"subnet-0d3d2cf4df53a862f"
],
"SecurityGroupIds": [
"sg-018da51b77f57eabf"
],
"VpcId": "vpc-0704ca4d3f652fe9e"
},
...
"RevisionId": "e55b6fa2-998a-4b18-a620-69a218882b4e"
}
Execution role:
$ aws list-attached-role-policies --role-name lambda_role
{
"AttachedPolicies": [
{
"PolicyName": "AWSLambdaVPCAccessExecutionRole",
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}
]
}
VPC:
$ aws ec2 describe-vpcs --vpc-ids vpc-0704ca4d3f652fe9e
{
"Vpcs": [
{
"CidrBlock": "10.1.0.0/16",
"DhcpOptionsId": "dopt-7764271f",
"State": "available",
"VpcId": "vpc-0704ca4d3f652fe9e",
"InstanceTenancy": "default",
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-0c110a5fa85eb8883",
"CidrBlock": "10.1.0.0/16",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": []
}
]
}
Security Group:
$ aws ec2 describe-security-groups --group-ids sg-018da51b77f57eabf
{
"SecurityGroups": [
{
"Description": "Security group for Lambdas",
"GroupName": "lambda-sg",
"IpPermissions": [],
"OwnerId": "xxxxx",
"GroupId": "sg-018da51b77f57eabf",
"IpPermissionsEgress": [
{
"FromPort": 0,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 65535,
"UserIdGroupPairs": []
}
],
"VpcId": "vpc-0704ca4d3f652fe9e"
}
]
}
RDS security group (specifies both public and private subnets):
$ aws ec2 describe-security-groups --group-ids sg-0fbf7205b5d5fa98c
{
"SecurityGroups": [
{
"Description": "Security group for RDS instance",
"GroupName": "rds-sg",
"IpPermissions": [
{
"FromPort": 3306,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "10.1.2.0/24"
},
{
"CidrIp": "10.1.1.0/24"
},
{
"CidrIp": "10.1.4.0/24"
},
{
"CidrIp": "10.1.3.0/24"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 3306,
"UserIdGroupPairs": []
}
],
"OwnerId": "xxxxxx",
"GroupId": "sg-0fbf7205b5d5fa98c",
"IpPermissionsEgress": [],
"VpcId": "vpc-0704ca4d3f652fe9e"
}
]
}
Linked:
AWS Lambda Function not joining VPC
Try to enable ICMP on any security group and any network ACL. It could simply be a PMTUD blackhole situation.