Related
The following command:
cat foo | jq -r '[.items[] | select(.metadata.random_field["foo/foo"] == "true") | .metadata.name]'
results in
[
"test_data"
]
I would like the following output to be:
[
{"name":"test", "data":"test_data"}
]
foo
{
"apiVersion": "v1",
"items":[
{
"metadata": {
"random_field": {
"foo/foo": "true"
},
"creationTimestamp": "2022-03-09T21:54:08Z",
"name": "test_data"
}
}
]
}
UPDATE: added test data so the point can be illustrated properly.
It's not entirely clear where the value for name in the output comes from in your question, but let's assume for a moment that it is the first part until the underscore of the name from the input. The the following produces your required output:
.items | map(
.metadata
| select(.random_field."foo/foo" == "true")
| { name: .name|split("_")|first, data: .name }
)
Output:
[
{
"name": "test",
"data": "test_data"
}
]
current code do work, is there any way to optimize it?
seems jq cmd cost lots of time to work result out of json struct.
which make autocomplete quite slow.
the code below try to define a nested shell command struct with json text, then use jq command to parse the json text, and parse autocomplete results.
function _auto_completion_linux_ext() {
COMPREPLY=()
local cmd="${COMP_WORDS[1]}"
local cur="${COMP_WORDS[COMP_CWORD]}"
local prev="${COMP_WORDS[COMP_CWORD-1]}"
local opts
# printf "\n[COMP_WORDS] -> ${COMP_WORDS}\n"
# printf "\n[COMP_CWORD] -> ${COMP_CWORD} -> ${cur}\n"
read -r -d '' var_basic_json << EOV
{
"init": {
"hostname": {}
},
"firewall": {
"enable": {},
"disable": {},
"reload": {},
"add": {
"tcp": {},
"udp": {}
},
"del": {
"tcp": {},
"udp": {}
},
"list": {}
},
"network": {
"ipv4": {
"dhcp": {},
"static": {}
},
"ipv6": {
"dhcp": {},
"static": {}
}
},
"status": {
"tcp": {},
"listening_port": {}
},
"upgrade": {}
}
EOV
case ${COMP_CWORD} in
1):
opts=$(echo ${var_basic_json} | jq -j ". | keys | join(\" \")")
;;
*):
local var_jq_idx=()
local var_jq_idx_str
for ((i = 1; i <= COMP_CWORD; i++)); do
var_jq_idx_str=$(echo ${var_jq_idx[#]} | tr " " ".")
if [[ $(echo ${var_basic_json} | jq -j ".${var_jq_idx} | has(\"${COMP_WORDS[i]}\")") ]]; then
var_jq_idx+=(${COMP_WORDS[i]})
else
break
fi
done
opts=$(echo ${var_basic_json} | jq -j ".${var_jq_idx_str} | keys | join(\" \")")
;;
esac
COMPREPLY=($(compgen -W "${opts}" -- ${cur}))
return 0;
}
complete -F _auto_completion_linux_ext system
First of all, sorry for my English, I'm French.
I'm working on a script, which retrieves tags and links from M3U files to store them into variables.
M3U:
#EXTM3U
#EXTINF:-1 tvg-id="TFX.fr" tvg-name="TFX" tvg-country="FR;AD;BE;LU;MC;CH" tvg-language="French" tvg-logo="http://www.exemple.com/image.jpg" group-title="",TFX (720p)
https://tfx-hls-live-ssl.tf1.fr/tfx/1/hls/live_2328.m3u8
script:
#!/bin/bash
tags='#EXTINF:-1 tvg-id="TFX.fr" tvg-name="TFX" tvg-country="FR;AD;BE;LU;MC;CH" tvg-language="French" tvg-logo="http://www.exemple.com/image.jpg" group-title="Fiction",TFX (720p)'
get_chno="$(echo "$tags" | grep -o 'tvg-chno="[^"]*' | cut -d '"' -f2)"
get_id="$(echo "$tags" | grep -o 'tvg-id="[^"]*' | cut -d '"' -f2)"
get_logo="$(echo "$tags" | grep -o 'tvg-logo="[^"]*' | cut -d '"' -f2)"
get_grp_title="$(echo "$tags" | grep -o 'group-title="[^"]*' | cut -d '"' -f2)"
get_title="$(echo "$tags" | grep -o ',[^*]*' | cut -d ',' -f2)"
get_name="$(echo "$tags" | grep -o 'tvg-name="[^"]*' | cut -d '"' -f2)"
get_country="$(echo "$tags" | grep -o 'tvg-country="[^"]*' | cut -d '"' -f2)"
get_language="$(echo "$tags" | grep -o 'tvg-language="[^"]*' | cut -d '"' -f2)"
echo -e "chno:\n $get_chno"
echo -e "id:\n $get_id"
echo -e "logo:\n $get_logo"
echo -e "grp 1:\n $get_grp_title"
echo -e "title:\n $get_title"
echo -e "name:\n $get_name"
echo -e "country:\n $get_country"
echo -e "lang:\n $get_language"
I would like to store these variables in a json file.
This json will be used to rebuild another playlist.
#EXTM3U
#EXTINF:-1 tvg-id="TFX.fr" tvg-name="TFX" tvg-country="FR;AD;BE;LU;MC;CH" tvg-language="French" tvg-logo="http://www.exemple.com/image.jpg" group-title="",TFX (720p)
https://tfx-hls-live-ssl.tf1.fr/tfx/1/hls/live_2328.m3u8
#EXTINF:-1 tvg-id="TFX.fr" tvg-name="TFX" tvg-country="FR;AD;BE;LU;MC;CH" tvg-language="French" tvg-logo="http://127.0.0.1/img/image.jpg" group-title="",TFX (local)
http://127.0.0.1:1234/tfx/live.m3u8
The file which contains multiple arrays and multiple objects.
Like this :
{
"Channels": [
{
"name": "TFX",
"old_name": "NT1",
"logo": "http://www.exemple.com/image.jpg",
"category": "Fiction",
"urls": {
"Official": [
{
"server_name": "TF1",
"IP_address": "8.8.8.8",
"url": "tfx-hls-live-ssl.tf1.fr",
"port": "",
"https_port": "443",
"path": "tfx/1/hls/",
"file_name": "live_2328",
"extension": ".m3u8",
"full_url": "https://tfx-hls-live-ssl.tf1.fr/tfx/1/hls/live_2328.m3u8"
}
],
"Xtream_Servers": [
{
"server_name": "local",
"user_name": "rickey",
"stream_id": "11",
"category_name": "Fiction",
"category_id": "12"
}
]
},
"languages": [
{
"code": "fr",
"name": "Français"
}
],
"countries": [
{
"code": "fr",
"name": "France"
},
{
"code": "be",
"name": "Belgium"
}
],
"tvg": {
"id": "TFX.fr",
"name": "TFX",
"url": ""
}
},
{
"name": "France 2",
"old_name": "",
"logo": "http://www.exemple.com/image.jpg",
"category": "Général",
"urls": {
"Official": [
{
"server_name": "France TV",
"IP_address": "8.8.8.8",
"url": "france2.fr",
"port": "",
"https_port": "443",
"path": "live/",
"file_name": "Playlist",
"extension": ".m3u8",
"full_url": "https://france2.fr/live/Playlist.m3u8"
}
],
"Xtream_Servers": [
{
"server_name": "localhost",
"user_name": "rickey",
"stream_id": "2",
"category_name": "Général",
"category_id": "10"
}
]
},
"languages": [
{
"code": "fr",
"name": "Français"
}
],
"countries": [
{
"code": "fr",
"name": "France"
},
{
"code": "be",
"name": "Belgique"
}
],
"tvg": {
"id": "France2.fr",
"name": "France 2",
"url": ""
}
},
{
"name": "M6",
"old_name": "",
"logo": "http://www.exemple.com/image.jpg",
"category": "Général",
"urls": {
"Official": [
{
"server_name": "6Play",
"IP_address": "8.8.8.8",
"url": "6play.fr",
"port": "",
"https_port": "443",
"path": "live/",
"file_name": "Playlist",
"extension": ".m3u8",
"full_url": "https://6play.fr/M6/live/Playlist.m3u8"
}
],
"Xtream_Servers": [
{
"server_name": "localhost",
"user_name": "rickey",
"stream_id": "6",
"category_name": "Général",
"category_id": "10"
}
]
},
"languages": [
{
"code": "fr",
"name": "Français"
}
],
"countries": [
{
"code": "fr",
"name": "France"
},
{
"code": "be",
"name": "Belgique"
}
],
"tvg": {
"id": "France2.fr",
"name": "France 2",
"url": ""
}
}
],
"Third_Party": {
"Xtream_Servers": [
{
"server_name": "local",
"url": "192.168.1.100",
"port": "8080",
"https_port": "8082",
"server_protocol": "http",
"rtmp_port": "12345",
"Users_list": [
{
"username": "rickey",
"password": "azerty01",
"created_at": "",
"exp_date": "",
"is_trial": "0",
"last_check": "",
"max_connections": "3",
"allowed_output_formats": [
"m3u8",
"ts",
"rtmp"
]
}
]
},
{
"server_name": "localhost",
"url": "127.0.0.1",
"port": "8080",
"https_port": "8082",
"server_protocol": "http",
"rtmp_port": "12345",
"Users_list": [
{
"username": "rickey123",
"password": "azerty321",
"created_at": "",
"exp_date": "",
"is_trial": "0",
"last_check": "",
"max_connections": "3",
"allowed_output_formats": [
"m3u8",
"ts",
"rtmp"
]
},
{
"username": "guest",
"password": "guest01",
"created_at": "",
"exp_date": "",
"is_trial": "1",
"last_check": "",
"max_connections": "1",
"allowed_output_formats": [
"ts"
]
}
]
}
]
}
}
First question: Is it a crappy json?
To add or modify this file, the script must have the entry number (I think, if you have any other ideas, I'm interested...)
cat File.json | jq '.Channels | to_entries[]'
output:
{
"key": 0,
"value": {
"name": "TFX",
"old_name": "NT1",
2nd question:
How to get value key (0 is this case) with the value of "name", for store into variable after ? (to avoid duplicates)
key_="$(cat file.json | jq ????????? search="name": "$get_name" ???? .key)"
echo $key_
"0"
key_2="$(cat file.json | jq ????????? search="name": "$get_url" ???? .key)"
echo $key_2
"0"
if [[ $key_ == $key_2 ]]; then
Chan_Name="$(cat $1 | jq '.Channels[$key_].name)"
Echo $Chan_Name
"TFX"
jq '.[] ????? += {???? , ??? }' file.json | sponge file.json
fi
last question (most important):
How to find and modify these f*** objects, when the script does not know any values of the keys of the objects / arrays ?!
I've been looking for 2 days, my brain is liquid.
Thank you. :)
Edit 1 :
I've found a partial solution to replace value:
{
"name": "TFX",
"old_name": "NT1",
"logo": "http://www.exemple.com/image.jpg",
"category": "Fiction",
with:
cat file.json | jq -C '(.Channels[] | select(.name=="TFX").category="test")'
output:
{
"name": "TFX",
"old_name": "NT1",
"logo": "http://www.exemple.com/image.jpg",
"category": "test",
"urls": {
but "{"Channels": [" is missing. :/
jq -C '(.Channels[] | select(.name=="TFX").category="test")'
You were so close - just one misplaced parenthesis:
jq '(.Channels[] | select(.name=="TFX")) .category="test"'
I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty
I have json code and need to filter it by the value of the attribute DNSName. The filter must be case insensitive.
How can I do that? Is there a possibility to solve it with jq?
This is how I create the json code:
aws elbv2 describe-load-balancers --region=us-west-2 | jq
My unfiltered source json code looks like this:
{
"LoadBalancers": [
{
"IpAddressType": "ipv4",
"VpcId": "vpc-abcdabcd",
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-2:000000000000:loadbalancer/app/MY-LB1/a00000000000000a",
"State": {
"Code": "active"
},
"DNSName": "MY-LB1-123454321.us-west-2.elb.amazonaws.com",
"SecurityGroups": [
"sg-00100100",
"sg-01001000",
"sg-10010001"
],
"LoadBalancerName": "MY-LB1",
"CreatedTime": "2018-01-01T00:00:00.000Z",
"Scheme": "internet-facing",
"Type": "application",
"CanonicalHostedZoneId": "ZZZZZZZZZZZZZ",
"AvailabilityZones": [
{
"SubnetId": "subnet-17171717",
"ZoneName": "us-west-2a"
},
{
"SubnetId": "subnet-27272727",
"ZoneName": "us-west-2c"
},
{
"SubnetId": "subnet-37373737",
"ZoneName": "us-west-2b"
}
]
},
{
"IpAddressType": "ipv4",
"VpcId": "vpc-abcdabcd",
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-2:000000000000:loadbalancer/app/MY-LB2/b00000000000000b",
"State": {
"Code": "active"
},
"DNSName": "MY-LB2-9876556789.us-west-2.elb.amazonaws.com",
"SecurityGroups": [
"sg-88818881"
],
"LoadBalancerName": "MY-LB2",
"CreatedTime": "2018-01-01T00:00:00.000Z",
"Scheme": "internet-facing",
"Type": "application",
"CanonicalHostedZoneId": "ZZZZZZZZZZZZZ",
"AvailabilityZones": [
{
"SubnetId": "subnet-54545454",
"ZoneName": "us-west-2a"
},
{
"SubnetId": "subnet-64646464",
"ZoneName": "us-west-2c"
},
{
"SubnetId": "subnet-74747474",
"ZoneName": "us-west-2b"
}
]
}
]
}
I now want some bash code to filter this result for the record with the DNSName property value MY-LB2-9876556789.us-west-2.elb.amazonaws.com, and need the entire LoadBalancer object back as a result. This is how I wish my result to look like:
{
"IpAddressType": "ipv4",
"VpcId": "vpc-abcdabcd",
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-2:000000000000:loadbalancer/app/MY-LB2/b00000000000000b",
"State": {
"Code": "active"
},
"DNSName": "MY-LB2-9876556789.us-west-2.elb.amazonaws.com",
"SecurityGroups": [
"sg-88818881"
],
"LoadBalancerName": "MY-LB2",
"CreatedTime": "2018-01-01T00:00:00.000Z",
"Scheme": "internet-facing",
"Type": "application",
"CanonicalHostedZoneId": "ZZZZZZZZZZZZZ",
"AvailabilityZones": [
{
"SubnetId": "subnet-54545454",
"ZoneName": "us-west-2a"
},
{
"SubnetId": "subnet-64646464",
"ZoneName": "us-west-2c"
},
{
"SubnetId": "subnet-74747474",
"ZoneName": "us-west-2b"
}
]
}
Does anyone know how to do it?
Update:
This solution works, but is not case insensitive:
aws elbv2 describe-load-balancers --region=us-west-2 | jq -c '.LoadBalancers[] | select(.DNSName | contains("MY-LB2"))'
Update:
This solution seems to work even better:
aws elbv2 describe-load-balancers --region=us-west-2 | jq -c '.LoadBalancers[] | select(.DNSName | match("my-lb2";"i"))'
But I did not have the chance to test in detail yet.
You probably should be using test/2 rather than match/2, but in either case, since the problem description calls for
case-insensitive equality, you would use an anchored regex:
.LoadBalancers[]
| select(.DNSName | test("^my-lb2-9876556789.us-west-2.elb.amazonaws.com$";"i"))
With the caveat that ascii_upcase only translates ASCII characters, it might be more efficient to use it:
.LoadBalancers[]
| select(.DNSName | ascii_upcase == "MY-LB2-9876556789.US-WEST-2.ELB.AMAZONAWS.COM")