how to optimize bash command with nested sub-command? - bash

current code do work, is there any way to optimize it?
seems jq cmd cost lots of time to work result out of json struct.
which make autocomplete quite slow.
the code below try to define a nested shell command struct with json text, then use jq command to parse the json text, and parse autocomplete results.
function _auto_completion_linux_ext() {
COMPREPLY=()
local cmd="${COMP_WORDS[1]}"
local cur="${COMP_WORDS[COMP_CWORD]}"
local prev="${COMP_WORDS[COMP_CWORD-1]}"
local opts
# printf "\n[COMP_WORDS] -> ${COMP_WORDS}\n"
# printf "\n[COMP_CWORD] -> ${COMP_CWORD} -> ${cur}\n"
read -r -d '' var_basic_json << EOV
{
"init": {
"hostname": {}
},
"firewall": {
"enable": {},
"disable": {},
"reload": {},
"add": {
"tcp": {},
"udp": {}
},
"del": {
"tcp": {},
"udp": {}
},
"list": {}
},
"network": {
"ipv4": {
"dhcp": {},
"static": {}
},
"ipv6": {
"dhcp": {},
"static": {}
}
},
"status": {
"tcp": {},
"listening_port": {}
},
"upgrade": {}
}
EOV
case ${COMP_CWORD} in
1):
opts=$(echo ${var_basic_json} | jq -j ". | keys | join(\" \")")
;;
*):
local var_jq_idx=()
local var_jq_idx_str
for ((i = 1; i <= COMP_CWORD; i++)); do
var_jq_idx_str=$(echo ${var_jq_idx[#]} | tr " " ".")
if [[ $(echo ${var_basic_json} | jq -j ".${var_jq_idx} | has(\"${COMP_WORDS[i]}\")") ]]; then
var_jq_idx+=(${COMP_WORDS[i]})
else
break
fi
done
opts=$(echo ${var_basic_json} | jq -j ".${var_jq_idx_str} | keys | join(\" \")")
;;
esac
COMPREPLY=($(compgen -W "${opts}" -- ${cur}))
return 0;
}
complete -F _auto_completion_linux_ext system

Related

how to display files in shell script

I have few files in BrowserStack and I want to display file name and then I want to delete that file with given custom_id. (BELOW IS SAMPLE OUTPUT)
[
{
"custom_id": "abcapp",
"app_id": "abcapp"
},
{
"custom_id": "xyzapp",
"app_id": "xyzapp"
},
{
"custom_id": "abcapp",
"app_id": "abcapp"
},
{
"custom_id": "xyzapp",
"app_id": "xyzapp"
},
{
"custom_id": "abcapp",
"app_id": "abcapp"
},
{
"custom_id": "abcapp",
"app_id": "abcapp"
}
]
I wrote shell script where I want to display & delete files
#!/bin/bash
for var in $(curl -u "xyz_u9sXFH:pqRfaqxRQpmxyJmzEqx" -X GET "https://api-cloud.browserstack.com/app-automate/recent_apps" | jq '.[] | select(.app_id=="abcapp")' | jq '.app_id')
do
echo "Deleting $var"
curl -u "xyz_u9sXFH:pqRfaqxRQpmxyJmzEqx" -X DELETE "https://api-cloud.browserstack.com/app-automate/app/delete/$var"
done
I am very new to shell script, so don't know what's wrong with it.

jq loop on items in new lines

I could read all of the entries it was set to, but they are put in one line instead of appending on new line.
Source:
#!/bin/bash
for row in $(jq -r '. | #base64' recipe.json); do
_jq() {
echo ${row} | base64 --decode | jq -r ${1}
}
#echo $(_jq '.dependencies[].name')
echo $(_jq '.dependencies[].name') >> file.log
done
recipe.json:
{
"florecipe-v": "1.0",
"name": "web3updater",
"dependencies": [{
"name": "ethereal",
"version": "2.6.0",
"source": {
"ipfs": "QmWmxakbb7msHFaD9QmGXBAmLpUHTdZG64BWhQg8dyx8gM",
"github": "https://github.com/wealdtech/ethereal.git"
},
"linux-amd64": {
"ipfs": "QmbwABiXz4U5MrmJLisfxaHEG7CyJKEnNECRjXogS7eAFw",
"github": "https://github.com/wealdtech/ethereal/releases/download/v2.6.0/ethereal-2.6.0-linux-amd64.tar.gz"
},
"linux-arm64": {
"ipfs": "QmV9agwSZuVXMJNYV1NhU8SY8TRg73z8RgTw4QaYrVxtrr",
"github": "https://github.com/wealdtech/ethereal/releases/download/v2.6.0/ethereal-2.6.0-linux-arm64.tar.gz"
},
"windows-i386": {
"ipfs": "",
"github": ""
}
},
{
"name": "new",
"version": "",
"source": {
"ipfs": "",
"github": ""
},
"linux-amd64": {
"ipfs": "",
"github": ""
},
"linux-arm64": {
"ipfs": "",
"github": ""
},
"windows-i386": {
"ipfs": "",
"github": ""
}
}
]
}
file.log properly contains ethereal and new, but they are on the same line instead of appending.
Instead of the dubious:
echo $(_jq '.dependencies[].name')
write:
_jq '.dependencies[].name'
IOW, KIS.
And speaking of KIS, have you considered using #base64d? As in:
printf "%s" "${row}" | jq -Rr "#base64d | fromjson | ${1}"

Elasticsearch: Auto delition with ILM doesn't work

I wanna delete an index after certain time(say 10s) but it doesn't work. I researched a lot but I couldn't find a different thing from my configs. Here are my configs:
my ILM config:
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "10s",
"actions": {
"delete": {}
}
}
}
}
}
my template:
{
"order": 0,
"index_patterns": ["myindex.*"],
"settings": {
"index": {
"lifecycle": {
"name": "myindex"
}
}
}
}
my index:
{
"job_id": 1,
"message": "sample data",
"#timestamp": "DATETIME"
}
then with the following script, I test my scenario:
#!/bin/bash
NAME=myindex
# Add ILM and template.
echo -n "Add policy..."
http PUT http://localhost:9200/_ilm/policy/"${NAME}" < ilm.json
echo -n "Add template..."
http PUT http://localhost:9200/_template/"${NAME}" < template.json
# Create index.
echo -n "Create index..."
http PUT http://localhost:9200/"${NAME}" | jq '.acknowledged'
# List of data stream and indices.
echo -n "List of data stream..."
http http://localhost:9200/_data_stream/ | jq '.data_streams'
echo "List of indices..."
http http://localhost:9200/_cat/indices/ | awk '{print $3}'
echo "Index ILM explain..."
http http://localhost:9200/"${NAME}"/_ilm/explain
# Insert a document.
echo -n "Insert a document..."
sed "s/DATETIME/$(date +%Y-%m-%dT%H:%M:%S)/g" < index.json | http POST http://localhost:9200/"${NAME}"/_doc | jq -r ".result"
# Wait until it reaches the TTL.
echo "Waiting to reach the TTL..."
sleep 10 # "$(( $(jq -r .policy.phases.delete.min_age < ilm.json | sed 's/s//g') + 2 ))"
# Search for the data and expected to find nothing.
echo "Search the inserted document..."
echo '{"query": {"match": {"job_id": 1}}}' | http http://localhost:9200/"${NAME}"/_search
In the last part when I search for the index I can find it. The index is still present!!!
I set indices.lifecycle.poll_interval and it works!
PUT /_cluster/settings
{
"persistent" : {
"indices.lifecycle.poll_interval": "5s"
}
}
References:
https://discuss.elastic.co/t/index-lifecycle-management-at-elasticsearch-6-6-0/168429/2
https://stackoverflow.com/a/66138564/4604579

Bash get value of matching substring in a long string

I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty

Replace an attribute or key in JSON using jq or sed

Have a big json like this
"envConfig": {
"environmentName": {
"versions": [
{
"name": "version1",
"value": "Dev"
},
{
"name": "version2",
"host": "qa"
}
],
"userRoles": [
{
"name": "Roles",
"entry": [
{
"name": "employees",
"value": "rwx"
},
{
"name": "customers",
"value": "rx"
}
]
}
]
}
},
I wanted to change the JSON attribute from "environmentName" to "prod". Below is the output i am expecting
"envConfig": {
"prod": {
"versions": [
...
],
"userRoles": [
...
]
}
}
Tried with sed command as below
sed "s/\('environmentName':\)/\1\"prod\"\,/g" version.json
Tried with jq as below but not working
cat version.json | jq ' with_entries(.value |= {"prod" : .environmentName} ) '
Any help here to replace the attribute/key of an json with desired value
You weren't too far off with the jq, how about this?
jq '.envConfig |= with_entries(.key |= sub("^environmentName$"; "prod"))'
Two differences: first off, we want to drill down to envConfig before doing a with_entries, and second, when we get there, the thing we want will be a key, not a value. In case there are any other keys besides environmentName they'll be preserved.
TL,TR
You can use the following command:
jq '(.envConfig |= (. + {"prod":.environmentName}|del(.environmentName)))' foo.json
Let's say you have the following json:
{
"foo": {
"hello" : "world"
}
}
You can rename the node foo to bar by first duplicating it and then remove the original node:
jq '. + {"bar":.foo}|del(.foo)' foo.json
Output:
{
"bar": {
"hello" : "world"
}
}
It get's a bit more complicated if you want to replace a child key somewhere in the tree. Let's say you have the following json:
{
"test": {
"foo": {
"hello": "world"
}
}
}
You can use the following jq command for that:
jq '(.test |= (. + {"bar":.foo}|del(.foo)))' foo.json
Note the additional parentheses and the use of the assignment operator |=.
Output:
{
"test": {
"bar": {
"hello": "world"
}
}
}
Using sed:
sed -i '/^ \"environmentName\":/ s/environmentName/prod/' <yourfile>
Keep in mind that -i will overwrite the file. You may want to make a backup first.

Resources