I'm using grep to search for specific text inside multiple files.
grep -n -r "text_to_match" *
The output is:
john:15
Hana:11
After that, am using awk to print whatever meets my criteria of the condition
grep -n -r "text_to_match" * | awk -F '[:]' '{print $1,$2}'
I want now to use jq to have the following json document
{
"data": [
{
"name": "john",
"age": 15
},
{
"name": "Hana",
"age": 11
}
]
}
I tried the following:
jq -R 'split(" ") | { file:.[0], start_line:.[1]}'
Output
{
"name": "john",
"age": 15
}
{
"name": "Hana",
"age": 11
}
Thank you
As jq can also do the splitting, you could start before doing it with awk, having
john:15
Hana:11
Then, with -R you can read in the raw lines. Using -n in combination with [inputs] makes it an array. You can split using the / operator, and wrap everything in an object of your choice.
jq -Rn '{data: [inputs / ":" | {name: .[0], age: .[1]}]}'
{
"data": [
{
"name": "john",
"age": "15"
},
{
"name": "Hana",
"age": "11"
}
]
}
Demo
I wanna delete an index after certain time(say 10s) but it doesn't work. I researched a lot but I couldn't find a different thing from my configs. Here are my configs:
my ILM config:
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "10s",
"actions": {
"delete": {}
}
}
}
}
}
my template:
{
"order": 0,
"index_patterns": ["myindex.*"],
"settings": {
"index": {
"lifecycle": {
"name": "myindex"
}
}
}
}
my index:
{
"job_id": 1,
"message": "sample data",
"#timestamp": "DATETIME"
}
then with the following script, I test my scenario:
#!/bin/bash
NAME=myindex
# Add ILM and template.
echo -n "Add policy..."
http PUT http://localhost:9200/_ilm/policy/"${NAME}" < ilm.json
echo -n "Add template..."
http PUT http://localhost:9200/_template/"${NAME}" < template.json
# Create index.
echo -n "Create index..."
http PUT http://localhost:9200/"${NAME}" | jq '.acknowledged'
# List of data stream and indices.
echo -n "List of data stream..."
http http://localhost:9200/_data_stream/ | jq '.data_streams'
echo "List of indices..."
http http://localhost:9200/_cat/indices/ | awk '{print $3}'
echo "Index ILM explain..."
http http://localhost:9200/"${NAME}"/_ilm/explain
# Insert a document.
echo -n "Insert a document..."
sed "s/DATETIME/$(date +%Y-%m-%dT%H:%M:%S)/g" < index.json | http POST http://localhost:9200/"${NAME}"/_doc | jq -r ".result"
# Wait until it reaches the TTL.
echo "Waiting to reach the TTL..."
sleep 10 # "$(( $(jq -r .policy.phases.delete.min_age < ilm.json | sed 's/s//g') + 2 ))"
# Search for the data and expected to find nothing.
echo "Search the inserted document..."
echo '{"query": {"match": {"job_id": 1}}}' | http http://localhost:9200/"${NAME}"/_search
In the last part when I search for the index I can find it. The index is still present!!!
I set indices.lifecycle.poll_interval and it works!
PUT /_cluster/settings
{
"persistent" : {
"indices.lifecycle.poll_interval": "5s"
}
}
References:
https://discuss.elastic.co/t/index-lifecycle-management-at-elasticsearch-6-6-0/168429/2
https://stackoverflow.com/a/66138564/4604579
I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty
I'm trying to make some script to run the curl cmd.
IP=$1
WEBPORT=$2
STREAMPORT=$3
NAME=$4
curl -X POST -H "Content-Type:application/json"
-d '{"Image": "kerberos/kerberos", "ExposedPorts": {
"80/tcp": {}, "8889/tcp": {} }, "HostConfig": {
"PortBindings": { "80/tcp": [{ "HostPort": "'+$WEBPORT+ '" }],
"8889/tcp": [{ "HostPort": "'+$STREAMPORT+'" }] } } }'
http://${IP}:4243/containers/create?name=${NAME}
It' is my script
But it's not working...
is there any way to run this curl cmd in the script and make this script more comfortable by making function.
You could try the following code :
curl_post() {
curl -X POST -H "Content-Type:application/json" -d "{'Image': 'kerberos/kerberos', 'ExposedPorts': { '80/tcp': {}, '8889/tcp': {} }, 'HostConfig': { 'PortBindings': { '80/tcp': [{ 'HostPort': '$1' }],'8889/tcp': [{ 'HostPort': '$2' }] } } }" "http://$3:4243/containers/create?name=$4"
}
To invoke the function do :
curl_post $webport $streamport $ip $name
It probably needs some modification, but i hope it helps.
sorry guys, but i don't know how clone server in zabbix, so please help.
i didn't find the json request with this function in zabbix documentation, so try to create personal script, but when i get items from server and try to add it in new, have this error :
{"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid
params.","data":"Item uses host interface from non-parent
host."},"id":1}
thanks
if need of course my code :
create_host() {
a='{ "jsonrpc": "2.0", "method": "host.create", "params": { "host": "'$selected_name'", "interfaces": [ { "type": 1, "main": 1, "useip": 1, "ip": "'$IP'", "dns": "", "port": "10050" } ], "groups": [ { "groupid": "1" } ], "templates": [ { "templateid": "10001" } ], "inventory": { "macaddress_a": "01234", "macaddress_b": "56768" } }, "auth": "'$AUTH_TOKEN'", "id": 1 }'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$a"
}
push_items() {
id_of_host=`echo $3 | tr -d ']' | tr -d '[u' | sed -s "s/'//g"`
echo "name"$1
echo "key"$2
request_push='{
"jsonrpc": "2.0",
"method": "item.create",
"params": {
"name": "'$1'",
"key_": "'$2'",
"hostid": "'$id_of_host'",
"type": 10,
"value_type": 0,
"interfaceid": "2",
"delay": 300
},
"auth": "'$AUTH_TOKEN'",
"id": 1
}'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$request_push"
}
get_items() {
request='{
"jsonrpc": "2.0",
"method": "item.get",
"params": {
"output": ["name", "key_"],
"host": "'$ETALON_SERVER'",
"sortfield": "name"
},
"auth": "'$AUTH_TOKEN'",
"id": 1
}'
wget -O- -o /dev/null $API --no-check-certificate --header 'Content-Type: application/json-rpc' --post-data "$request"
}
AUTH_TOKEN=$(authenticate)
if [ -z "$AUTH_TOKEN" ]; then
echo "Connection not established"
exit 1
else
echo "everything is ok"
echo $AUTH_TOKEN
fi
check_host=$(check_exist_host)
host_create=$(create_host)
#echo "$host_create"
id_host=`echo "$host_create" | python -c 'import json, sys; print json.load(sys.stdin)["result"]["hostids"]'`
items=$(get_items)
#echo $items
keys=`echo "$items" | python -c 'import json, sys; print ("".join(i["name"]+";"+i["key_"] +"|" for i in json.load(sys.stdin)["result"]))'`
while IFS='[;]' read -r s1 s2; do
name_item=$s1
item_key=$s2
push_items "$s1" "$s2" "$id_host"
done < <(printf "%b" "${keys//|/\\n}")
As old as this question is, it still comes up in the top search results...
The API doesn't allow host.clone() because it's supposed to be performed through host.get() and host.create(), depending on your needs. The following code clones
host name
host visible name
first IP address
groups, adding a new one (static ID)
macros
and changes
templates (statid ID)
The old host is renamed and disabled, in order to maintain data history.
from pyzabbix import ZabbixAPI
ZAPI = ZabbixAPI(api_url)
ZAPI.login(username, password)
batch_list = [
"host1-asdasd",
"host2-asdasd"
]
for hostname in batch_list:
try:
original_host = ZAPI.host.get(
filter={'host': hostname},
selectGroups='extend',
selectInterfaces='extend',
selectMacros='extend'
)[0]
disable = ZAPI.host.update(
hostid=original_host['hostid'],
status=1,
host=original_host['host'] + '-history',
name=original_host['name'] + ' (history)'
)
print(disable)
clone = ZAPI.host.create(
host=original_host['host'],
name=original_host['name'],
proxy_hostid=original_host['proxy_hostid'],
groups=original_host['groups'] + [{'groupid': 802}],
macros=original_host['macros'],
interfaces=[{'main': '1', 'type': '1', 'useip': '1', 'dns': '', 'port': '10050', 'bulk': '1',
'ip': original_host['interfaces'][0]['ip']}],
templates={'templateid': 25708}
)
print(clone)
except:
print('something went wrong with: ' + hostname)