I want to do this:
ENV=development cat dbconfig.json | jq '.database.$ENV'
where config looks like:
{
"database": {
"development": {
"name": "example",
"user": "user",
"password": "asdfasdf"
}
}
}
I want to specify the ENV in the command and use it to fetch the key "development" from the json.
In a makefile specifically, I would like to do:
# ensure env is set
ifeq ($(ENV),)
$(error ENV is not set)
endif
CONFIG := $(shell cat config.json)
DATABASE_NAME = $(shell echo CONFIG | jq 'database.$(ENV).name')
migrate:
#migrate -source backend/migrations -database postgres://localhost:5432/$(DATABASE_NAME) up
.PHONY: migrate
I am new to jq, and somewhat to makefiles, how could this be accomplished?
Using a shell variable may not be the best idea, but if that's what you want, this should give you the hint you're looking for:
ENV=development jq -n 'env.ENV'
Using this approach, though, you'd have to provide the filename as an argument to jq.
Notice that using this approach avoids any conflict with jq's "$-variables".
Related
I realize this is a simple question but I haven't been able to find the answer. Thank you to anyone who may be able to help me understand what I am doing wrong.
Goal: Search and replace a string in a specific key in a JSON file with a string in a Bash variable using jq.
For example, in the following JSON file:
"name": "Welcome - https://docs.mysite.com/",
would become
"name": "Welcome",
Input (file.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome - https://docs.mysite.com/",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
Failing script (using variable)
siteUrl="docs.mysite.com"
jq --arg siteUrl "$siteUrl" '.[].name|= (gsub(" - https://$siteUrl/"; ""))' file.json > file1.json`
Desired output (file1.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
I've tried various iterations on removing quotes, changing between ' and ", and adding and removing backslashes.
Successful script (not using variable)
jq '.[].name|= (gsub(" - https://docs.mysite.com/"; ""))' file.json > file1.json
More specifically, if it matters, I am processing an export of a website's usage data from Azure App Insights. Unfortunately, the same page may be assigned different names. I sum the Ocurrences of the two objects with the newly identical url later. If it is better to fix this in App Insights I am grateful for that insight, although I know Bash better than Kusto queries. I am grateful for any help or direction.
Almost. Variables are not automatically expanded within a string. You must interpolate them explicitly with \(…):
jq --arg siteUrl 'docs.mysite.com' '.[].name |= (gsub(" - https://\($siteUrl)/"; ""))' file.json
Alternatively, detect a suffix match and extract the prefix by slicing:
jq --arg siteUrl 'docs.mysite.com' '" - https://\($siteUrl)/" as $suffix | (.[].name | select(endswith($suffix))) |= .[:$suffix|-length]' file.json
The below value of private_localaddress needs to passed to "range_start" in yaml file. how to do this ?
Here is the script:
#!/bin/bash
MAC=`curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/`
myarr=($MAC)
for val in ${myarr[#]}; do
interfaceindex=`curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$val/device-number`
if [ $interfaceindex == 2 ];
then
private_localaddress=`curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$val/local-ipv4s`
echo $private_localaddress
fi
done
Here is the YAML file :
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: worker-private-eth2
spec:
config: '{
"cniVersion": "0.3.1",
"ipam": {
"type": "whereabouts",
"range": "10.30.11.0/25",
"range_start": $private_localaddress,
"range_end": "10.30.11.127",
"routes": [
{ "dst": "10.93.123.0/24", "gw": "10.30.11.1" }
],
Ffffff
Use a proper yaml parser. Here are some examples:
Using mikefarah/yq:
value="$private_localaddress" yq \
'.spec.config |= (fromjson | .ipam.range_start = strenv(value) | tojson)'
Using kislyuk/yq:
yq -y --arg value "$private_localaddress" \
'.spec.config |= (fromjson | .ipam.range_start = $value | tojson)'
Using itchyny/gojq:
gojq --arg value "$private_localaddress" --yaml-input --yaml-output \
'.spec.config |= (fromjson | .ipam.range_start = $value | tojson)'
Perhaps the simplest would be to make private_localaddress an environment variable and use envsubst to apply the changes to your YAML.
However to end up with correct YAML code, you need to ensure that you have a quoted string to the right of the colon. The best would be if you can already ensure that your YAML is written as
"range_start": "$private_localaddress",
If this is not feasible, you have to add the quotes when creating the variable, i.e.
export private_localaddress=\"$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$val/local-ipv4s)\"
My script snippet is as below:
End goal to accomplish is to create a Azure DevOps variable group and inject key-values from another variable group into it(the newly created Azure DevOps Variable Group)
set -x
echo "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" | az devops login --organization https://dev.azure.com/tempcompenergydev
az devops configure --defaults organization=https://dev.azure.com/tempcompenergydev project=Discovery
export target_backend=automation
export target_backend="tempcomp.Sales.Configuration.Spa ${target_backend}"
export new_env="abc"
values=(addressSearchBaseUrl addressSearchSubscriptionKey cacheUrl calendarApiUrl checkoutBffApiUrl cpCode)
az_create_options=""
for ptr in "${values[#]}"
do
result=$(
az pipelines variable-group list --group-name "${target_backend}" | jq '.[0].variables.'${ptr}'.value'
)
printf "%s\t%s\t%d\n" "$ptr" "$result" $?
# add the variable and value to the array
az_create_options="${az_create_options} ${ptr}=${result}"
done
az pipelines variable-group create \
--name "test ${new_env}" \
--variables "${az_create_options}"
However, when the above command executes, I get an unexpected output as below:
+ az pipelines variable-group create --name 'test abc' --variables ' addressSearchBaseUrl="https://qtruat-api.platform.tempcomp.com.au/shared" addressSearchSubscriptionKey="xxxxxxxxxxxxxxxxxxx" cacheUrl="https://tempcompdstqtruat.digital.tempcomp.com.au/app/config" calendarApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/" checkoutBffApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/" cpCode="1067076"'
cpCode "1067076" 0
WARNING: Command group 'pipelines variable-group' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
{
"authorized": false,
"description": null,
"id": 1572,
"name": "test abc",
"providerData": null,
"type": "Vsts",
"variables": {
"addressSearchBaseUrl": {
"isSecret": null,
"value": "\"https://qtruat-api.platform.tempcomp.com.au/shared\" addressSearchSubscriptionKey=\"xxxxxxxxxxxxxxxxxxxxxxxxx\" cacheUrl=\"https://tempcompdstqtruat.digital.tempcomp.com.au/app/config\" calendarApiUrl=\"https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/\" checkoutBffApiUrl=\"https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/\" cpCode=\"1067076\""
}
}
}
##[section]Finishing: Bash Script
On a side note, If I run manually, I do get correct response. Example below:
az pipelines variable-group create --name "test abc" --variables addressSearchBaseUr="https://qtruat-api.platform.tempcomp.com.au/shared" addressSearchSubscriptionKey="xxxxxxxxxxxxxxxxxxxxxxxxx" cacheUrl="https://tempcompdstqtruat.digital.tempcomp.com.au/app/config" calendarApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/" checkoutBffApiUrl="https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/" cpCode="1067076"
Output:
+ az pipelines variable-group create --name 'test abc' --variables addressSearchBaseUr=https://qtruat-api.platform.tempcomp.com.au/shared addressSearchSubscriptionKey=xxxxxxxxxxxxxxxxxxx cacheUrl=https://tempcompdstqtruat.digital.tempcomp.com.au/app/config 'calendarApiUrl=https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/' 'checkoutBffApiUrl=https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/' cpCode=1067076
WARNING: Command group 'pipelines variable-group' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
{
"authorized": false,
"description": null,
"id": 1573,
"name": "test abc",
"providerData": null,
"type": "Vsts",
"variables": {
"addressSearchBaseUr": {
"isSecret": null,
"value": "https://qtruat-api.platform.tempcomp.com.au/shared"
},
"addressSearchSubscriptionKey": {
"isSecret": null,
"value": "xxxxxxxxxx"
},
"cacheUrl": {
"isSecret": null,
"value": "https://tempcompdstqtruat.digital.tempcomp.com.au/app/config"
},
"calendarApiUrl": {
"isSecret": null,
"value": "https://qtruat-api.platform.tempcomp.com.au/sales/calendar/v1;rev=deadpool/AvailableDates/EnergyMovers/"
},
"checkoutBffApiUrl": {
"isSecret": null,
"value": "https://qtruat-api.platform.tempcomp.com.au/sales/checkout-experience/v1;rev=deadpool/"
},
"cpCode": {
"isSecret": null,
"value": "1067076"
}
}
}
##[section]Finishing: Bash Script
Work Around
I am not a bash pro, but I found a solution that should work for you. I think the core of the issue is that when you print out your array there, it is treating the set of --variables arguments like they're one long string. So you're getting this...
az pipelines variable-group create --name "test ${new_env}" --variables ' addressSearchBaseUrl="val" addressSearchSubscriptionKey="val" ...'
instead of this...
az pipelines variable-group create --name "test ${new_env}" --variables addressSearchBaseUrl="val" addressSearchSubscriptionKey="val" ...
Bad: I think you could get around this using eval instead of printing out arguments but using eval in almost any situation is advised against.
Good: Instead, I thought this problem could be tackled a different way. Instead of doing a batch where all variables are created at once, this script will copy the desired variables to a new group one at a time:
set -x
echo "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" | az devops login --organization https://dev.azure.com/tempcompenergydev
az devops configure --defaults organization=https://dev.azure.com/tempcompenergydev project=Discovery
export target_backend=automation
export target_backend="tempcomp.Sales.Configuration.Spa ${target_backend}"
export new_env="abc"
values=(addressSearchBaseUrl addressSearchSubscriptionKey cacheUrl calendarApiUrl checkoutBffApiUrl cpCode)
az_create_options=()
for ptr in "${values[#]}"
do
result=$( az pipelines variable-group list --group-name "${target_backend}" | jq '.[0].variables.'${ptr}'.value' )
echo "$result"
# Result from this az command always wraps the response in double quotes. They should be removed.
stripQuote=$( echo $result | sed 's/^.//;s/.$//' )
printf "%s\t%s\t%d\n" "$ptr" "$stripQuote" $?
# add the variable and value to the array
# When adding this way you ensure each key|value pair gets added as a separate element
az_create_options=("${az_create_options[#]}" "${ptr}|${stripQuote}")
# ^ Note on this: The | is used because there needs to be a way to separate
# the key from the value. What you're really trying to do here is an
# Associative Array, but not all versions of bash support that out of the
# box. This is a bit of a hack to get the same result. CAUTION if your
# variable value contains a | character in it, it will cause problems
# during the copy.
done
# Create the new group, save the group id (would be best to add a check to see if the group by this name exists or not first)
groupId=$( az pipelines variable-group create --name "test ${new_env}" --variables bootstrap="start" | jq '.id' )
for var in "${az_create_options[#]}"
do
# Split the key|value pair at the | character and use the set to create the new variable in the new group
# Some check could also be added here to see if the variable already exists in the new group or not if you want to use this create or update an existing group
arrVar=(${var//|/ })
echo "Parsing variable ${arrVar[0]} with val of ${arrVar[1]}"
az pipelines variable-group variable create \
--id "${groupId}" \
--name "${arrVar[0]}" \
--value "${arrVar[1]}"
done
# This is needed cause the az command is dumb and won't let you create a group in the first place without a --variables argument. So I had it create a dummy var and then delete it
az pipelines variable-group variable delete --id "${groupId}" --name "bootstrap" --yes
Notes to call out
I would suggest using an Associative Array instead of the array I listed here if possible. However, Associative Arrays are only supported in bash v4 or higher. Use bash --version to see if you'd be able to use them. See this guide as an example of some ways you could work with those if you're able.
If using my method, be wary that any | character in a variable value that's being copied will cause the script to fail. You may need to pick a different delimiter to split on.
The output of the az pipelines variable-group list command will give the values of the variables wrapped in ". If you try to turn around and throw the exact result you get from the list at a variable create command, you will get a bunch of variables in your group with values like
{
"variable": {
"isSecret": null,
"value": "\”value\”"
}
}
instead of
{
"variable": {
"isSecret": null,
"value": "value"
}
}
The az command is dumb and won't let you create a new variable group in the first place without a --variables argument. So I had it create a dummy var bootstrap="start" and then delete it.
As I mentioned, there may be a better way to print out the array that I'm missing. Comments on my post are encouraged if you know a better way.
This might help....
There is an open issue on how Azure hosted agents authenticate back to ADO using the az-pipelines command.
Feel this is related but if it's not feel free to respond and I'll remove the answer.
I have the following JSON file:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
In the terminal, using bash as shell, I can execute the following commands:
export schedules="1"
echo $(jq '.[env.schedules]["media_content"]' json_file.json)
Which results in outputing this:
test3.xspf
So it works as expected, but when I place that jq command in a script and run it, it just returns null.
I did echo the values of schedules to make sure the value is non-null inside the script, and it is ok:
echo $schedules
But I did not manage to find the reason, why this command works when run directly in shell and does not work when run in script.
I run the script in the following ways:
bash script.sh
./script.sh
PS: yes, I did offer execute permission: chmod +x script.sh
HINT: env.schedules represents the environment variable 'schedules', and I did make sure that it is assigned in the script before calling jq.
EDIT: I am posting now a whole script, specifying the files tree.
There is one directory containing:
script.sh
json_file.json
static.json
script.sh:
export zone=$(cat static.json | jq '.["1"]');
echo "json block: "$zone
export schedules="$(echo $zone | jq '.schedules')"
echo "environment variable: "$schedules
export media_content=$(jq '.[env.schedules]["media_content"]' json_file.json)
echo "What I want to get: \"test3.xspf\""
echo "What I get: "$media_content
json_file.json:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
static.json:
{
"1":
{
"x": "0",
"y": "0",
"width": "960",
"height": "540",
"schedules":"1"
}
}
If I run the script, it displays:
json block: { "x": "0", "y": "0", "width": "960", "height": "540", "schedules": "1" }
environment variable: "1"
What I want to get: "test3.xspf"
What I get: null
If I hardcode the variable:
export schedules="1"
The problem no longer occurs
The problem is simple.
It's not jq's fault.
It the unproper way the schedule's value is piped to the next command.
You have to remove the "s that surround the variable's value, add the second command that uses sed to do that:
export schedules="$(echo $zone | jq '.schedules')"
schedules=$( echo $schedules | sed s/\"//g )
Long answer
Let's see:
here schedules is a string and echo shows its value as being 1:
export schedules="1" ; echo $schedules
here even though double quotes are not mentioned:
export schedules=1 ; echo $schedules
But the result from this also generates additional "s:
export schedules=$(echo $zone | jq '.schedules')
If you print it now you will see additional "s:
echo $schedules # "1"
So just remove the "s from the value:
schedules=$( echo $schedules | sed s/\"//g )
I am having below code in JSON file.
{
"comment": {
"vm-updates": [],
"site-ops-updates": [
{
"comment": {
"message": "You can start maintenance on this resource"
},
"hw-name": "Machine has got missing disks. "
}
]
},
"object_name": "4QXH862",
"has_problems": "yes",
"tags": ""
}
I want to separate "hw-name" from this JSON file using jq. I've tried below combinations, but nothing worked.
cat jsonfile | jq -r '.comment[].hw-name'
cat json_file.json | jq -r '.comment[].site-ops-updates[].hw-name'
Appreciated help from StackOverflow!!!
It should be:
▶ cat jsonfile | jq -r '.comment."site-ops-updates"[]."hw-name"'
Machine has got missing disks.
Or better still:
▶ jq -r '.comment."site-ops-updates"[]."hw-name"' jsonfile
Machine has got missing disks.
From the docs:
If the key contains special characters, you need to surround it with double quotes like this: ."foo$", or else .["foo$"].