Related
I have a YAML file which I am converting into JSON files using yq.
It generates the following output,
{
"a" : 1,
"b" : 2,
"c" : {
"id": "9ee ...",
"parent": "abc..."
}
}
Then I am creating another JSON based on the above JSON key & value. Please find the below code snippet,
# Extract the properties from the YAML file
json=$(yq -j "$file_name")
# Iterate over the properties
parameters=""
for key in $(echo "${json}" | jq -r 'keys[]'); do
# Extract the key and value of the property
value=$(echo "${json}" | jq -r ".$key")
echo "Adding parameter $key with value $value to SSM"
# Add the property to the list of parameters
parameters+="{\"Name\": \"$key\", \"Value\": \"$value\", \"Type\": \"String\", \"Overwrite\": true}"
done
Since the value in the 1st JSON is already JSON so we couldn't able to generate the 2nd JSON which is generating an invalid-JSON error.
Is there any way to escape/stringify the JSON characters in bash script or jq, so that we can able to generate the 2nd JSON?
Any help would be really appreciated.
Actual output:
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "c",
"Value": "{
"id": "9ee ...",
"parent": "abc..."
}",
"Type": "String",
"Overwrite": "true"
}
]
The above one is not a valid JSON.
Expected output:
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": "true"
},
{
"Name": "c",
"Value": "{\r\n \"id\": \"9ee ...\",\r\n \"parent\": \"abc...\"\r\n }",
"Type": "String",
"Overwrite": "true"
}
]
Why try to emulate jq's behavior with a shell loop (which should generally be avoided) instead of using jq directly?
yq -j "$file_name" | jq 'to_entries | map({
Name: .key,
Value: .value,
Type: "String",
Overwrite: true
})'
Or directly transform using yq only:
yq 'to_entries | map({
Name: .key,
Value: .value,
Type: "String",
Overwrite: true
})' -j "$file_name"
Update after clarifying edit of the question: It has become clear that you want to transform the value into a string. jq has the tostring filter for that, the program thus becomes:
to_entries | map({
Name: .key,
Value: (.value | tostring),
Type: "String",
Overwrite: true
})
Note that this will not keep the line breaks and indents, but formats the JSON object in a "compact" way. Let us know if that's a problem.
$ jq 'to_entries | map({
Name: .key,
Value: (.value | tostring),
Type: "String",
Overwrite: true
})' <<JSON
{
"a": 1,
"b": 2,
"c": {
"id": "9ee ...",
"parent": "abc..."
}
}
JSON
[
{
"Name": "a",
"Value": "1",
"Type": "String",
"Overwrite": true
},
{
"Name": "b",
"Value": "2",
"Type": "String",
"Overwrite": true
},
{
"Name": "c",
"Value": "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}",
"Type": "String",
"Overwrite": true
}
]
This should achieve what's expected :
jq 'to_entries | map({
Name: .key,
Value: (.value|if (type == "object")
then tojson
else tostring
end),
Type: "String",
Overwrite: true})
' input.json
To JSON-encode data from within jq you can use the tojson (or #json) builtins.
Also, to get the actual type of some data, there is the type function.
Maybe you are trying to accomplish something like this:
# Convert the the YAML file into JSON
json="$(yq -j "$file_name")"
# Transform the JSON data into desired format
jq 'to_entries | map({Name: .key} + (.value | {
EncodedValue: tojson,
OriginalType: type,
Overwrite: true
}))' <<< "$json" > "$new_file_name"
Checking the contents of the new file would now give you something like:
[
{
"Name": "a",
"EncodedValue": "1",
"OriginalType": "number",
"Overwrite": true
},
{
"Name": "b",
"EncodedValue": "2",
"OriginalType": "number",
"Overwrite": true
},
{
"Name": "c",
"EncodedValue": "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}",
"OriginalType": "object",
"Overwrite": true
}
]
I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty
I have to parse a JSON which includes an array of of class/name:
{
"_class": "model.ListView",
"jobs": [
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-different-node-full"
},
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-jms-activemq-full"
},
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-txpropag-jpa-full"
}
]
}
I need to retrieve the list of "name".
Looking at the examples I've found of jq, I have tried with:
cat jobs.json | jq '.[].name'
It fails with:
jq: error (at <stdin>:0): Cannot index string with string "name"
How should I reference the name element of the array?
Thanks
found it:
cat jobs.json | jq '.jobs[].name'
I'm creating a bash script to provision multiple Azure resources via the Azure CLI. So far so good, however I'm having a problem tagging resources.
My goal is to store multiple tags in a variable and provide that variable to the --tags option of several az commands in the script. The problem however is that a space in the value will be interpreted as a new key.
If we take for example the command az group update (which will update a resource group) the docs state the following about the --tags option:
--tags
Space-separated tags in 'key[=value]' format. Use "" to clear existing tags.
When a value (or key) contains spaces it must be enclosed in quotes.
So when we provide the key-value pairs directly to the command including a value with spaces, like in the following example, the result will be as expected:
az group update --tags owner="FirstName LastName" application=coolapp --name resource-group-name
The result will be that two tags have been added to the resource group:
{
"id": "/subscriptions/1e42c44c-bc55-4b8a-b35e-de1dfbcfe481/resourceGroups/resource-group-name",
"location": "westeurope",
"managedBy": null,
"name": "resource-group-name",
"properties": {
"provisioningState": "Succeeded"
},
"tags": {
"application": "coolapp",
"owner": "FirstName LastName"
}
}
However, when we store the same value we used in the previous step in a variable the problem occurs.
tag='owner="FirstName LastName" application=coolapp'
I use echo $tag to validate that the variable contains exactly the same value as we provided in the previous example to the --tags option:
owner="FirstName LastName" application=coolapp
But when we provide this tag variable to the tags option of the command as shown in the next line:
az group update --tags $tag --name resource-group-name
The result will be three tags instead of the expected two:
{
"id": "/subscriptions/1e42c44c-bc55-4b8a-b35e-de1dfbcfe481/resourceGroups/resource-group-name",
"location": "westeurope",
"managedBy": null,
"name": "resource-group-name",
"properties": {
"provisioningState": "Succeeded"
},
"tags": {
"LastName\"": "",
"application": "coolapp",
"owner": "\"FirstName"
}
}
I've already tried defining the variable in the following ways, but no luck so far:
tag="owner=FirstName LastName application=coolapp"
tag=owner="Firstname Lastname" application=cool-name
tag='`owner="Firstname Lastname" application=cool-name`'
I even tried defining the variable as an array and providing it to the command as shown on the next line, but also that didn't provide the correct result:
tag=(owner="Firstname Lastname" application=cool-name)
az group update --tags ${tag[*]}--name resource-group-name
I also tried putting quotes around the variable in the command, as was suggested by #socowi, but this leads to the following incorrect result of one tag instead of two:
az group update --tags "$tag" --name resource-group-name
{
"id": "/subscriptions/1e42c44c-bc55-4b8a-b35e-de1dfbcfe481/resourceGroups/resource-group-name",
"location": "westeurope",
"managedBy": null,
"name": "resource-group-name",
"properties": {
"provisioningState": "Succeeded"
},
"tags": {
"owner": "Firstname Lastname application=cool-name"
}
}
Does anybody know how to solve this?
Define your tags as
tags=("owner=Firstname Lastname" "application=cool-name")
then use
--tags "${tags[#]}"
I've found the following works. It requires a resource group already be created.
I used the following template:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"resourceName": {
"type": "string",
"metadata": {
"description": "Specifies the name of the resource"
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for the resources."
}
},
"resourceTags": {
"type": "object",
"defaultValue": {
"Cost Center": "Admin"
}
}
},
"resources": [
{
"apiVersion": "2019-06-01",
"kind": "StorageV2",
"location": "[parameters('location')]",
"name": "[parameters('resourceName')]",
"properties": {
"supportsHttpsTrafficOnly": true
},
"sku": {
"name": "Standard_LRS"
},
"type": "Microsoft.Storage/storageAccounts",
"tags": "[parameters('resourceTags')]"
}
]
}
In the Azure CLI using Bash, you can pass in the tag as a JSON object. In the following example, a template file with a location requires two parameters, resourceName and the tags which is an ARM object named resourceTags:
az deployment group create --name addstorage --resource-group myResourceGroup \
--template-file $templateFile \
--parameters resourceName=abcdef45216 resourceTags='{"owner":"bruce","Cost Cen":"2345-324"}'
If you want to pass it as an environment variable, use:
tags='{"owner":"bruce","Cost Center":"2345-324"}'
az deployment group create --name addstorage --resource-group myResourceGroup \
--template-file $templateFile \
--parameters resourceName=abcdef4556 resourceTags="$tags"
The $tags must be in double quotes. (You are passing in a JSON object string)
The JSON string also works when you are passing in the tags into Azure DevOps pipeline. See https://github.com/MicrosoftDocs/azure-devops-docs/issues/9051
First, build your string like so and double quote all keys/values just in case of spaces in either: (Sorry this is PoSH just example)
[string] $tags = [string]::Empty;
97..99 |% {
$tags += "&`"$([char]$_)`"=`"$($_)`"";
}
The results of this is a string "&"a"="97"&"b"="98"&"c"="99".
Now pass it as a string array using the split function of the base string class which results in a 4 element array, the first element is blank. The CLI command ignores the first empty element. Here I set the tags for a storage account.
$tag='application=coolapp&owner="FirstName LastName"&"business Unit"="Human Resources"'
az resource tag -g rg -n someResource --resource-type Microsoft.Storage/storageaccounts -tags $tag.split("&")
I also employed this approach when I wanted to override the parameters provided in a parameter file for a resource group deployment.
az group deployment create --resource-group $rgName --template-file $templatefile --parameters $parametersFile --parameters $($overrideParams.split("&"));
I have a json file like this:
{
"name": "Job",
"type": "xdb",
"typeLogoUrl": "public/app/plugins/logo.svg",
"access": "proxy",
"url": "http://xdb:80",
"password": {},
"user": "xx",
"database": "Job",
"basicAuth": true,
"basicAuthUser": "xx",
"basicAuthPassword": {},
"withCredentials": true,
"isDefault": false,
"jsonData": {},
"secureJsonFields": null
}
Now all I want is to pass environment variables to key password and basicAuthPassword and to generate new json file as below:
{
"name": "Job",
"type": "xdb",
"typeLogoUrl": "public/app/plugins/logo.svg",
"access": "proxy",
"url": "http://xdb:80",
"password": "password" ,
"user": "xx",
"database": "Job",
"basicAuth": true,
"basicAuthUser": "xx",
"basicAuthPassword": "password",
"withCredentials": true,
"isDefault": false,
"jsonData": {},
"secureJsonFields": null
}
I have tried like below to replace at least one key and it's giving me null.
/usr/local/bin/jq -n --arg "password" '.password = $arg' < input.json
Can anyone suggest me how to achieve this?
Security wise, it's a bad idea to pass password in command line, but I hope you know what you're doing.
Assuming that, you need to modify your jq command like this:
jq --arg p "password" '.password = $p | .basicAuthPassword = $p' < input.json
If the password is already an environment variable (which may not be such a good idea though), then to avoid exposing it on the command-line, you could use the env builtin, along the lines of:
.password = env.password
etc.
An alternative to consider would be putting the password into a temporary file, and then using the --argfile command-line option, for example. Or if your shell supports it, --argfile pw <(echo "\"$password\"")