spec:
templates:
- name: some_step1
script:
image: some_image
imagePullPolicy: IfNotPresent
- name: some_step2
activeDeadlineSeconds: 300
script:
image: some_image
imagePullPolicy: IfNotPresent
env:
- name: HOST
value: "HOST"
- name: CONTEXT
value: "CONTEXT"
I would like to update the value for CONTEXT. The following command updates the value however it does not return the whole yaml:
yq '.spec.templates | map(select(.name == "some_step2")).[].script.env | map(select(.name == "CONTEXT").value = "CONTEXT1")' test.yaml
The output of the command above:
- name: HOST
value: "HOST"
- name: CONTEXT
value: "CONTEXT1"
How can I make changes in the yq command above to update the value for CONTEXT and return the whole yaml?
I am using mikefarah/yq version v4.30.6.
Use parentheses around the LHS of the assignment to retain the context, i.e. (…) = ….
( .spec.templates[]
| select(.name == "some_step2").script.env[]
| select(.name == "CONTEXT").value
) = "CONTEXT1"
spec:
templates:
- name: some_step1
script:
image: some_image
imagePullPolicy: IfNotPresent
- name: some_step2
activeDeadlineSeconds: 300
script:
image: some_image
imagePullPolicy: IfNotPresent
env:
- name: HOST
value: "HOST"
- name: CONTEXT
value: "CONTEXT1"
Related
When starting the cluster Elasticsearch 7.17 in Openshift . Cluster writes an error
chroot: cannot change root directory to '/': Operation not permitted`
Kibana started ok.
Code :
`apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: elasticsearch-pp
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: NEXUS/elasticsearch:7.17.7
env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.seed_hosts
value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "es-cluster-0,es-cluster-1,es-cluster-2"
- name: ES_JAVA_OPTS
value: "-Xms4012m -Xmx4012m"
ports:
- containerPort: 9200
name: client
- containerPort: 9300
name: nodes
volumeMounts:
- name: data-cephfs
mountPath: /usr/share/elasticsearch/data
initContainers:
- name: fix-permissions
image: NEXUS/busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
volumeMounts:
- name: data-cephfs
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: NEXUS/busybox
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args: ["-c", "sysctl -w vm.max_map_count=262144; echo vm.max_map_count=262144 >> /etc/sysctl.conf ; sysctl -p"]
# image: NEXUS/busybox
# command: ["sysctl", "-w", "vm.max_map_count=262144"]
- name: increase-fd-ulimit
image: NEXUS/busybox
command: ["sh", "-c", "ulimit -n 65536"]
serviceAccount: elk-anyuid
serviceAccountName: elk-anyuid
restartPolicy: Always
volumes:
- name: data-cephfs
persistentVolumeClaim:
claimName: data-cephfs `
I tried changing the cluster settings, disabled initContainers , the error persists
I have a YAML document like this
services:
- name: newlogd
image: NEWLOGD_TAG
cgroupsPath: /eve/services/newlogd
oomScoreAdj: -999
- name: edgeview
image: EDGEVIEW_TAG
cgroupsPath: /eve/services/eve-edgeview
oomScoreAdj: -800
- name: debug
image: DEBUG_TAG
cgroupsPath: /eve/services/debug
oomScoreAdj: -999
- name: wwan
image: WWAN_TAG
cgroupsPath: /eve/services/wwan
oomScoreAdj: -999
I need to insert a new object AFTER given element e.g. with name == "edgeview". so the output looks like this
services:
- name: newlogd
image: NEWLOGD_TAG
cgroupsPath: /eve/services/newlogd
oomScoreAdj: -999
- name: edgeview
image: EDGEVIEW_TAG
cgroupsPath: /eve/services/eve-edgeview
oomScoreAdj: -800
- name: new_element_name
image: new_element_image
- name: debug
image: DEBUG_TAG
cgroupsPath: /eve/services/debug
oomScoreAdj: -999
- name: wwan
image: WWAN_TAG
cgroupsPath: /eve/services/wwan
oomScoreAdj: -999
I couldn't find anything about it in YQ documentation. Is it even possible using YQ?
UPDATE: I'm using YQ https://github.com/mikefarah/yq version 4.28.1. I was not aware that there several tools with the same name.
Using the YAML processor yq:
yq --arg insertAfter edgeview -y '
limit(1; .services | to_entries[] | select(.value.name == $insertAfter) | .key + 1) as $idx
| .services |= .[0:$idx] +
[{name: "new_element_name", image: "new_element_image"}] +
.[$idx:]'
In the first line, the index of the element to be inserted after is determined and stored into $idx.
If you have several elements with the same name, only the first match is used (limit).
In the following filter step, $idx is used to split the array and insert the new element at the desired position.
Output
services:
- name: newlogd
image: NEWLOGD_TAG
cgroupsPath: /eve/services/newlogd
oomScoreAdj: -999
- name: edgeview
image: EDGEVIEW_TAG
cgroupsPath: /eve/services/eve-edgeview
oomScoreAdj: -800
- name: new_element_name
image: new_element_image
- name: debug
image: DEBUG_TAG
cgroupsPath: /eve/services/debug
oomScoreAdj: -999
- name: wwan
image: WWAN_TAG
cgroupsPath: /eve/services/wwan
oomScoreAdj: -999
In the mikefarah/yq (v.4.30+) you can do:
yq '.services |= (
(.[] | select(.name == "edgeview") | key + 1) as $pos |
.[:$pos] +
[{"name": "new_element_name", "image": "new_element_image"}] +
.[$pos:])' examples/data1.yaml
Explanation:
You want to update the 'services' array, so we have '.services |= '
Find the position to update the array in, which is one after then one with name "edgeview". Splat the array, find the matching entry, get its index and add one (.[] | select(.name == "edgeview") | key + 1)
Assign the value of that position to $pos.
Now you can use the array slice operator to reconstruct the array with the new element.
Add the start of the array, [:$pos] with the new element {"name": "new_element_name"} to the end of the array [$pos:]
Disclaimer: I wrote yq
I am using the .yaml file below to create Katib Experiment in Kubeflow. However, I am getting
Failed to reconcile: cannot restore struct from: string
errors. Have any solutions for this? Most of the Katib Experiment example codes doesn't have a volume in it, but I'm trying to mount a volume after downloading the data from my S3.
apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
namespace: apple
labels:
controller-tools.k8s.io: "1.0"
name: transformer-experiment
spec:
objective:
type: maximize
goal: 0.8
objectiveMetricName: Train-accuracy
additionalMetricNames:
- Train-loss
algorithm:
algorithmName: random
parallelTrialCount: 3
maxTrialCount: 12
maxFailedTrialCount: 3
metricsCollectorSpec:
collector:
kind: StdOut
parameters:
- name: --lr
parameterType: double
feasibleSpace:
min: "0.01"
max: "0.03"
- name: --dropout_rate
parameterType: double
feasibleSpace:
min: "0.005"
max: "0.020"
- name: --layer_count
parameterType: int
feasibleSpace:
min: "2"
max: "5"
- name: --d_model_count
parameterType: categorical
feasibleSpace:
list:
- "64"
- "128"
- "256"
trialTemplate:
goTemplate:
rawTemplate: |-
apiVersion: batch/v1
kind: Job
metadata:
name: {{.Trial}}
namespace: {{.NameSpace}}
spec:
template:
spec:
volumes:
- name: train-data
emptyDir: {}
containers:
- name: data-download
image: amazon/aws-cli
command:
- "aws s3 sync s3://kubeflow/kubeflowdata.tar.gz /train-data"
volumeMounts:
- name: train-data
mountPath: /train-data
- name: {{.Trial}}
image: <Our Image>
command:
- "cd /train-data"
- "ls"
- "python"
- "/opt/ml/src/main.py"
- "--train_batch=64"
- "--test_batch=64"
- "--num_workers=4"
volumeMounts:
- name: train-data
mountPath: /train-data
{{- with .HyperParameters}}
{{- range .}}
- "{{.Name}}={{.Value}}"
{{- end}}
{{- end}}
restartPolicy: Never
As answered here, the following works for me:
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: training-container
image: docker.io/romeokienzler/claimed-train-mobilenet_v2:0.4
command:
- "ipython"
- "/train-mobilenet_v2.ipynb"
- "optimizer=${trialParameters.optimizer}"
volumeMounts:
- mountPath: /data/
name: data-volume
restartPolicy: Never
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: data-pvc
I have the following yaml file.
resources:
- apiVersion: v1
kind: Deployment
metadata:
labels:
app: test
name: test-cluster-operator
namespace: destiny001
spec:
selector:
matchLabels:
name: test-cluster-operator
test.io/kind: cluster-operator
strategy:
type: Recreate
template:
metadata:
labels:
name: test-cluster-operator
test.io/kind: cluster-operator
spec:
containers:
- args:
- /path/test/bin/cluster_operator_run.sh
env:
- name: MY_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthy
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 1
name: test-cluster-operator
readinessProbe:
failureThreshold: 3
httpGet:
path: /ready
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: '1'
memory: 256Mi
requests:
cpu: 200m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/data
name: data-cluster-operator
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: test-cluster-operator
serviceAccountName: test-cluster-operator
terminationGracePeriodSeconds: 30
volumes:
- name: data-cluster-operator
persistentVolumeClaim:
claimName: data-cluster-operator
I am trying to get the the value of env variable called MY_NAMESPACE.
This is what I tried in Ansible to get to the env tree path.
- name: "set test fact"
set_fact:
myresult: "{{ yaml_file_variable | json_query(\"resources[?metadata.name=='test-cluster-operator'].spec.template.spec\") | json_query(\"containers[?name=='test-cluster-operator'].env\") }}"
- name: "debug"
debug:
msg: "{{ myresult }}"
This produces an empty list, however the first json_query works well.
How do I use json_query correctly in this case?
Can I achieve this with just one json_query?
EDIT:
I seem to be closer to a solution but the result ist a list and not string, which I find annoying.
- name: "set test fact"
set_fact:
myresult: "{{ yaml_file_variable | json_query(\"resources[?metadata.name=='test-cluster-operator'].spec.template.spec\") | json_query(\"[].containers[?name=='test-cluster-operator']\") | json_query(\"[].env[?name=='MY_NAMESPACE'].name\") }}"
This prints - - MY_NAMESPACE instead of just MY_NAMESPACE.
Do I have to use first filter every time after json_query? I know for sure that there is only one containers element. I don't understand why json_query returns a list.
This is finally working but no idea whether it's correct way to do it.
- name: "set test fact"
set_fact:
myresult: "{{ yaml_file_variable | json_query(\"resources[?metadata.name=='test-cluster-operator'].spec.template.spec\") | first | json_query(\"containers[?name=='test-cluster-operator']\") | first | json_query(\"env[?name=='MY_NAMESPACE'].valueFrom \") | first }}"
json_query uses jmespath and jmespath always returns a list. This is why your first example isn't working. The first query returns a list but the second is trying to query a key. You've corrected that in the second with [].
You're also missing the jmespath pipe expression: | which is used pretty much as you might expect - the result of the first query can be piped into a new one. Note that this is separate from ansible filters using the same character.
This query:
resources[?metadata.name=='test-cluster-operator'].spec.template.spec | [].containers[?name=='test-cluster-operator'][].env[].valueFrom
Should give you the following output:
[
{
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
]
Your task should look like this:
- name: "set test fact"
set_fact:
myresult: "{{ yaml_file_variable | json_query(\"resources[?metadata.name=='test-cluster-operator'].spec.template.spec | [].containers[?name=='test-cluster-operator'][].env[].valueFrom\") | first }}"
To answer your other question, yes you'll need to the first filter. As mentioned jmespath will always return a list, so if you just want the value of a key you'll need to pull it out.
I have yml file in below format.
Backendapp:
Name: spring-rest
Image: "testuser/backend"
ImageTag: "latest"
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
Image: "testuser/frontend"
ImageTag: "latest"
ImagePullPolicy: "Always"
replicaCount: 2
How to replace 'Image', 'ImageTag' for Backendapp through Bash?
Edit: say file name is test.yml with above contents. Then, I would like to replace 'Image' and 'ImageTag' with some another value(say 'teststring','latest2' resp.) for 'Backendapp' part.
Modified file should look like below.
Backendapp:
Name: spring-rest
Image: "teststring"
ImageTag: "latest2"
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
Image: "testuser/frontend"
ImageTag: "latest"
ImagePullPolicy: "Always"
replicaCount: 2
Following awk may help you on same:
awk '
/:$/{
flag=""
}
/Backendapp/||/Frontendapp/{
flag=1
}
flag && NF && (/Image:/||/ImageTag:/){
match($0,/^[[:space:]]+/);
val=substr($0,RSTART,RLENGTH);
$NF="teststring";
print val $0;
next
}
1
' Input_file
Output will be as follows:
Backendapp:
Name: spring-rest
Image: teststring
ImageTag: teststring
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
Image: teststring
ImageTag: teststring
ImagePullPolicy: "Always"
replicaCount: 2
Solution 2nd: As per OP, OP needs to have shell variable into awk variable too then following may help on same:
awk -v val_shell="$shell_variable" '
/:$/{
flag=""
}
/Backendapp/||/Frontendapp/{
flag=1
}
flag && NF && (/Image:/||/ImageTag:/){
match($0,/^[[:space:]]+/);
val=substr($0,RSTART,RLENGTH);
$NF=val_shell;
print val $0;
next
}
1
' Input_file
Here's your solution. It basically uses SED to do the replacement. Just copy and paste the bits between "START OF SCRIPT" and "END OF SCRIPT" into any bash script then run (simple instructions below):
START OF SCRIPT ...
#!/bin/bash
# SED script will be stored here ...
sedscript=/tmp/sedscript.sed
# first argument to script is the parent name ...
parent="${1}"
# second argument to the script is child name...
child="${2}"
# third argument is the value to replace the CHILD value with ...
value="$3"
cat << EOF > "${sedscript}"
# find parent ...
/^${parent}:.*$/{
p
# load next line into input ...
n
# another parent found to print it ...
/^[a-z|A-Z|0-9]\{1\}:.*$/{
# we have found another parent
# so stop processing ...
b print_default_text
}
:search_for_child
/^[ | ]*$child[ | ]*:.*$/{
s/^\([^:]*:[ |\"| ]*\)[^\"]*\(.*\)$/\1$value\2/p
b dont_print_default_text
}
p
n
b search_for_child
}
:print_default_text
p
:dont_print_default_text
EOF
sed -n -f "${sedscript}" "input.yml"
... END OF SCRIPT
Now we save the script as alter.sh (for example) then run it like this (changing the Frontendapp/Image to "another-image-name"):
./alter.sh Frontendapp Image "another-image-name"
INPUT FILE ....
Backendapp:
Name: spring-rest
Image: "teststring"
ImageTag: "latest2"
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
Image: "testuser/frontend"
ImageTag: "latest"
ImagePullPolicy: "Always"
replicaCount: 2
Output file ....
Backendapp:
Name: spring-rest
Image: "teststring"
ImageTag: "latest2"
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
Image: "another-image-name"
ImageTag: "latest"
ImagePullPolicy: "Always"
replicaCount: 2
Note: the script takes care of quotes too, i.e. if the original value has quotes then the new value will be surrounded in quotes also.
Hope this helps ....
Awk solution:
awk 'NF==1 && /^[^[:space:]]+:/{ f=(/^(Back|Front)endapp:/) }
f && $1 ~ /^Image(Tag)?:/{ $1=" teststring" }1' test.yml
The output:
Backendapp:
Name: spring-rest
teststring: "testuser/backend"
teststring: "latest"
ImagePullPolicy: "Always"
Port: 8080
replicaCount: 2
Frontendapp:
Name: spring-js
teststring: "testuser/frontend"
teststring: "latest"
ImagePullPolicy: "Always"
replicaCount: 2