I need to have an "anchor" file and then several files that "inherit" from it. is it possible?
file1:
tests: &flow1
flow:
- simulaor
test:
- test1
- test2
- test2
file2:
run_tests:
<<: *flow1
Related
I am attempting to merge two yaml files and would like any shared keys under a specific key to use values from one of the yaml files, and not merge both. This problem may be better described using an example. GIven file1.yaml and file2.yaml, I am trying to achieve the following:
file1.yaml
name: 'file1'
paths:
path1:
content: "t"
path2:
content: "b"
file2.yaml
name: 'file2'
paths:
path1:
value: "t"
My ideal result in merging is the following file:
file3.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Specifically, I would like to overwrite any key under paths such that if both yaml files have the same key under paths, then only use the value from file2. Is there some tool that enables this? I was looking into yq but I'm not sure if that tool would work
Please specify which implementation of yq you are using. They are quite similar, but sometimes differ a lot.
For instance, using kislyuk/yq, you can use input to access the second file, which you can provide alongside the first one:
yq -y 'input as $in | .name = $in.name | .paths += $in.paths' file1.yaml file2.yaml
name: file2
paths:
path1:
value: t
path2:
content: b
With mikefarah/yq, you'd use load with providing the second file in the code, while only the first one is your regular input:
yq 'load("file2.yaml") as $in | .name = $in.name | .paths += $in.paths' file1.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Assume I have a file root.yml
keyA: valA
keyB: valB
myList:
Then I receive some yml file, such as
1.yml
project_id: abc
description: xyz
2.yml
project_id: cba
description: zyx
And so on (they may stored in same folder)
Now I want to append the content of 1.yml, 2.yml (and so on) to the myList of root.yml and output to console
Expected:
keyA: valA
keyB: valB
myList:
- project_id: abc
description: xyz
- project_id: cba
description: zyx
- (so on...)
I have searched some examples but they hard code the list item in the yq command, like this post: Stack Overflow
But I want it load from files, not from hard code
Please forgive for my bad english
With mikefarah/yq, you could use the load function with the filename of the YAML files to be included, i.e. with your example
yq '.myList += [ load("1.yaml"), load("2.yaml") ]' root.yml
producing a YAML result as
keyA: valA
keyB: valB
myList:
- project_id: abc
description: xyz
- project_id: cba
description: zyx
As indicated in your comment, if one of the object has a parent structure and you want to extract the element from it, you can do
yq 'load("1.yaml") as $f | .myList += [ $f.config[], load("2.yaml") ]' root.yml
Tested on yq version 4.27.2
I am assembling a YAML build pipeline using bash as follows.
cat <<EOT >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: ???
$(cat schedule.yml)
tz: ${TZ}
EOT
What I want is to store the following YAML into schedule as a string which I can reuse in a another part of the pipeline.
version: 1.4
jobs:
DockerJob:
cmd: docker ps
time: "*"
notifyOnSuccess:
- type: stdout
data:
- stdout
But it seems it needs to be indented.
You can use the pr utility:
cat <<EOF >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: $(printf "\n" && pr -to 8 schedule.yml)
tz: ${TZ}
EOT
I use printf "\n" because you'd need to place the $(…) at the first column if you want to write $(…) on a new line, since every line including the first one will be offset by the given number of spaces.
Suppose I have two similar files:
a.yaml
data:
- name: a1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: a2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-a"]
values: ["else-in-a"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
b.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: b-value
key2: b-value
key3: b-value
My goal is to combine parts of a and b file such that I have a new file which consists of file content before substitutions: from b.yaml and content including and after substitutions: from a.yaml
So in this case, my desired output would be like this:
c.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
The parts before and after substitutions: in both file contents might have different lengths.
Currently, my method is like this:
head -q -n `awk '/substitution/ {print FNR-1}' b.yaml` b.yaml >! c.yaml ; \
tail -q -n `awk '/substitution/ {ROWNUM=FNR} END {print NR-ROWNUM+1}' a.yaml` a.yaml >> c.yaml; \
rm a.yaml b.yaml; mv c.yaml a.yaml; # optional newfile renaming to original
But I wonder if there's an alternative or better method for combining parts of different files based on a common text string in bash?
Use awk, you just need to flag the flow based on the string:
awk '$1 == "substitutions:"{skip = FNR==NR ? 1:0}!skip' b.yaml a.yaml
Explaination:
FNR==NR: if true, process lines in the first file b.yaml, otherwise the 2nd file a.yaml
!skip: if TRUE, print the line, otherwise skip the line.
{
head -B9999 'substitutions:' a.yaml | head -n -1
head -A9999 'substitutions:' b.yaml
} > c.yaml
A oneliner:
{ head -B9999 'substitutions:' a.yaml | head -n -1; head -A9999 'substitutions:' b.yaml; } > c.yaml
The -A9999 and -B9999 are a bit dirty, here's a solution with sed's:
{
sed '/substitutions:/,$d' a.yaml
echo substitutions:
sed '1,/substitutions:/d' b.yaml
} > c.yaml
This may be a very noob question, but I have no idea of how to solve it. I have a C program and a folder called "input". Is it possible to write rules that allow me to type "make run" and then execute all test instances, writing the result into a single file?
I imagine that it's something like:
run: test1 test2 test3
test1:
./myproj < arq1.in > output.out
test2:
./myproj < arq2.in > output.out
...
But how do I achieve that everyone writes into the same file, ie, appending the results, instead of rewriting it? Is it possible?
Thanks in advance
First switch to ">>" to append to the output file:
run: test1 test2 test3
test1:
./myproj < arq1.in >> output.out
test2:
./myproj < arq2.in >> output.out
test3:
./myproj < arq3.in >> output.out
Then make those targets PHONY, because they are (i.e. they do not build files named "test1", "test2" and "test3"):
.PHONY: test1 test2 test3
Then switch to a static pattern rule:
run: test1 test2 test3
.PHONY: test1 test2 test3
test1 test2 test3: test%:
./myproj < arq$*.in >> output.out
Further refinements are possible, but that's enough for now.