Use yq to update array value - shell

Question

I have two yaml files I am using. The first yaml file looks like this:
spring.yml
spring:
cloud:
gateway:
routes:
- id: someid
uri: someUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someOtherPath
I have another file that just contains routes and looks like this:
routes.yml
routes:
- id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
My goal is to update the route in the first file with the value of the route in the second file. Note that the first file in reality will have many routes but for demonstration purposes I am only showing the first in this example. I have the following script which loops through to update the routes as necessary when the id's match:
#!/bin/sh
OVERRIDE_ROUTE_IDS=$(yq eval '.routes.[].id' routes.yml)
GENERATED_ROUTE_IDS=$(yq eval '.spring.cloud.gateway.routes.[].id' spring.yml)
SAVEIFS=$IFS # Save current IFS (Internal Field Separator)
IFS=$'\n' # Change IFS to newline char
OVERRIDE_ROUTE_IDS=($OVERRIDE_ROUTE_IDS) # split the `OVERRIDE_ROUTE_IDS` string into an array by the same name
GENERATED_ROUTE_IDS=($GENERATED_ROUTE_IDS) # split the `GENERATED_ROUTE_IDS` string into an array by the same name
IFS=$SAVEIFS # Restore original IFS
for (( i=0; i<${#OVERRIDE_ROUTE_IDS[#]}; i++ ))
do
if [[ "${GENERATED_ROUTE_IDS[*]}" =~ "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "route ID ${OVERRIDE_ROUTE_IDS[$i]} exists in generated routes"
for (( j=0; j<${#GENERATED_ROUTE_IDS[#]}; j++ ))
do
if [[ "${GENERATED_ROUTE_IDS[$j]}" == "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "index of route ${GENERATED_ROUTE_IDS[$j]} is $j"
echo "$i"
ROUTE_TO_USE=$(yq eval ".routes.[$i]" routes.yml)
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" spring.yml)
fi
done
else
echo "no match so add to top of routes"
fi
done
My assumption is this command should update spring.yml file with the new route in place of the one that was identified with the same id:
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" application.yml)
But I am getting the following error
Error: Parsing expression: Lexer error: could not match text starting at 1:37 failing at 1:39 unmatched text: "id"
I'm stumped on this and not sure what I'm doing wrong at this point. For reference I am using yq version 4.17.2.

Answer 1

Be aware that yq does not emit a data structure, it emits a string. $ROUTE_TO_USE would be, for example,
id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
This is YAML source. Pasting this into the following yq command leads to invalid syntax; yq's expression syntax is not literal YAML. This is what the error tries to tell you.
What you want to do is to process both inputs in a single yq command:
yq ea "select(fi==0).spring.cloud.gateway.routes.[$j] = "`
`"select(fi==1).routes.[$i] | select(fi==0)" spring.yml routes.yml
ea is shorthand for eval-all which you need for processing multiple input files at the same time. fi is a shorthand for fileIndex, which is used to select the appropriate file. Piping the result to | select(fi==0) ensures that only the first (modified) file is written out. I split the long string into multiple lines using backticks for readability.

Answer 2

I ended up getting a solution from the creator of yq. The solution below is what I used:
yq ea '
(select(fi==0) | .spring.cloud.gateway.routes.[].id) as $currentIds |
(select(fi==1) | [.routes.[] | select( [$currentIds != .id] | all )] ) as $newRoutes |
( $newRoutes + .spring.cloud.gateway.routes + .routes) as $routesToMerge |
(
(($routesToMerge | .[] | {.id: .}) as $item ireduce ({}; . * $item )) as $uniqueMap
| ( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
) as $mergedArray
| select(fi == 0) | .spring.cloud.gateway.routes = $mergedArray
' spring.yml routes.yml
This matches on id. If there is a match it uses the value of what's in routes.yml. If there is no match it add it the top top of the routes.