Google workflow http.get url_encode to Cloud Run function returns 403 - google-workflows

I have a CloudRun function that accepts multiple URL inputs and this works fine via cURL from Google Shell, e.g
curl https://myapp.a.run.app/function1/input1/input2
and where input1 is an email address, input2 a string with no spaces, input3 and int and input4 a filename with spaces, e.g
curl https://myapp.a.run.app/function2/input1/input2/input3/input4
I have a Cloud worklow that calls this function in a few steps, building up a URL from previous responses. The calls to the function in steps before are working fine, however when I attempt to call the function via a URL where the email address/filename inputs are text.url_encoded prior, the workflow fails with a 403 error.
- step3_assign_input1:
assign:
- step3_URL: '${"https://myapp.a.run.app/function1/" + step2result.body.field1.field2["b:input1"]}'
- step4_get_input1:
try:
call: http.get
args:
url: ${step3_URL}
result: step4result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- step4a_sleep:
call: sys.sleep
args:
seconds: 5
- step5_assign_input2:
assign:
- step5_URL: '${"https://myapp.a.run.app/function2/" + step4result.body.field1.field2["a:input2"]}'
- step6_get_input2:
try:
call: http.get
args:
url: ${step5_URL}
result: step6result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- step7a_encode_email:
call: text.url_encode
args:
source: ${args.email}
result: encode_email
- step7b_encode_filename:
call: text.url_encode
args:
source: '${step6result.body.field1.field2["a:FileName"]}'
result: encode_filename
- step8_assign_emailURL:
assign:
- emailURL: '${"https://myapp.a.run.app/email/" + encode_email + "/" + args.type + "/" + args.serial + "/" + encode_filename}'
- step9_get_emailURL:
try:
call: http.get
args:
url: ${emailURL}
result: step9result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- returnOutput:
return: ${step9result}
custom_predicate:
params: [e]
steps:
- what_to_repeat:
switch:
- condition: ${e.code in [429, 500, 502, 503, 504]}
return: true
- otherwise:
return: false
From the workflow logs I can see the URL is being constructed as
https://myapp.a.run.app/email/someone%40email.com/String/12345678/THIS%20File%20Name%01.pdf
which if I call via cURL GET in CloudShell it returns with a 200 response code.
In the Workflow logs I get a HTTP server responded with error code 403, and Error: Forbidden\u003c/h1\u003e\n\u003ch2\u003eAccess is forbidden.
{
"insertId": "sl7uy9bd1",
"jsonPayload": {
"activityTime": "2023-01-06T05:44:39Z",
"state": "FAILED",
"#type": "type.googleapis.com/google.cloud.workflows.type.ExecutionsSystemLog",
"failure": {
"source": "main.step9_get_emailURL, line: 76",
"exception": "HTTP server responded with error code 403\nin step \"step9_get_emailURL\", routine \"main\", line: 76: {\"body\":\"\\n\\u003chtml\\u003e\\u003chead\\u003e\\n\\u003cmeta http-equiv=\\\"content-type\\\" content=\\\"text/html;charset=utf-8\\\"\\u003e\\n\\u003ctitle\\u003e403 Forbidden\\u003c/title\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody text=#000000 bgcolor=#ffffff\\u003e\\n\\u003ch1\\u003eError: Forbidden\\u003c/h1\\u003e\\n\\u003ch2\\u003eAccess is forbidden.\\u003c/h2\\u003e\\n\\u003ch2\\u003e\\u003c/h2\\u003e\\n\\u003c/body\\u003e\\u003c/html\\u003e\\n\",\"code\":403,\"headers\":{\"Alt-Svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000,h3-Q050=\\\":443\\\"; ma=2592000,h3-Q046=\\\":443\\\"; ma=2592000,h3-Q043=\\\":443\\\"; ma=2592000,quic=\\\":443\\\"; ma=2592000; v=\\\"46,43\\\"\",\"Content-Length\":\"235\",\"Content-Type\":\"text/html; charset=UTF-8\",\"Date\":\"Fri, 06 Jan 2023 05:44:38 GMT\",\"X-Appengine-Country\":\"ZZ\"},\"message\":\"HTTP server responded with error code 403\",\"tags\":[\"HttpError\"]}"
}
},
"resource": {
"type": "workflows.googleapis.com/Workflow",
"labels": {
"workflow_id": "myworkflow",
"location": "australia-southeast1",
"resource_container": "323152299552"
}
},
"timestamp": "2023-01-06T05:44:39.101297442Z",
"severity": "ERROR",
"labels": {
"workflows.googleapis.com/revision_id": "000036-15c",
"workflows.googleapis.com/execution_id": "f2175706-7e8a-4321-916b-487231a10d6b"
},
"logName": "projects/myproject/logs/workflows.googleapis.com%2Fexecutions_system",
"receiveTimestamp": "2023-01-06T05:44:40.026366626Z"
}
If I look at the logs of the CloudRun function, I see successful logs from the earlier steps in the workflow. but, no logs from the step that is failing.
Hopefully someone can provide some insight why this fails via a workflow steps.

Related

Access HTTP response data saved in a variable with colon

I'm using Google Cloud Workflow to call via http.get a CloudRun app that returns a XML document that has been converted to json, the below json gets successfully returned to Workflow in Step 2 which contains the converted XML to json in the body.
{
"body": {
"ResponseMessage": {
"#xmlns": "http://someurl.com/services",
"Response": {
"#xmlns:a": "http://someurl.com/data",
"#xmlns:i": "http://www.w3.org/2001/XMLSchema-instance",
"a:ReferenceNumber": {
"#i:nil": "true"
},
"a:DateTime": "2023-01-01T00:17:38+0000",
"a:TransactionId": "154200432",
"a:Environment": "Development",
"a:RequestDateTime": "2023-01-01T11:17:39",
}
},
"code": 200,
"headers": {
"Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
"Content-Length": "1601",
"Content-Type": "application/json",
"Date": "Sun, 01 Jan 2023 00:17:39 GMT",
"Server": "Google Frontend",
"X-Cloud-Trace-Context": "931754ab82102397eb07775171485850"
}
}
}
The full yaml of the workflow is below and without step3/step4 it works. In step3 I try to access an element in the json which is returned from step2 as per https://cloud.google.com/workflows/docs/http-requests#access-data
main:
params: [args]
steps:
- step1:
assign:
- searchURL: ${"https://myfunction.a.run.app/search/" + args.type + "/" + args.serial}
- step2:
call: http.get
args:
url: ${searchURL}
result: s2result
- step3:
assign:
- resubmitURL: '${https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId} }'
- step4:
call: http.get
args:
url: ${resubmitURL}
result: s4result
- returnOutput:
return: ${s4result}
However due to the colon : in the field I'm trying to access there are yaml parsing errors when I attempt to save another variable assignment. How can I access a HTTP response data saved in a variable when there are colons in the property field.
The errors in the console are similar too
Could not deploy workflow: main.yaml:14:25: parse error: in workflow 'main', step 'step3': token recognition error at: ':'
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId}'
^
main.yaml:14:25: parse error: in workflow 'main', step 'step3': mismatched input '+' expecting {'[', LOGICAL_NOT, '(', '-', TRUE, FALSE, NULL, NUM_FLOAT, NUM_INT, STRING, IDENTIFIER}
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId}'
Two techniques are required to reference map keys with special characters like this:
As recommended in the documentation, all expressions should be wrapped in single quotes to avoid YAML parsing errors (i.e. '${...}').
When referencing keys with special characters, you can use array notation to wrap the key name in quotes (i.e. var["KEY"]).
Together, it looks like this:
main:
steps:
- init:
assign:
- var:
key:
"co:lon": bar
- returnOutput:
return: '${"foo" + var.key["co:lon"]}'
In your code you are using an expression inside an expression:
- resubmitURL: '**${**"https://myfunction.a.run.app/resubmit/" + **${**s2result.body.ResponseMessage.Response.a:TransactionId**}**'
In this sample from your error message your not even closing the expressions right.
If you pack everything into one expression and use the hint from Kris with the key, it should deploy:
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + s2result.body.ResponseMessage.Response["a:TransactionId"]}'
Here is my full test case:
main:
params: [args]
steps:
- init_assign:
assign:
- input: ${args}
- s2result:
body:
ResponseMessage:
Response:
"a:TransactionId": "Test"
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + s2result.body.ResponseMessage.Response["a:TransactionId"]}'
- log1:
call: sys.log
args:
text: ${resubmitURL}
severity: INFO
With the log: 'https://myfunction.a.run.app/resubmit/Test'

GCP Workflow: Handling http functions responses other than 200

When calling http endpoint in GCP workflow, only HttpStatus 200 is considered a success.
How to handle other Success Status codes? 201, 202, etc.
Example workflow from samples:
- readItem:
try:
call: http.get
args:
url: https://example.com/someapi
auth:
type: OIDC
result: APIResponse
except:
as: e
steps:
- knownErrors:
switch:
- condition: ${not("HttpError" in e.tags)}
next: connectionProblem
- condition: ${e.code == 404}
next: urlNotFound
- condition: ${e.code == 403}
next: authProblem
- UnhandledException:
raise: ${e}
- urlFound:
return: ${APIResponse.body}
- connectionProblem:
return: "Connection problem; check URL"
- urlNotFound:
return: "Sorry, URL wasn't found"
- authProblem:
return: "Authentication error"
If the api endpoint https://example.com/someapi returns anything other than a 200 status code the connectionProblem is invoked.
This is the same if its a GET or POST request.
What is the best way of handling this?
There is nothing referencing how to handle other 200s statuses in the documentation for Google Workflows so I assume this is not possible without treating them as errors.
This means that in order to do it you are going to need to add an extra step to deal with this status as an error handling strategy, like - condition: ${e.code == 201} for example.
Alternatively you could open a feature request in Google's Issue Tracker so that they can consider implementing different treatements of such status codes or at least so that this is touched on more details in the documentation.
At present response codes >= 400 and <= 599 are considered an error and will raise an exception. i.e. 200s are considered a success and will not.
Alternatively, if you want to trigger an exception handler for return codes in this range (or for any other reason), this can be done by adding an additional step to the try call, for example (illustration only):
main:
steps:
- getStuff:
try:
steps:
- callStep:
call: http.get
args:
url: <SOME URL>
result: r
- checkNotOK:
switch:
- condition: ${r.code == 202}
raise: ${r}
retry:
predicate: ${custom_predicate}
max_retries: 5
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
custom_predicate:
params: [e]
steps:
- what_to_repeat:
switch:
- condition: ${e.code == 202}
return: true
- otherwise:
return: false

how do you start a workflow from another workflow and retrieve the return value of called workflow

I am testing google workflow and would like to call a workflow from another workflow but as a separate process (not a subworkflow)
I am able to start the execution but currently unable to retrieve the return value. I receive instead an instance of the execution:
{
"argument": "null",
"name": "projects/xxxxxxxxxxxx/locations/us-central1/workflows/child-workflow/executions/9fb4aa01-2585-42e7-a79f-cfb4b57b22d4",
"startTime": "2020-12-09T01:38:07.073406981Z",
"state": "ACTIVE",
"workflowRevisionId": "000003-cf3"
}
parent-workflow.yaml
main:
params: [args]
steps:
- callChild:
call: http.post
args:
url: 'https://workflowexecutions.googleapis.com/v1beta/projects/my-project/locations/us-central1/workflows/child-workflow/executions'
auth:
type: OAuth2
scope: 'https://www.googleapis.com/auth/cloud-platform'
result: callresult
- returnValue:
return: ${callresult.body}
child-workflow.yaml:
- getCurrentTime:
call: http.get
args:
url: https://us-central1-workflowsample.cloudfunctions.net/datetime
result: CurrentDateTime
- readWikipedia:
call: http.get
args:
url: https://en.wikipedia.org/w/api.php
query:
action: opensearch
search: ${CurrentDateTime.body.dayOfTheWeek}
result: WikiResult
- returnOutput:
return: ${WikiResult.body[1]}
also as an added question how can create a dynamic url from a variable. ${} doesn't seem to work there
As Executions are async API calls, you need to POLL for the workflow to see when finished.
You can have the following algorithm:
main:
steps:
- callChild:
call: http.post
args:
url: ${"https://workflowexecutions.googleapis.com/v1beta/projects/"+sys.get_env("GOOGLE_CLOUD_PROJECT_ID")+"/locations/us-central1/workflows/http_bitly_secrets/executions"}
auth:
type: OAuth2
scope: 'https://www.googleapis.com/auth/cloud-platform'
result: workflow
- waitExecution:
call: CloudWorkflowsWaitExecution
args:
execution: ${workflow.body.name}
result: workflow
- returnValue:
return: ${workflow}
CloudWorkflowsWaitExecution:
params: [execution]
steps:
- init:
assign:
- i: 0
- valid_states: ["ACTIVE","STATE_UNSPECIFIED"]
- result:
state: ACTIVE
- check_condition:
switch:
- condition: ${result.state in valid_states AND i<100}
next: iterate
next: exit_loop
- iterate:
steps:
- sleep:
call: sys.sleep
args:
seconds: 10
- process_item:
call: http.get
args:
url: ${"https://workflowexecutions.googleapis.com/v1beta/"+execution}
auth:
type: OAuth2
result: result
- assign_loop:
assign:
- i: ${i+1}
- result: ${result.body}
next: check_condition
- exit_loop:
return: ${result}
What you see here is that we have a CloudWorkflowsWaitExecution subworkflow which will loop 100 times at most, also has a 10 second delay, it will stop when the workflow has finished, and returns the result.
The output is:
argument: 'null'
endTime: '2020-12-09T13:00:11.099830035Z'
name: projects/985596417983/locations/us-central1/workflows/call_another_workflow/executions/05eeefb5-60bb-4b20-84bd-29f6338fa66b
result: '{"argument":"null","endTime":"2020-12-09T13:00:00.976951808Z","name":"projects/985596417983/locations/us-central1/workflows/http_bitly_secrets/executions/2f4b749c-4283-4c6b-b5c6-e04bbcd57230","result":"{\"archived\":false,\"created_at\":\"2020-10-17T11:12:31+0000\",\"custom_bitlinks\":[],\"deeplinks\":[],\"id\":\"j.mp/2SZaSQK\",\"link\":\"//<edited>/2SZaSQK\",\"long_url\":\"https://cloud.google.com/blog\",\"references\":{\"group\":\"https://api-ssl.bitly.com/v4/groups/Bg7eeADYBa9\"},\"tags\":[]}","startTime":"2020-12-09T13:00:00.577579042Z","state":"SUCCEEDED","workflowRevisionId":"000001-478"}'
startTime: '2020-12-09T13:00:00.353800247Z'
state: SUCCEEDED
workflowRevisionId: 000012-cb8
in the result there is a subkey that holds the results from the external Workflow execution.
The best method is now the workflows.executions.run helper method, which formats the request and blocks until the workflow execution has completed:
- run_execution:
try:
call: googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run
args:
workflow_id: ${workflow}
location: ${location} # Defaults to current location
project_id: ${project} # Defaults to current project
argument: ${arguments} # Arguments could be specified inline as a map instead.
result: r1
except:
as: e
steps: ... # handle a failed execution

How do I make that my hasura actions are ready to be used for my ci / cd tests?

I have started building up a backend with hasura. That backend is validated on my CI / CD service with api tests, among other things.
Within my hasura backend, I have implemented openfaas functions. I am deploying everything on a kubernetes cluster. Before running the tests, I wait until all jobs and all deployments are done. I am deploying with devspace which deploys everything through helm charts. So, at the end of the deployment, I am dead-sure the deployments are all done (ultimately, I've checked directly on the k8s cluster). Even the openfaas functions are deployed and ready to use.
Yet, when I run my acceptance tests, I run into issues. If I don't wait long enough, then my actions are not working properly. They return some strange errors that e.g. the response returned invalid json
Error: GraphQL error: not a valid json response from webhook
or the mutation is not in the mutation root
Error: GraphQL error: field "login" not found in type: 'mutation_root'
However, the openfaas functions themselves log only success. There is no error there. They are called and they apparently throw no error.
Waiting 3-5 minutes after hasura deployment or trying to call the actions until they return something relevant works fine, however. My current work-around is to wait an additional 5 minutes after my deployments have been done and only then run my api tests.
Is that normal? Is there a more efficient way to get feedback on when hasura really is ready to accept calls to its actions? I am currently working with version 1.2.1.
EDIT
After re-verification, waiting "long enough" does not help. What, however, helps, is calling some actions until they return successful answer. Currently, what I am doing is
#! /bin/sh
if [ "$#" -lt "3" ] ; then
echo "Usage: $0 <hasura-endpoint> <profile> <auth-app-id> [<timeout-in-sec> <deltat-in-sec>]"
exit 1
fi
ENDPOINT=$1
PROFILE=$2
AUTH_APP_ID=$3
TIMEOUT=${4:-300}
DELTA_T=${5:-5}
FIXTURES_FILE=./shared/fixtures/${PROFILE}/database/Users/auth.json
username=$(jq -r '.[1].email' $FIXTURES_FILE)
password=$(jq -r '.[1].password' $FIXTURES_FILE)
user_id=$(jq -r '.[1].id' $FIXTURES_FILE)
echo "Trying to login with $username / $password / $AUTH_APP_ID"
for iteration in `seq 1 $TIMEOUT`; do
result=$(gq $ENDPOINT -q 'mutation($username: String!, $password: String!, $appId: uuid!) { login(username: $username, password: $password, appId: $appId) { userId }}' -v "username=$username" -v "password=$password" -v "appId=$AUTH_APP_ID" | jq -r '.data.login.userId')
if [ "$result" == "$user_id" ] ; then
exit 0
else
sleep $DELTA_T
fi
done
echo "Hasura actions availability timed out" && exit 1
That performs logins with valid credentials until the action returns the right user id, and not an error. The log of this script on my ci / cd is something like
$ ./scripts/login_until_it_works.sh ${API_ENDPOINT}/v1/graphql $PROFILE $AUTH_ADMIN_APP_ID
Trying to login with nathalie.droz#test-vtxnet.ch / yl2YOuSrz_ / [MASKED]
Executing query... error
Error: ApolloError: GraphQL error: not a valid json response from webhook
at new ApolloError (/usr/local/lib/node_modules/graphqurl/node_modules/apollo-client/bundle.umd.js:92:26)
at Object.next (/usr/local/lib/node_modules/graphqurl/node_modules/apollo-client/bundle.umd.js:1297:31)
at notifySubscription (/usr/local/lib/node_modules/graphqurl/node_modules/zen-observable/lib/Observable.js:135:18)
at onNotify (/usr/local/lib/node_modules/graphqurl/node_modules/zen-observable/lib/Observable.js:179:3)
at SubscriptionObserver.next (/usr/local/lib/node_modules/graphqurl/node_modules/zen-observable/lib/Observable.js:235:7)
at /usr/local/lib/node_modules/graphqurl/node_modules/apollo-client/bundle.umd.js:1102:36
at Set.forEach (<anonymous>)
at Object.next (/usr/local/lib/node_modules/graphqurl/node_modules/apollo-client/bundle.umd.js:1101:21)
at notifySubscription (/usr/local/lib/node_modules/graphqurl/node_modules/zen-observable/lib/Observable.js:135:18)
at onNotify (/usr/local/lib/node_modules/graphqurl/node_modules/zen-observable/lib/Observable.js:179:3) {
graphQLErrors: [
{
extensions: [Object],
message: 'not a valid json response from webhook'
}
],
networkError: null,
message: 'GraphQL error: not a valid json response from webhook',
extraInfo: undefined
}
Executing query... done
Notice that the second query, 5 seconds after the first, is successful. My action is defined as follows:
- args:
enums: []
input_objects: []
objects:
- description: null
fields:
- description: null
name: token
type: String!
- description: null
name: refreshToken
type: String!
- description: null
name: userId
type: uuid!
name: LoginResponse
scalars: []
type: set_custom_types
- args:
comment: null
definition:
arguments:
- description: null
name: username
type: String!
- description: null
name: password
type: String!
- description: null
name: appId
type: uuid!
forward_client_headers: false
handler: http://gateway.openfaas:8080/function/login.{{FUNCTION_NAMESPACE}}
headers: []
kind: synchronous
output_type: LoginResponse
type: mutation
name: login
type: create_action
- args:
action: login
definition:
select:
filter: {}
role: incognito
type: create_action_permission
When you deploy via Helm, it creates the Deployments and everything else you've defined and tells you it's done. That doesn't mean that whatever you deployed is ready to serve requests. That's because each service may have its own boot time, especially the services who advertise High Availability.
Kubernetes is designed to address this issue with the help of "liveness/readiness probes". Basically, in your yaml/helm files you instruct K8s what it needs to check before it returns that a pod is ready. This could be for example a 200 HTTP status code from /live endpoint in your app or whatever.
Check this out: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Dredd passing trailing square bracket to API

I'm using Dredd to test an API I have written. It works fine until I try to vary the action uri within a resource. When I have an action of the form
## Retrieve Task [GET /task/{id}]
it sends a request to Drakov with the ] appended. This Drakov server is running the blueprint document.
Drakov 0.1.16 Listening on port 8090
[LOG] GET /task/myid]
[LOG] DELETE /task/myid]
[LOG] GET /task/myid]
You can see this request has an extra ] on the end.
This is my blueprint. It is a subset of the example from the Api Blueprint examples:
FORMAT: 1A
# Advanced Action API
A resource action is – in fact – a state transition. This API example demonstrates an action - state transition - to another resource.
## API Blueprint
# Tasks [/tasks/tasks{?status,priority}]
+ Parameters
+ status `test` (string)
+ priority `1` (number)
## Retrieve Task [GET /task/{id}]
This is a state transition to another resource
+ Parameters
+ id: `myid` (string)
+ Response 200 (application/json)
{
"id": 123,
"name": "Go to gym",
"done": false,
"type": "task"
}
What am I doing wrong?
Your API Blueprint has multiple errors. For instance,
+ Parameters
+ status `test` (string)
+ priority `1` (number)
...should be:
+ Parameters
+ status: `test` (string)
+ priority: `1` (number)
Also, you are defining a resource Tasks with URI Template /tasks/tasks{?status,priority} and then you are trying to define a GET action for the resource with a different URI Template. That is confusing.
I tried to create a sample API Blueprint (saved as sample-blueprint.md) like this:
FORMAT: 1A
# My API
## Task [/task/{id}]
### Retrieve Task [GET]
+ Parameters
+ id: `123` (string)
+ Response 200 (application/json)
{
"id": 123,
"name": "Go to gym",
"done": false,
"type": "task"
}
Then I launched a Drakov server in one terminal like this:
drakov -f *.md
Then I tried to run Dredd:
dredd sample-blueprint.md http://localhost:3000
Everything passed correctly:
$ dredd sample-blueprint.md http://localhost:3000
info: Beginning Dredd testing...
pass: GET /task/123 duration: 42ms
complete: 1 passing, 0 failing, 0 errors, 0 skipped, 1 total
complete: Tests took 50ms
Is this something you originally wanted to achieve?

Resources