Access HTTP response data saved in a variable with colon - google-workflows

I'm using Google Cloud Workflow to call via http.get a CloudRun app that returns a XML document that has been converted to json, the below json gets successfully returned to Workflow in Step 2 which contains the converted XML to json in the body.
{
"body": {
"ResponseMessage": {
"#xmlns": "http://someurl.com/services",
"Response": {
"#xmlns:a": "http://someurl.com/data",
"#xmlns:i": "http://www.w3.org/2001/XMLSchema-instance",
"a:ReferenceNumber": {
"#i:nil": "true"
},
"a:DateTime": "2023-01-01T00:17:38+0000",
"a:TransactionId": "154200432",
"a:Environment": "Development",
"a:RequestDateTime": "2023-01-01T11:17:39",
}
},
"code": 200,
"headers": {
"Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
"Content-Length": "1601",
"Content-Type": "application/json",
"Date": "Sun, 01 Jan 2023 00:17:39 GMT",
"Server": "Google Frontend",
"X-Cloud-Trace-Context": "931754ab82102397eb07775171485850"
}
}
}
The full yaml of the workflow is below and without step3/step4 it works. In step3 I try to access an element in the json which is returned from step2 as per https://cloud.google.com/workflows/docs/http-requests#access-data
main:
params: [args]
steps:
- step1:
assign:
- searchURL: ${"https://myfunction.a.run.app/search/" + args.type + "/" + args.serial}
- step2:
call: http.get
args:
url: ${searchURL}
result: s2result
- step3:
assign:
- resubmitURL: '${https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId} }'
- step4:
call: http.get
args:
url: ${resubmitURL}
result: s4result
- returnOutput:
return: ${s4result}
However due to the colon : in the field I'm trying to access there are yaml parsing errors when I attempt to save another variable assignment. How can I access a HTTP response data saved in a variable when there are colons in the property field.
The errors in the console are similar too
Could not deploy workflow: main.yaml:14:25: parse error: in workflow 'main', step 'step3': token recognition error at: ':'
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId}'
^
main.yaml:14:25: parse error: in workflow 'main', step 'step3': mismatched input '+' expecting {'[', LOGICAL_NOT, '(', '-', TRUE, FALSE, NULL, NUM_FLOAT, NUM_INT, STRING, IDENTIFIER}
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + ${s2result.body.ResponseMessage.Response.a:TransactionId}'

Two techniques are required to reference map keys with special characters like this:
As recommended in the documentation, all expressions should be wrapped in single quotes to avoid YAML parsing errors (i.e. '${...}').
When referencing keys with special characters, you can use array notation to wrap the key name in quotes (i.e. var["KEY"]).
Together, it looks like this:
main:
steps:
- init:
assign:
- var:
key:
"co:lon": bar
- returnOutput:
return: '${"foo" + var.key["co:lon"]}'

In your code you are using an expression inside an expression:
- resubmitURL: '**${**"https://myfunction.a.run.app/resubmit/" + **${**s2result.body.ResponseMessage.Response.a:TransactionId**}**'
In this sample from your error message your not even closing the expressions right.
If you pack everything into one expression and use the hint from Kris with the key, it should deploy:
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + s2result.body.ResponseMessage.Response["a:TransactionId"]}'
Here is my full test case:
main:
params: [args]
steps:
- init_assign:
assign:
- input: ${args}
- s2result:
body:
ResponseMessage:
Response:
"a:TransactionId": "Test"
- resubmitURL: '${"https://myfunction.a.run.app/resubmit/" + s2result.body.ResponseMessage.Response["a:TransactionId"]}'
- log1:
call: sys.log
args:
text: ${resubmitURL}
severity: INFO
With the log: 'https://myfunction.a.run.app/resubmit/Test'

Related

Google workflow http.get url_encode to Cloud Run function returns 403

I have a CloudRun function that accepts multiple URL inputs and this works fine via cURL from Google Shell, e.g
curl https://myapp.a.run.app/function1/input1/input2
and where input1 is an email address, input2 a string with no spaces, input3 and int and input4 a filename with spaces, e.g
curl https://myapp.a.run.app/function2/input1/input2/input3/input4
I have a Cloud worklow that calls this function in a few steps, building up a URL from previous responses. The calls to the function in steps before are working fine, however when I attempt to call the function via a URL where the email address/filename inputs are text.url_encoded prior, the workflow fails with a 403 error.
- step3_assign_input1:
assign:
- step3_URL: '${"https://myapp.a.run.app/function1/" + step2result.body.field1.field2["b:input1"]}'
- step4_get_input1:
try:
call: http.get
args:
url: ${step3_URL}
result: step4result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- step4a_sleep:
call: sys.sleep
args:
seconds: 5
- step5_assign_input2:
assign:
- step5_URL: '${"https://myapp.a.run.app/function2/" + step4result.body.field1.field2["a:input2"]}'
- step6_get_input2:
try:
call: http.get
args:
url: ${step5_URL}
result: step6result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- step7a_encode_email:
call: text.url_encode
args:
source: ${args.email}
result: encode_email
- step7b_encode_filename:
call: text.url_encode
args:
source: '${step6result.body.field1.field2["a:FileName"]}'
result: encode_filename
- step8_assign_emailURL:
assign:
- emailURL: '${"https://myapp.a.run.app/email/" + encode_email + "/" + args.type + "/" + args.serial + "/" + encode_filename}'
- step9_get_emailURL:
try:
call: http.get
args:
url: ${emailURL}
result: step9result
retry:
predicate: ${custom_predicate}
max_retries: 10
backoff:
initial_delay: 2
max_delay: 60
multiplier: 2
- returnOutput:
return: ${step9result}
custom_predicate:
params: [e]
steps:
- what_to_repeat:
switch:
- condition: ${e.code in [429, 500, 502, 503, 504]}
return: true
- otherwise:
return: false
From the workflow logs I can see the URL is being constructed as
https://myapp.a.run.app/email/someone%40email.com/String/12345678/THIS%20File%20Name%01.pdf
which if I call via cURL GET in CloudShell it returns with a 200 response code.
In the Workflow logs I get a HTTP server responded with error code 403, and Error: Forbidden\u003c/h1\u003e\n\u003ch2\u003eAccess is forbidden.
{
"insertId": "sl7uy9bd1",
"jsonPayload": {
"activityTime": "2023-01-06T05:44:39Z",
"state": "FAILED",
"#type": "type.googleapis.com/google.cloud.workflows.type.ExecutionsSystemLog",
"failure": {
"source": "main.step9_get_emailURL, line: 76",
"exception": "HTTP server responded with error code 403\nin step \"step9_get_emailURL\", routine \"main\", line: 76: {\"body\":\"\\n\\u003chtml\\u003e\\u003chead\\u003e\\n\\u003cmeta http-equiv=\\\"content-type\\\" content=\\\"text/html;charset=utf-8\\\"\\u003e\\n\\u003ctitle\\u003e403 Forbidden\\u003c/title\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody text=#000000 bgcolor=#ffffff\\u003e\\n\\u003ch1\\u003eError: Forbidden\\u003c/h1\\u003e\\n\\u003ch2\\u003eAccess is forbidden.\\u003c/h2\\u003e\\n\\u003ch2\\u003e\\u003c/h2\\u003e\\n\\u003c/body\\u003e\\u003c/html\\u003e\\n\",\"code\":403,\"headers\":{\"Alt-Svc\":\"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000,h3-Q050=\\\":443\\\"; ma=2592000,h3-Q046=\\\":443\\\"; ma=2592000,h3-Q043=\\\":443\\\"; ma=2592000,quic=\\\":443\\\"; ma=2592000; v=\\\"46,43\\\"\",\"Content-Length\":\"235\",\"Content-Type\":\"text/html; charset=UTF-8\",\"Date\":\"Fri, 06 Jan 2023 05:44:38 GMT\",\"X-Appengine-Country\":\"ZZ\"},\"message\":\"HTTP server responded with error code 403\",\"tags\":[\"HttpError\"]}"
}
},
"resource": {
"type": "workflows.googleapis.com/Workflow",
"labels": {
"workflow_id": "myworkflow",
"location": "australia-southeast1",
"resource_container": "323152299552"
}
},
"timestamp": "2023-01-06T05:44:39.101297442Z",
"severity": "ERROR",
"labels": {
"workflows.googleapis.com/revision_id": "000036-15c",
"workflows.googleapis.com/execution_id": "f2175706-7e8a-4321-916b-487231a10d6b"
},
"logName": "projects/myproject/logs/workflows.googleapis.com%2Fexecutions_system",
"receiveTimestamp": "2023-01-06T05:44:40.026366626Z"
}
If I look at the logs of the CloudRun function, I see successful logs from the earlier steps in the workflow. but, no logs from the step that is failing.
Hopefully someone can provide some insight why this fails via a workflow steps.

how do you start a workflow from another workflow and retrieve the return value of called workflow

I am testing google workflow and would like to call a workflow from another workflow but as a separate process (not a subworkflow)
I am able to start the execution but currently unable to retrieve the return value. I receive instead an instance of the execution:
{
"argument": "null",
"name": "projects/xxxxxxxxxxxx/locations/us-central1/workflows/child-workflow/executions/9fb4aa01-2585-42e7-a79f-cfb4b57b22d4",
"startTime": "2020-12-09T01:38:07.073406981Z",
"state": "ACTIVE",
"workflowRevisionId": "000003-cf3"
}
parent-workflow.yaml
main:
params: [args]
steps:
- callChild:
call: http.post
args:
url: 'https://workflowexecutions.googleapis.com/v1beta/projects/my-project/locations/us-central1/workflows/child-workflow/executions'
auth:
type: OAuth2
scope: 'https://www.googleapis.com/auth/cloud-platform'
result: callresult
- returnValue:
return: ${callresult.body}
child-workflow.yaml:
- getCurrentTime:
call: http.get
args:
url: https://us-central1-workflowsample.cloudfunctions.net/datetime
result: CurrentDateTime
- readWikipedia:
call: http.get
args:
url: https://en.wikipedia.org/w/api.php
query:
action: opensearch
search: ${CurrentDateTime.body.dayOfTheWeek}
result: WikiResult
- returnOutput:
return: ${WikiResult.body[1]}
also as an added question how can create a dynamic url from a variable. ${} doesn't seem to work there
As Executions are async API calls, you need to POLL for the workflow to see when finished.
You can have the following algorithm:
main:
steps:
- callChild:
call: http.post
args:
url: ${"https://workflowexecutions.googleapis.com/v1beta/projects/"+sys.get_env("GOOGLE_CLOUD_PROJECT_ID")+"/locations/us-central1/workflows/http_bitly_secrets/executions"}
auth:
type: OAuth2
scope: 'https://www.googleapis.com/auth/cloud-platform'
result: workflow
- waitExecution:
call: CloudWorkflowsWaitExecution
args:
execution: ${workflow.body.name}
result: workflow
- returnValue:
return: ${workflow}
CloudWorkflowsWaitExecution:
params: [execution]
steps:
- init:
assign:
- i: 0
- valid_states: ["ACTIVE","STATE_UNSPECIFIED"]
- result:
state: ACTIVE
- check_condition:
switch:
- condition: ${result.state in valid_states AND i<100}
next: iterate
next: exit_loop
- iterate:
steps:
- sleep:
call: sys.sleep
args:
seconds: 10
- process_item:
call: http.get
args:
url: ${"https://workflowexecutions.googleapis.com/v1beta/"+execution}
auth:
type: OAuth2
result: result
- assign_loop:
assign:
- i: ${i+1}
- result: ${result.body}
next: check_condition
- exit_loop:
return: ${result}
What you see here is that we have a CloudWorkflowsWaitExecution subworkflow which will loop 100 times at most, also has a 10 second delay, it will stop when the workflow has finished, and returns the result.
The output is:
argument: 'null'
endTime: '2020-12-09T13:00:11.099830035Z'
name: projects/985596417983/locations/us-central1/workflows/call_another_workflow/executions/05eeefb5-60bb-4b20-84bd-29f6338fa66b
result: '{"argument":"null","endTime":"2020-12-09T13:00:00.976951808Z","name":"projects/985596417983/locations/us-central1/workflows/http_bitly_secrets/executions/2f4b749c-4283-4c6b-b5c6-e04bbcd57230","result":"{\"archived\":false,\"created_at\":\"2020-10-17T11:12:31+0000\",\"custom_bitlinks\":[],\"deeplinks\":[],\"id\":\"j.mp/2SZaSQK\",\"link\":\"//<edited>/2SZaSQK\",\"long_url\":\"https://cloud.google.com/blog\",\"references\":{\"group\":\"https://api-ssl.bitly.com/v4/groups/Bg7eeADYBa9\"},\"tags\":[]}","startTime":"2020-12-09T13:00:00.577579042Z","state":"SUCCEEDED","workflowRevisionId":"000001-478"}'
startTime: '2020-12-09T13:00:00.353800247Z'
state: SUCCEEDED
workflowRevisionId: 000012-cb8
in the result there is a subkey that holds the results from the external Workflow execution.
The best method is now the workflows.executions.run helper method, which formats the request and blocks until the workflow execution has completed:
- run_execution:
try:
call: googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run
args:
workflow_id: ${workflow}
location: ${location} # Defaults to current location
project_id: ${project} # Defaults to current project
argument: ${arguments} # Arguments could be specified inline as a map instead.
result: r1
except:
as: e
steps: ... # handle a failed execution

How do I provide the correct headers for Basic Auth on a GET in OpenTest?

I'm attempting to make a GET call to a test management system that exposes an API. I want to provide Basic Auth in the header of the HTTPRequest provided as an action in OT like so.
includes: login.js
actors:
- actor: WEB
segments:
- segment: 1
actions:
- script: var xUsername = $env("X_USERNAME");
- script: var xPassword = $env("X_PASSWORD");
- script: $log("This is the username " + xUsername);
- script: $log("This is the password " + xPassword);
- description: Sample Reading testid
action: org.getopentest.actions.HttpRequest
args:
$checkpoint: true
$localData:
testRailCaseInfo: $output.body
url: https://sub.domain.io/api/v2/get_results/1234
headers:
Content-Type: application/json
Authorization: Basic xUsername xPassword
verb: GET
Is this correct?
Here are two ways to do it (please note I didn't test this code). You can either build the Authorization header value using a JavaScript expression,
- description: Read test ID
action: org.getopentest.actions.HttpRequest
args:
url: https://sub.domain.io/api/v2/get_results/1234
verb: GET
headers:
Content-Type: application/json
Authorization: |
$script
"Basic " + $base64($env("X_USERNAME") + ":" + $env("X_PASSWORD"))
or build the header value in a script action, ahead of time:
- script: |
// Build the authentication header
var authHeader =
"Basic " + $base64($env("X_USERNAME") + ":" + $env("X_PASSWORD"));
- description: Read test ID
action: org.getopentest.actions.HttpRequest
args:
url: https://sub.domain.io/api/v2/get_results/1234
verb: GET
headers:
Content-Type: application/json
Authorization: $script authHeader
I should probably explain what is the role or $script prefix in the two examples. When the value of an action argument starts with a dollar-prefixed symbol (like $json, $data, $format, etc.), the test actor understands that the expression is JavaScript code, evaluates the expression and uses the result as the value for the argument. When a JS expression doesn't start with a dollar-prefixed symbol (e.g. our expressions start with "Basic" and authHeader, respectively) we must prefix the expression with $script followed by one or more whitespace characters, to let the test actor know that the string that follows is JavaScript code and not just an ordinary string literal.
As for the format of the basic authentication scheme, you can find more information here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization.

Swagger yaml and json file for REST application developed using Play framework in Scala

I am trying to configure swagger for my application. Being new to this field I went to different tutorials and tried to convert the below json to YAML but it's giving errors like bad indentation, response missing etc. The main problem I am facing is in recognizing syntax to represent array of list in YAML format, then to add block in YAML which shows expected values for a particular block.
JSON Format to be converted to YAML:
{
"abc":[
{
"xyz":[ //array of list
{
"id":"",
"name":"",
"relation":[ //array of list
{
"first":{
"xxx":"",
"xxx":"",
"xxx":[ //array of string
""
]
},
"second":{
"xxx":"",
"xxx":"",
"xxx":[
""
],
"type":""
}
},
{
"first":{
"xxx":"",
"xxx":"",
"xxx":[ //array of string
""
]
},
"second":{
"xxx":"",
"xxx":"",
"xxx":[
""
],
"type":""
}
}
],
"rows":[
]
}
]
}
YAML is as below:
swagger: "2.0"
info:
version: 1.0.0
title: xxxx
description: xxxx
schemes:
- https
host: xxxx
basePath: xxxx
paths:
/xxx:
post:
summary: xxxx
consumes:
- application/json
produces:
- application/json
parameters:
abc:
- xyz:
id: string
name: string
relation: string
- first:
id: string
name: string
relation: string
second:
id: string
name: string
relation: string
- first:
id: string
name: string
relation: string
second:
id: string
name: string
relation: string
responses:
'200':
description: Created
When working with YAML you talk about sequences (besides map an scalar). A sequences is what gets mapped to a list in Python and an array in some other languages.
So if you are talking about "represent array of list in YAML" you are actually referring to a sequence of sequences. There are three ways to represent this in YAML.
block-style within block-style:
- - a
- b
- - c
- d
, flow-style within block-style:
- [a, b]
- [c, d]
, or flow-style within flow-style:
[[a, b,], [c, d,],]
Any online YAML parser will show you that the above amounts to the same.
Please note:
You cannot have block-style within flow-style
You can have trailing commas (something that JSON doesn't allow, and which makes JSON unnecessarily hard to edit for humans).
In your example YAML output, which is correct YAML, there are no sequence of sequences (or array of lists in your terminology).

Dredd passing trailing square bracket to API

I'm using Dredd to test an API I have written. It works fine until I try to vary the action uri within a resource. When I have an action of the form
## Retrieve Task [GET /task/{id}]
it sends a request to Drakov with the ] appended. This Drakov server is running the blueprint document.
Drakov 0.1.16 Listening on port 8090
[LOG] GET /task/myid]
[LOG] DELETE /task/myid]
[LOG] GET /task/myid]
You can see this request has an extra ] on the end.
This is my blueprint. It is a subset of the example from the Api Blueprint examples:
FORMAT: 1A
# Advanced Action API
A resource action is – in fact – a state transition. This API example demonstrates an action - state transition - to another resource.
## API Blueprint
# Tasks [/tasks/tasks{?status,priority}]
+ Parameters
+ status `test` (string)
+ priority `1` (number)
## Retrieve Task [GET /task/{id}]
This is a state transition to another resource
+ Parameters
+ id: `myid` (string)
+ Response 200 (application/json)
{
"id": 123,
"name": "Go to gym",
"done": false,
"type": "task"
}
What am I doing wrong?
Your API Blueprint has multiple errors. For instance,
+ Parameters
+ status `test` (string)
+ priority `1` (number)
...should be:
+ Parameters
+ status: `test` (string)
+ priority: `1` (number)
Also, you are defining a resource Tasks with URI Template /tasks/tasks{?status,priority} and then you are trying to define a GET action for the resource with a different URI Template. That is confusing.
I tried to create a sample API Blueprint (saved as sample-blueprint.md) like this:
FORMAT: 1A
# My API
## Task [/task/{id}]
### Retrieve Task [GET]
+ Parameters
+ id: `123` (string)
+ Response 200 (application/json)
{
"id": 123,
"name": "Go to gym",
"done": false,
"type": "task"
}
Then I launched a Drakov server in one terminal like this:
drakov -f *.md
Then I tried to run Dredd:
dredd sample-blueprint.md http://localhost:3000
Everything passed correctly:
$ dredd sample-blueprint.md http://localhost:3000
info: Beginning Dredd testing...
pass: GET /task/123 duration: 42ms
complete: 1 passing, 0 failing, 0 errors, 0 skipped, 1 total
complete: Tests took 50ms
Is this something you originally wanted to achieve?

Resources