How can i form the property in compose to return int(0) if condition is true and not return anything if condition is false? - expression

How can i form this expression to return int value of 0 if true and don't return the property if false? warehouse event is an array and the property is inside a compose.
Expression:
if(contains(variables('WareHouseEvent'), 'OB_2910'), int(0), <not
return anything)

An alternative to the first answer is to always add and then remove it after the fact.
This is an example you can copy into your own tenant for testing.
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose_JSON_Object": {
"inputs": {
"AnotherProperty": "Another Value",
"TestProperty": "#variables('Data')"
},
"runAfter": {
"Initialize_Integer": [
"Succeeded"
]
},
"type": "Compose"
},
"Initialize_Integer": {
"inputs": {
"variables": [
{
"name": "Data",
"type": "integer",
"value": 0
}
]
},
"runAfter": {},
"type": "InitializeVariable"
},
"New_JSON_Object": {
"inputs": {
"variables": [
{
"name": "New Object",
"type": "object",
"value": "#if(equals(variables('Data'), 1), removeProperty(outputs('Compose_JSON_Object'), 'TestProperty'), outputs('Compose_JSON_Object'))"
}
]
},
"runAfter": {
"Compose_JSON_Object": [
"Succeeded"
]
},
"type": "InitializeVariable"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"conditions": [],
"inputs": {},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
I have an integer variable at the top which stores a value of either 1 or 0.
Then in my compose, I add that value to a property in the compose statement.
Then beneath that, I set a new variable with the (potentially) updated object using an expression to determine if the added property should be removed or not.
You'd just need to adjust the condition portion of the IF statement with your expression.
if(equals(variables('Data'), 1), removeProperty(outputs('Compose_JSON_Object'), 'TestProperty'), outputs('Compose_JSON_Object'))
The property will be removed depending on the value of the Data variable.
Removed
Retained

One of the workarounds is that you can use Condition connector when if the mentioned condition satisfies it executes true block else it executes false. From there you can use the same Compose content.
Here is the screenshot of the logic app -
output :-

Related

How to ensure a Step Function executes Parameterized Query properly in AWS?

I'm currently trying to execute an Athena Query during a State Machine. The query itself needs a date variable to use in several WHERE statements so I'm using a Lambda to generate it.
When I run EXECUTE prepared-statement USING 'date', 'date', 'date'; directly in Athena, I get the results I expect so I know the query is formed correctly, but when I try to do it in the state machine, it gives me the following error:
SYNTAX_ERROR: line 19:37: Unexpected parameters (integer) for function date. Expected: date(varchar(x)) , date(timestamp) , date(timestamp with time zone)
So my best guess is that I'm somehow not passing the execution parameters correctly.
The Lambda that calculates the date returns it in a string with the format %Y-%m-%d, and in the State Machine I make sure to pass it to the output of every State that needs it. Then I get a named query to create a prepare statement from within the state machine. I then use that prepared statement to run an EXECUTE query that requires the date multiple times, so I use an intrinsic function to turn it into an array:
{
"StartAt": "calculate_date",
"States": {
"calculate_date": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload.$": "$",
"FunctionName": "arn:aws:lambda:::function:calculate_date:$LATEST"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "get_query",
"ResultSelector": {
"ExecDate.$": "$.Payload.body.exec_date"
}
},
"get_query": {
"Type": "Task",
"Next": "prepare_query",
"Parameters": {
"NamedQueryId": "abc123"
},
"Resource": "arn:aws:states:::aws-sdk:athena:getNamedQuery",
"ResultPath": "$.Payload"
},
"prepare_query": {
"Type": "Task",
"Next": "execute_query",
"Parameters": {
"QueryStatement.$": "$.Payload.NamedQuery.QueryString",
"StatementName": "PreparedStatementName",
"WorkGroup": "athena-workgroup"
},
"Resource": "arn:aws:states:::aws-sdk:athena:createPreparedStatement",
"ResultPath": "$.Payload"
},
"execute_query": {
"Type": "Task",
"Resource": "arn:aws:states:::athena:startQueryExecution",
"Parameters": {
"ExecutionParameters.$": "States.Array($.ExecDate, $.ExecDate, $.ExecDate)",
"QueryExecutionContext": {
"Catalog": "catalog_name",
"Database": "database_name"
},
"QueryString": "EXECUTE PreparedStatementName",
"WorkGroup": "athena-workgroup",
"ResultConfiguration": {
"OutputLocation": "s3://bucket"
}
},
"End": true
}
}
}
The execution of the State Machine returns successfully, but the query doesn't export the results to the bucket, and when I click on the "Athena query execution" link in the list of events, it takes me to the Athena editor page where I see the error listed above
https://i.stack.imgur.com/pxxOm.png
Am I generating the ExecutionParameters wrong? Does the createPreparedStatement resource need a different syntax for the query parameters? I'm truly at a lost here, so any help is greatly appreciated
I just solved my problem. And I'm posting this answer in case anyone comes across the same issue.
Apparently, the ExecutionParameters paremeter in an Athena StartQueryExecution state does not respect the variable type of a JSONPath variable, so you need to manually add the single quotes when forming your array. I solved this by adding a secondary output from the lambda, with the date wrapped in single quotes so that when I make the array using instrinsic functions and pass it to the query execution, it forms the query string correctly.
Transform the output from the lambda like this:
"ExecDateQuery.$": "States.Format('\\'{}\\'', $.Payload.body.exec_date)"
And use ExecDateQuery in the array intrinsic function, instead of ExecDate.
The final State Machine would look like this:
{
"StartAt": "calculate_date",
"States": {
"calculate_date": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload.$": "$",
"FunctionName": "arn:aws:lambda:::function:calculate_date:$LATEST"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "get_query",
"ResultSelector": {
"ExecDate.$": "$.Payload.body.exec_date",
"ExecDateQuery.$": "States.Format('\\'{}\\'', $.Payload.body.exec_date)"
}
},
"get_query": {
"Type": "Task",
"Next": "prepare_query",
"Parameters": {
"NamedQueryId": "abc123"
},
"Resource": "arn:aws:states:::aws-sdk:athena:getNamedQuery",
"ResultPath": "$.Payload"
},
"prepare_query": {
"Type": "Task",
"Next": "execute_query",
"Parameters": {
"QueryStatement.$": "$.Payload.NamedQuery.QueryString",
"StatementName": "PreparedStatementName",
"WorkGroup": "athena-workgroup"
},
"Resource": "arn:aws:states:::aws-sdk:athena:createPreparedStatement",
"ResultPath": "$.Payload"
},
"execute_query": {
"Type": "Task",
"Resource": "arn:aws:states:::athena:startQueryExecution",
"Parameters": {
"ExecutionParameters.$": "States.Array($.ExecDateQuery, $.ExecDateQuery, $.ExecDateQuery)",
"QueryExecutionContext": {
"Catalog": "catalog_name",
"Database": "database_name"
},
"QueryString": "EXECUTE PreparedStatementName",
"WorkGroup": "athena-workgroup",
"ResultConfiguration": {
"OutputLocation": "s3://bucket"
}
},
"End": true
}
}
}

Handling events in interactive Vega legends - race condtion

The linked chart contains only a legend and the legend works as follows:
clicking on a fruit name toggles it on and off
shift-clicking on a fruit name switches it ON and switches OFF all other fruit names
Legend display is controlled by two entities:
data set SELECTED remembers selected items
signal FILTERMODE toggles the type of the filter between include and exclude
Currently, if only one fruit name is ON, then a click on it switches it OFF (so all fruit names become OFF).
I would like to modify this behavior so that a click on the last enabled fruit name would switch everything ON.
(In other words - it would not be possible to deselect everything.)
In order to switch everything ON I only need to change the value of signal FILTERMODE to exclude. This is where I hit a snag.
I have tried the following in the signal definition:
"update": "event.shiftKey? 'include' : (length(data('selected'))? filtermode : 'exclude')",
This does not work. I am fairly sure this happens because of a race condition.
When I check for the length of data('source'), it is still non-empty.
So the sequence of events is the following:
click
update signal FILTERMODE (check if the data set SELECTED is empty - it is not)
update data set SELECTED (only now it has become empty)
What would be the most elegant work-around?
Try this instead. It is the same as your code but also checks the length of the array which your single line doesn't currently do.
You can now shift click melon and then click it normally and the filter mode will switch.
Editor
{
"$schema": "https://vega.github.io/schema/vega/v5.json",
"description": "A scatter plot example with interactive legend and x-axis.",
"width": 200,
"height": 200,
"padding": 5,
"autosize": "pad",
"signals": [
{
"name": "shift",
"value": false,
"on": [
{
"events": "#legendSymbol:click, #legendLabel:click",
"update": "event.shiftKey",
"force": true
}
]
},
{
"name": "clicked",
"value": null,
"on": [
{
"events": "#legendSymbol:click, #legendLabel:click",
"update": "{value: datum.value}",
"force": true
}
]
},
{
"name": "filtermode",
"value": "exclude",
"on": [
{
"events": "#legendSymbol:click, #legendLabel:click",
"update": "event.shiftKey? 'include' : (length(data('selected') == 0)? filtermode : 'exclude')",
"force": true
}
]
}
],
"data": [
{
"name": "source",
"values": [
{"fruit": "apple"},
{"fruit": "plum"},
{"fruit": "pear"},
{"fruit": "melon"},
{"fruit": "grape"},
{"fruit": "strawberry"}
]
},
{
"name": "selected",
"on": [
{"trigger": "clicked && (event.shiftKey)", "remove": true},
{"trigger": "clicked && (event.shiftKey)", "insert": "clicked"},
{"trigger": "clicked && (!event.shiftKey)", "toggle": "clicked"}
]
}
],
"scales": [
{
"name": "color",
"type": "ordinal",
"range": {"scheme": "category10"},
"domain": {"data": "source", "field": "fruit"}
}
],
"legends": [
{
"stroke": "color",
"title": "Fruit",
"encode": {
"symbols": {
"name": "legendSymbol",
"interactive": true,
"update": {
"fill": {"value": "transparent"},
"strokeWidth": {"value": 2},
"opacity": [
{
"test": "filtermode == 'exclude' && !indata('selected', 'value', datum.value)",
"value": 1
},
{
"test": "filtermode == 'include' && indata('selected', 'value', datum.value)",
"value": 1
},
{"value": 0.15}
],
"size": {"value": 64}
}
},
"labels": {
"name": "legendLabel",
"interactive": true,
"update": {
"opacity": [
{
"test": "filtermode == 'exclude' && !indata('selected', 'value', datum.value)",
"value": 1
},
{
"test": "filtermode == 'include' && indata('selected', 'value', datum.value)",
"value": 1
},
{"value": 0.25}
]
}
}
}
}
]
}
Are you checking the length of the correct array? It is hard to understand precisely what the desired behaviour is but if I add the code (depending on whether filter mode is include or exclude)
length(data('selected')) == 6
or
length(data('selected')) == 0
then it seems to work.
Editor

Pass Step Function variable to AWS Glue Job Not Working

I'm trying to pass an AWS Step Function variable to a Glue Job parameter, similar to this:
aws-passing-job-parameters-value-to-glue-job-from-step-function
However, this is not working for me. The glue job error message indicates that it's getting the passed variable name--not the actual value of the variable. Here's my Step Function code:
{
"Comment": "Converts CSV files to parquet for a date range.",
"StartAt": "ConfigureCount",
"States": {
"ConfigureCount": {
"Type": "Pass",
"Result": {
"start": 201601,
"end": 201602,
"index": 201601
},
"ResultPath": "$.iterator",
"Next": "Iterator"
},
"Iterator": {
"Type": "Task",
"Resource": "arn:aws:lambda:eu-west-1:123456789:function:date-iterator",
"ResultPath": "$.iterator",
"Next": "IsCountReached"
},
"IsCountReached": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.iterator.continue",
"BooleanEquals": true,
"Next": "ConvertToParquet"
}
],
"OutputPath": "$.iterator",
"Default": "Done"
},
"ConvertToParquet": {
"Comment": "Your application logic, to run a specific number of times",
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "convert-to-parquet",
"Arguments": {
"--DATE_RANGE": "$.iterator.index"
}
},
"ResultPath": "$.iterator.index",
"Next": "Iterator"
},
"Done": {
"Type": "Pass",
"End": true
}
}
}
The step "Iterator"step is calling a Lambda called "date-iterator" which returns JSON similar to the following:
{
"start": "201601",
"end": "201602",
"index": "201601"
}
This was based on this article, so that I can loop through values: Iterating a Loop Using Lambda
My Step Function fails, saying "$.iterator.index" is not a valid date.
How do I pass this value, and not the variable name?
from Amazon States Language (https://states-language.net/spec.html):
If any field within the Payload Template (however deeply nested) has a name ending with the characters ".$", its value is transformed according to rules below and the field is renamed to strip the ".$" suffix.
Based on that adding .$ should solve your issue:
"Parameters": {
"JobName": "convert-to-parquet",
"Arguments": {
"--DATE_RANGE.$": "$.iterator.index"
}
},

Step Functions ignoring Parameters, executing it with default input

I don't know if this is a bug of my own or Amazon's as I'm unable to properly understand what's going on.
I have the following Step Function (It's currently a work in progress), which takes care of validating an input and then sending it to an SQS lambda (CreateInAuth0 Step).
My problem is that apparently the SQS function is executing with the default parameters, fails, retries and then uses the ones specified in the Step Function. (Or at least that's what I'm gathering from the logs).
I want to know why it is ignoring the Parameters in the first go. Here's all the information I can gather.
There seems to be a difference between TaskStateEntered and TaskScheduled (which I believe it's ok).
{
"Comment": "Add the students to the platform and optionally adds them to their respective grades",
"StartAt": "AddStudents",
"States": {
"AddStudents": {
"Comment": "Validates that the students are OK",
"Type": "Task",
"Resource": "#AddStudents",
"Next": "Parallel"
},
"Parallel": {
"Type": "Parallel",
"Next": "FinishExecution",
"Branches": [
{
"StartAt": "CreateInAuth0",
"States": {
"CreateInAuth0": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "#SQS_StudentsRegistrationSQSFifo",
"MessageBody": {
"execId.$": "$.result.execId",
"tenantType.$": "$.result.tenantType",
"tenantId.$": "$.result.tenantId",
"studentsToCreate.$": "$.result.newStudents.success",
"TaskToken.$": "$$.Task.Token",
"studentsSucceeded": []
}
},
"End": true
}
}
}
]
},
"FinishExecution": {
"Comment": "This signals the ending in DynamoDB that the transaction finished",
"Type": "Task",
"End": true,
"Resource": "arn:aws:states:::dynamodb:putItem",
"ResultPath": "$.dynamoDB",
"Parameters": {
"TableName": "#SchonDB",
"Item": {
"pk": {
"S.$": "$.arguments.tenantId:exec:$.id"
},
"sk": {
"S": "result"
},
"status": {
"S": "FINISHED"
},
"type": {
"S": "#EXEC_TYPE"
}
}
},
"Retry": [
{
"ErrorEquals": [
" States.Timeout"
],
"IntervalSeconds": 1,
"MaxAttempts": 5,
"BackoffRate": 2.0
}
],
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "FinishExecutionIfFinishErrored"
}
]
},
"FinishExecutionIfFinishErrored": {
"Type": "Pass",
"End": true
}
}
}
Here's the visual:
Here's the execution event history:
Note: I have a try/catch statement inside the SQS function that wraps the entire SQS' execution.

Getting started with Step Functions and I can't get Choice to work correctly

I'm doctoring up my first step function, and as a newb into this I am struggling to make this work right. The documentation on AWS is helpful but lacks examples of what I am trying understand. I found a couple similar issues on the site here, but they didn't really answer my question either.
I have a test Step Function that works really simply. I have a small Lambda function that kicks out a single line JSON with a "Count" from a request in a DynamoDB:
def lambda_handler(event, context):
"""lambda_handler
Keyword arguments:
event -- dict -- A dict of parameters to be validated.
context --
Return:
json object with the hubID from DynamoDB of the new hub.
Exceptions:
None
"""
# Prep the Boto3 resources needed
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('TransitHubs')
# By default we assume there are no new hubs
newhub = { 'Count' : 0 }
# Query the DynamoDB to see if the name exists or not:
response = table.query(
IndexName='Status-index',
KeyConditionExpression=Key('Status').eq("NEW"),
Limit=1
)
if response['Count']:
newhub['Count'] = response['Count']
return json.dumps(newhub)
A normal output would be:
{ "Count": 1 }
And then I create this Step Function:
{
"StartAt": "Task",
"States": {
"Task": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:OMGSUPERSECRET:function:LaunchNode-get_new_hubs",
"TimeoutSeconds": 60,
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.Count",
"NumericEquals": 0,
"Next": "Failed"
},
{
"Variable": "$.Count",
"NumericEquals": 1,
"Next": "Succeed"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Failed": {
"Type": "Fail"
}
}
}
So I kick off the State Function and I get this output:
TaskStateExited
{
"name": "Task",
"output": {
"Count": 1
}
}
ChoiceStateEntered
{
"name": "Choice",
"input": {
"Count": 1
}
}
ExecutionFailed
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Choice' (entered at the event id #7). Invalid path '$.Count': The choice state's condition path references an invalid value."
}
So my question: I don't get why this is failing with that error message. Shouldn't Choice just pick up that value from the JSON? Isn't the default "$" input the "input" path?
I have figured out what the issue is, and here it is:
In my Python code, I was attempting to json.dumps(newhub) for the response thinking that what I needed was a string output representing the json formatted response. But it appears that is incorrect. When I alter the code to be simply "return newhub" and return the DICT, the step-functions process accepts that correctly. I'm assuming it parses the DICT to JSON for me? But the output difference is clearly obvious:
old Task output from above returning json.dumps(newhub):
{
"name": "Task",
"output": {
"Count": 1
}
}
new Task output from above returning newhub:
{
"Count": 1
}
And the Choice now correctly matches the Count variable in my output.
In case this is helpful for someone else. I also experienced the kind of error you did ( you had the below... just copy-pasting yours... )
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Choice' (entered at the event id #7). Invalid path '$.Count': The choice state's condition path references an invalid value."
}
But my problem turned out when I was missing the "Count" key all together.
But I did not want verbose payloads.
But per reading these docs I discovered I can also do...
"Choice": {
"Type": "Choice",
"Choices": [
{
"And": [
{
"Variable": "$.Count",
"IsPresent": true
},
{
"Variable": "$.Count",
"NumericEquals": 0,
}
],
"Next": "Failed"
},
{
"And": [
{
"Variable": "$.Count",
"IsPresent": true
},
{
"Variable": "$.Count",
"NumericEquals": 1,
}
],
"Next": "Succeed"
}
]
},
even I was facing the same issue. Give the ResultPath as InputPath and give the choice value as .value name
{
"Comment": "Step function to execute lengthy synchronous requests",
"StartAt": "StartProcessing",
"States": {
"StartProcessing": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:703569030910:function:shravanthDemo-shravanthGetItems-GHF7ZA1p6auQ",
"InputPath": "$.lambda",
"ResultPath": "$.lambda",
"Next": "VerifyProcessor"
},
"VerifyProcessor": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.lambda.cursor",
"IsNull": false,
"Next": "StartProcessing"
},
{
"Variable": "$.lambda.cursor",
"IsNull": true,
"Next": "EndOfUpdate"
}
]
},
"EndOfUpdate": {
"Type": "Pass",
"Result": "ProcessingComplete",
"End": true
}
}
}

Resources