Is it possible to use the name of an Amazon Data Pipeline as a variable inside the Data Pipeline itself? If yes, how can you do that?
Unfortunately, you can't refer to the name. You can refer to the pipeline ID using the expression #pipelineId. For example, you could define a ShellCommandActivity to print the pipeline ID:
{
"id": "ExampleActivity",
"name": "ExampleActivity",
"runsOn": { "ref": "SomeEc2Resource" },
"type": "ShellCommandActivity",
"command": "echo \"Hello from #{#pipelineId}\""
}
Related
I want my output to be as below:-
{
"EVENT": "start",
"NAME": "schedule-rule-start",
"ECS_CLUSTER": "gps-aws-infra-ecs-cluster",
"ECS_SERVICE_NAME": [
“abc”,
"def"
],
"DESIRED_COUNT": "1",
"REGION": “eu-central-1"
}
My Input in yaml is as below:
Input: !Sub '{"EVENT": "${self:custom.input.StartECSServiceRule.Event}", "NAME": "${self:custom.input.StartECSServiceRule.Name}", "ECS_CLUSTER": "${self:custom.input.StartECSServiceRule.ClusterName}", "ECS_SERVICE_NAME": "${self:custom.input.StartECSServiceRule.ECSServiceName}","DESIRED_COUNT": "${self:custom.input.StartECSServiceRule.DesiredCount}","REGION": "${self:provider.region}"}'
I want the generated cloudformation json as below:-
“{\“EVENT\“:\“start\“,\“NAME\“:\“schedule-rule-start\“,\“ECS_CLUSTER\“:\“gps-aws-infra-ecs-cluster\“,\“ECS_SERVICE_NAME\“:["abc","def"],\“DESIRED_COUNT\“:\“1\“,\“REGION\“:\“eu-central-1\“}”
Need help on how should i provide the value of "ECSServiceName" as below doesnot work in the Input I gave in the yaml file???
"${self:custom.input.StartECSServiceRule.ECSServiceName}
Please help to correct my yaml file Input. Right now it comes as string , i want it to come as array.
I have a terraform file that I'm reusing to create several AWS Eventbridge (as triggers for some lambdas).
On a different part of the file I'm able to use For Each method to create several eventbridge and naming them accordingly. My problem is that I'm not being able to do the same thing inside EOF tag (that need to be different on each Eventbridge) since it takes everything as a string.
I would need to replace the ARN in "prefix": "arn:aws:medialive:us-west-2:11111111111:channel:3434343" with a variable. How could I do that?
This is the EOF part of the terraform code:
event_pattern = <<EOF
{
"source": ["aws.medialive"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["medialive.amazonaws.com"],
"eventName": ["StopChannel"],
"responseElements": {
"arn": [{
"prefix": "arn:aws:medialive:us-west-2:11111111111:channel:3434343"
}]
}
}
}
EOF
}
It's called a Heredoc String, not an EOF tag. "EOF" just happens to be the string you are using to tag the beginning and ending of a multi-line string. You could use anything there that doesn't occurr in your actual multiline string. You could be replacing "EOF" with "MYMULTILINESTRING".
To place the value of a variable in a Heredoc String in Terraform, you do the exact same thing you would do with other strings in Terraform: You use String Interpolation.
event_pattern = <<EOF
{
"source": ["aws.medialive"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["medialive.amazonaws.com"],
"eventName": ["StopChannel"],
"responseElements": {
"arn": [{
"prefix": "${var.my_arn_variable}"
}]
}
}
}
EOF
}
I have this step function that can sometimes fail and I'd like to record this in a (dynamo) DB. What would be handy is if I could just create a new error handling step and that guy would just pick up the "execution name" from somewhere (didn't find it in the context) and record this as a failure.
Is that possible?
AWS Step Functions released recently a feature called context object.
Using $$ notation inside the Parameters block you can access information regarding your execution, including execution name, arn, state machine name, arn and others.
https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html
You can create a state to extract the context details that are then accessible to all the other states, such as:
{
"StartAt": "ExtractContextDetails",
"States": {
"ExtractContextDetails": {
"Parameters": {
"arn.$": "$$.Execution.Id"
},
"Type": "Pass",
"ResultPath": "$.contextDetails",
"Next": "NextStateName"
}
}
....
}
Yes, it can, but it is not as straight-forward as you might hope.
You are right to expect that a Lambda should be able to get the name of the calling state machine. Lambdas are passed in a context object that returns information on the caller. However, that object is null when a state machine calls your Lambda. This means two things. You will have to work harder to get what you need, and that this might be implemented in the future.
Right now, the only way I know of achieving this is by starting the execution of the state machine from within another Lambda and passing in the name in the input Json. Here is my code in Java...
String executionName = //generate a unique name...
StartExecutionRequest startExecutionRequest = new StartExecutionRequest()
.withStateMachineArn(stateMachineArn)
.withInput(" {"executionName" : executionName} ") //make sure you escape the quotes
.withName(executionName);
StartExecutionResult startExecutionResult = sf.startExecution(startExecutionRequest);
String executionArn = startExecutionResult.getExecutionArn();
If you do this, you will now have the name of your execution in the input JSON of your first step. If you want to use it in other steps, you should pass it around.
You might also want the ARN of the the execution so you can call state machine methods from within your activities or tasks. You can construct the ARN yourself by using the executionName...
arn:aws:states:us-east-1:acountid:execution:statemachinename:executionName
No. Unless you pass that information in the event, Lambda doesn't know whether or not it's part of a step function. Step functions orchestrate lambdas and maintain state between lambdas.
"States": {
"Get Alter Query": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"FunctionName": "arn:aws:lambda:ap-northeast-2:1111111:function:test-stepfuction:$LATEST",
"Payload": {
"body.$": "$",
"context.$": "$$"
}
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException",
"Lambda.TooManyRequestsException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "Alter S3 Location"
}
}
I solved it by adding context to the payload.
I highly recommend when using step functions to specify some sort of key in the step function configuration. For my step functions I always provide:
"ResultPath": "$",
"Parameters": {
"source": "StepFunction",
"type": "LAMBDA_METHOD_SWITCH_VALUE",
"payload.$": "$"
},
And have each call to lambda use the type field to determine what code to call. When your code fails, wrap it in a try/catch and explicitly use the passed in type which can be the name of the step to determine what to do next.
I'm using the gcloud describe command to get metadata information about instances.What's the best way to filter the json response with jq to get the name of the instance - if it contains "kafka" as a key.
.name + " " + .metadata.items[]?.key | select(contains("kafka"))'
Basically if items contains kafka print name.This is just a small excerpt from the json file.
"metadata": {
"fingerprint": "xxxxx=",
"items": [
{
"key": "kafka",
"value": "xxx="
},
{
"key": "some_key",
"value": "vars"
}
],
"kind": "compute#metadata"
},
"name": "instance-name",
"networkInterfaces": [
{
"accessConfigs": [
{
"kind": "compute#accessConfig",
"name": "External NAT",
"natIP": "ip",
"type": "ONE_TO_ONE_NAT"
}
],
"kind": "compute#networkInterface",
"name": "",
"network": xxxxx
}
],
I'm sure this is possible with jq, but in general working with gcloud lists is going to be easier using the built-in formatting and filtering:
$ gcloud compute instances list \
--filter 'metadata.items.key:kafka' \
--format 'value(name)'
--filter tells you which items to pick; in this case, it grabs the instance metadata, looks at the items, and checks the keys for those containing kafka (use = instead to look for keys that are exactly kafka).
--format tells you to grab just one value() (as opposed to a table, JSON, YAML) from each matching item; that item will be the name of the instance.
You can learn more by running gcloud topic filters, gcloud topic formats, and gcloud topic projections.
Here is a simple jq solution using if and any:
if .metadata.items | any(.key == "kafka") then . else empty end
| .name
I'm trying to extract coverage data from our SonarQube instance using the 'api/measures/component' endpoint. This is for a multi module java project. Using POSTMAN i can query for a single java file using a 'componentKey' which is build up of string like this
[maven-groupId]:[maven-artifactId]:[path to java file]
which generates a string like
com.i.pc.e.components.o:om-wf-e:src/main/java/com/i/e/o/wf/actions/Xxxx.java
and returns this json response
{
"component": {
"id": "AVci_6G7elHqFlTqG_OC",
"key": "com.i.pc.e.components.o:om-wfl-e:src/main/java/com/i/e/o/wf/actions/Xxxx.java",
"name": "Xxxx.java",
"qualifier": "FIL",
"path": "src/main/java/com/i/e/om/wf/actions/Xxxxx.java",
"language": "java",
"measures": []
},
"metrics": [
{
"key": "coverage",
"name": "Coverage",
"description": "Coverage by unit tests",
"domain": "Coverage",
"type": "PERCENT",
"higherValuesAreBetter": true,
"qualitative": true,
"hidden": false,
"custom": false,
"decimalScale": 1,
"bestValue": "100.0",
"worstValue": "0.0"
}
]
}
This is fine for a one-off request but I need to do this for a number of Java source files and I won't have the Maven details to hand.
Is there a convention or documentation on the format of the 'componentKey'?
Ideally i'd like to define the full package and java source file name in the 'compomentKey' or is there a way to look up the 'componentId' via some other REST API call?
Use the components service to iterate through your tree (or sub-trees within your tree) starting from your project and feeding your sonar.projectKey value into the baseComponentKey parameter. Pay particular attention to the strategy parameter, which determines what sub-set of children a call retrieves.
Once you have the data on each child, it should be simple to iterate through them to get their measures.