Azure Data Factory how to deploy Alerts & Metrics to other environments with DevOps - continuous-integration

We have a Azure datafactory fully integrated with DevOps. Every change I make to the datafactory is deployed to all environments (OTAP), except alerts & metrics. I cannot find anything on how to deploy these to the other environments. Is this possible at all?

Is this possible at all?'
Quick answer is NO so far. I contacted Microsoft ADF team and got below response:
Azure Data Factory utilizes Azure Resource Manager templates to store
the configuration of your various ADF entities. Entities on Alerts &
Metrics does not get exported in the ARM template, so Alerts & Metrics
won’t be integrated using DevOps.
I did 2 verifications:
1.Check ARM template supported entities in ADF, Alerts & Metrics doesn't exist.
2.Try to export ARM template in the ADF UI but still no Alerts & Metrics
Really understand you would like to integrate all elements in Data Factory including Alerts & Metrics with DevOps. I suggest you submitting feedback here to push improvements of ADF, any voice is welcome.

There is a way to work around this one.
ADF alert is a "Microsoft.Insights/metricalerts" resource that you can deploy using ARM deployment operation from Azure Devops.
You can try to create an alert in ADF and then go to Portal, search for: Monitor > Alert > Alert Rule, and find the Alert you created in ADF. In my case there is an Alert called Test
Here is the ARM template exported from the alert
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"metricalerts_Alert_name": {
"defaultValue": "Alert",
"type": "String"
},
"factories_test_externalid": {
"defaultValue": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/yyyyy/providers/Microsoft.DataFactory/factories/test",
"type": "String"
},
"actionGroups_actiongroup1_externalid": {
"defaultValue": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/yyyyy/providers/microsoft.insights/actionGroups/actiongroup1",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Insights/metricalerts",
"apiVersion": "2018-03-01",
"name": "[parameters('metricalerts_Alert_name')]",
"location": "global",
"tags": {
"CreatedTimeUtc": "2022-09-13T05:28:46.0663823Z"
},
"properties": {
"severity": 0,
"enabled": true,
"scopes": [
"[parameters('factories_test_externalid')]"
],
"evaluationFrequency": "PT1M",
"windowSize": "PT15M",
"criteria": {
"allOf": [
{
"threshold": 1,
"name": "PipelineFailedRuns",
"metricNamespace": "Microsoft.DataFactory/factories",
"metricName": "PipelineFailedRuns",
"dimensions": [
{
"name": "Name",
"operator": "Include",
"values": [
"pipeline2"
]
},
{
"name": "FailureType",
"operator": "Include",
"values": [
"UserError",
"SystemError",
"BadGateway"
]
}
],
"operator": "GreaterThanOrEqual",
"timeAggregation": "Total",
"criterionType": "StaticThresholdCriterion"
}
],
"odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria"
},
"actions": [
{
"actionGroupId": "[parameters('actionGroups_actiongroup1_externalid')]",
"webHookProperties": {}
}
]
}
}
]
}
For the actionGroups you can refer to
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"groupName": {
"defaultValue": "actiongroup1",
"type": "string"
},
"email_receiver_address": {
"defaultValue": "someEmail#gmail.com",
"type": "string"
}
},
"variables": {},
"resources": [
{
"type": "microsoft.insights/actionGroups",
"apiVersion": "2019-03-01",
"name": "[parameters('groupName')]",
"location": "global",
"tags": {
"CreatedTimeUtc": "2020-10-21T07:24:08.2808723Z",
},
"properties": {
"groupShortName": "test",
"enabled": true,
"emailReceivers": [
{
"name": "test email received",
"emailAddress": "[parameters('email_receiver_address')]",
"useCommonAlertSchema": false
}
],
"smsReceivers": [],
"webhookReceivers": [],
"itsmReceivers": [],
"azureAppPushReceivers": [],
"automationRunbookReceivers": [],
"voiceReceivers": [],
"logicAppReceivers": [],
"azureFunctionReceivers": []
}
}
]
}

Related

Unable to start AWSFIS-Run-CPU-Stress

While running AWSFIS-Run-CPU-Stress i am getting below error:
Unable to start action, due to a platform mismatch between the specified document and the targeted instances. I am trying this in Windows EC2 instance
My Experiment script look like this(removed confidential server info):
{
"description": "Test CPU stress predefined SSM document",
"targets": {
"testInstance": {
"resourceType": "aws:ec2:instance",
"resourceArns": [
"arn:aws:ec2:region:123456789012:instance/instance_id"
],
"selectionMode": "ALL"
}
},
"actions": {
"runCpuStress": {
"actionId": "aws:ssm:send-command",
"parameters": {
"documentArn": "arn:aws:ssm:region::document/AWSFIS-Run-CPU-Stress",
"documentParameters": "{\"DurationSeconds\":\"120\"}",
"duration": "PT5M"
},
"targets": {
"Instances": "testInstance"
}
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:region:123456789012:alarm:awsec2-instance_id-GreaterThanOrEqualToThreshold-CPUUtilization"
}
],
"roleArn": "arn:aws:iam::123456789012:role/AllowFISSSMActions",
"tags": {}
}

ARM implicitly created Event Grid System topic .How Event grid subscription attach to Custom Event gird topic

I have facing issue on deploying a resources via ARM . I have 2 queries .
Query 1 .
Why this below Linked ARM implicitly created Event Grid System Topic This Linked ARM is only used to get trigger URL of azure function
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"functionAppName": {
"type": "String"
}
},
"variables": {
"sitesWebApiVersion": "2016-08-01"
},
"resources": [],
"outputs": {
"triggerUrl": {
"type": "String",
"value": "[listsecrets(resourceId('Microsoft.Web/sites/functions', parameters('functionAppName'), 'NPEventGridDataProcessor'), variables('sitesWebApiVersion')).trigger_url]"
}
}
}
Query2
In ARM how I create Event Subscription that attach to explicitly created custom Event Topic. As see the below the master ARM code snippet , event subscription attach unwanted to the Event Grid System Topic . But I want to add it on Custom Event Grid topic .
{
"apiVersion": "2020-06-01",
"scope": "[format('Microsoft.EventGrid/topics/{0}', variables('eventGridName'))]",
"name": "[variables('eventGridSubwebhookName')]",
"type": "Microsoft.EventGrid/eventSubscriptions",
"tags": {
"displayName": "Webhook Subscription"
},
"dependsOn": [
"[resourceId('Microsoft.EventGrid/topics/', variables('eventGridName'))]",
"[resourceId('Microsoft.Resources/deployments/', variables('hackForGettingTriggerUrl'))]"
],
"properties": {
"destination": {
"endpointType": "WebHook",
"properties": {
"endpointUrl": "[reference(variables('hackForGettingTriggerUrl')).outputs.triggerUrl.value]"
}
}
}
}
]
If you want to create event gird subscription for custom topic, please refer to the following template
"resources": [
{
"type": "Microsoft.EventGrid/topics",
"apiVersion": "2020-06-01",
"name": "[parameters('eventGridTopicName')]",
"location": "[parameters('location')]"
},
{
"type": "Microsoft.EventGrid/eventSubscriptions",
"apiVersion": "2020-06-01",
"scope": "[format('Microsoft.EventGrid/topics/{0}', parameters('eventGridTopicName'))]",
"name": "[parameters('eventGridSubscriptionName')]",
"properties": {
"destination": {
"endpointType": "WebHook",
"properties": {
"endpointUrl": "[parameters('eventGridSubscriptionUrl')]"
}
}
},
"dependsOn": [
"[resourceId('Microsoft.EventGrid/topics', parameters('eventGridTopicName'))]"
]
}
],

Alexa.Discovery response: no device detected by Alexa

I am implementing my Alexa Home Skill using AWS Lambda.
Given the following request I receive when I try to detect new devices on Alexa Skil test page:
{directive={header={namespace=Alexa.Discovery, name=Discover, payloadVersion=3, messageId=0160c7e7-031f-47ee-a1d9-a23f38f87a9e}, payload={scope={type=BearerToken, token=...}}}}
I respond with the following:
{
"event": {
"payload": {
"endpoints": [
{
"displayCategories": [
"SMARTPLUG"
],
"capabilities": [
{
"type": "AlexaInterface",
"interface": "Alexa",
"version": "3"
},
{
"type": "AlexaInterface",
"interface": "Alexa.PowerController",
"version": "3",
"properties": {
"retrievable": true,
"supported": [
{
"name": "powerState"
}
],
"proactivelyReported": true
}
},
{
"type": "AlexaInterface",
"interface": "Alexa.EndpointHealth",
"version": "3",
"properties": {
"retrievable": true,
"supported": [
{
"name": "connectivity"
}
],
"proactivelyReported": true
}
}
],
"manufacturerName": "mirko.io",
"endpointId": "ca84ef6d-53b1-430a-8a5e-a62f174eac5e",
"description": "mirko.io forno (id: ca84ef6d-53b1-430a-8a5e-a62f174eac5e)",
"friendlyName": "forno"
}
]
},
"header": {
"payloadVersion": "3",
"namespace": "Alexa.Discovery",
"name": "Discover.Response",
"messageId": "c0555cc8-ad7a-4377-b310-9de9b9ab6282"
}
}
}
Despite that, for some reasons Alexa answers that it did not find any new device.
I may be mistaken but I am pretty sure it used to work before I decided to add the Alexa.EndpointHealth interface.
Your response object looks right to me, except the extra "endpoint" field.
"endpoint": {
"endpointId": "INVALID",
"scope": {
"type": "BearerToken",
"token": "INVALID"
}
}
There's no such field in the Alexa.Discovery documentation. Try removing it and see if it resolves the issue.

IBM Social Business Toolkit getContacts API Fails With 403 Forbidden

I used to be able to use the IBM SBT getContacts API from either a perl script (with authentication provided in the request) or an internet browser (authentication popup would appear) and it would return the data in JSON format as documented on the IBM Connections Developers API reference page:
http://www-10.lotus.com/ldd/appdevwiki.nsf/xpAPIViewer.xsp?lookupName=API+Reference#action=openDocument&res_title=Contacts_getContacts_API&content=apicontent&sa=true
Recently, however, the API call has been returning a 403 forbidden status. If I sign in to the IBM Connections Cloud before trying to use the API via a browser, it is converted to:
https://apps.na.collabserv.com/mycontacts/home.html
Does anyone know what I need to do to get the API to work again?
Thanks.
The best API to use are
https://apps.na.collabserv.com/lotuslive-shindig-server/social/rest/people/#me/#all?filterBy=type&filterOp=equals&filterValue=contacts&format=json
You'll get a list of the following entries back...
{
"photos":
[
],
"telephone": "",
"profileUrl": "",
"mobilephone": "",
"orgs":
[
{
"value": "CONTACTS",
"type": "Source"
},
{
"value": "Mentorship Expressway",
"type": "Org"
}
],
"website": "",
"id": "na.collabserv.com:contact:160000909070",
"fax": "",
"connectedToId": 201238541234,
"addresses":
[
],
"emailAddress": "asdfasdf#us.ibm.com",
"websites":
[
],
"objectId": 160000909070,
"type": "FRIEND",
"jobtitle": "",
"updated": "2014-07-01T16:44:21.000Z",
"ims":
[
],
"emails":
[
{
"value": "CONTACTS",
"type": "Source"
},
{
"title": "Primary Email",
"email": "asdfasdf#us.ibm.com"
}
],
"org":
{
"name": "Mentorship Expressway"
},
"displayName": "asdfasdf",
"address": "",
"companyId": 22285,
"phoneNumbers":
[
{
"value": "CONTACTS",
"type": "Source"
},
{
"title": "MOBILE",
"phone": "+ asdfasdf"
}
]
}

Need to execute a ruby file from Amazon web service Data pipeline

i have a ruby file in my application and i need to call and execute a ruby file as background job from amazon web service data pipeline
i have given the json file below
#json file
{ "objects": [
{
"id": "ScheduleId4",
"startDateTime": "2013-08-01T00:00:00",
"name": "schedule",
"type": "Schedule",
"period": "15 Minutes"
},
{
"id": "DataNodeId2",
"schedule": {
"ref": "ScheduleId4"
},
"name": "Input",
"directoryPath": "s3://pipeline_test/input/",
"type": "S3DataNode"
},
{
"id": "ActivityId1",
"input": {
"ref": "DataNodeId2"
},
"schedule": {
"ref": "ScheduleId4"
},
"stdout": "s3://pipeline_test/logs",
"scriptUri": "s3://pipeline_test/input/sample.sh",
"name": "Shell",
"runsOn": {
"ref": "ResourceId5"
},
"stderr": "s3://pipeline_test/logs",
"type": "ShellCommandActivity",
"output": {
"ref": "DataNodeId3"
},
"stage": "true"
},
{
"terminateAfter": "1 Hours",
"id": "ResourceId5",
"schedule": {
"ref": "ScheduleId4"
},
"name": "Resource1",
"logUri": "s3://pipeline_test/logs/",
"type": "Ec2Resource"
},
{
"id": "Default",
"scheduleType": "timeseries",
"name": "Default",
"role": "DataPipelineDefaultRole",
"resourceRole": "DataPipelineDefaultResourceRole"
},
{
"id": "DataNodeId3",
"schedule": {
"ref": "ScheduleId4"
},
"directoryPath": "s3://pipeline_test/output1/",
"name": "Output",
"type": "S3DataNode"
}
]
}
sample.sh
echo "Hello"
ruby sample.rb
sample.rb
puts "Hello world"
i have given correct path of sample.sh file. Still i am not to get the sample.rb calling or not.
Anyone tell me step by step procedure to follow it as i am newbie to amazon web service datapipeline.
Help me to solve it.
The default image launched by Data Pipeline does not actually have ruby on it. You'll have to build your own image and install ruby by hand first. Then, reference that image in your Resource by instanceId

Resources