Azure Data Factory Until loop slow to end/terminate - performance

We have an until loop in a ADFv2 pipeline.
The time it takes to stop/terminate once the expression condition is met seems to corrolate between the length of time the until loop takes to completes its activities.
This particular until loop performs alot of activites and can take anywhere between 90-120 mins to complete. So it takes almost as long to end/terminate (break out of the loop).
If I "hack" it so that it only performs a handful of activities it will quickly end and break once it's finished and the expression to terminate is met.
It's like a spinning wheel that keeps spinning even after the power is turned off. The momentum that was built up while connected takes a while to slow down and eventually stop.
Is this a known issue, how can I troubleshoot the exact cause here or fix it?

Incorrect usage of nested loop switch could cause this.
Here is a Until component, and some activities after it:
Until
In the Until, some activities,
Correct:
In until - correct
Incorrect (Slow to end/terminate):
In until - incorrect
Why?
For the incorrect case, the last activity If waiting depends on three activities, perhaps its behavior was counterintuitive.
// pay attention to "dependsOn"
{
"name": "If waiting",
"type": "IfCondition",
"dependsOn": [
{
"activity": "Set loop_waiting_refresh_status to True",
"dependencyConditions": [
"Succeeded"
]
},
{
"activity": "WeChatEP_Notifier Info Get new bearer",
"dependencyConditions": [
"Succeeded"
]
},
{
"activity": "Set loop_waiting_refresh_status to False",
"dependencyConditions": [
"Succeeded"
]
}
],
...
}

Related

Redis clients can update cache simultaneously causing wrong state to be saved

I have been building a simple application that uses Redis as cache to store data regarding a game where each user has a score and after a user completes a task the score is updated for the user.
My problem is when a user completes a task his score is updated which means that it will update the record in redis by replacing the previous value with the new one (in my case it will replace the entire room object with the new one even though the room has not changed but only the score of the player inside the room has changed).
The thing is if multiple users complete a task at the same time they will send each at the same time the new record to redis and only the last one will receive the update.
For example:
In the redis cache this is the starting value: { roomId: "...", score:[{ "player1": 0 }, { "player2": 0 }] }
Player 1 completes a task and sends:
{ roomId: "...", score:[{ "player1": 1 }, { "player2": 0 }] }
At the same time Player 2 completes a task and sends:
{ roomId: "...", score:[{ "player1": 0 }, { "player2": 1 }] }
In the redis cache first it will be saved the value received from Player1 let's say and then the value from player 2 which means that the new value in the cache will be:
{ roomId: "...", score:[{ "player1": 0 }, { "player2": 1 }] }
Even though this is wrong because the correct value would be: { roomId: "...", score:[{ "player1": 1 }, { "player2": 1 }] } where both changes are present.
At the moment I am also using a pub/sub system to keep track of changes so that does are reflected to every server and each user connected to the server.
What can I do to fix this? For reference consider the following image as the architecture of the system:
The issue appears to be that you are interleaving one read/write set of operation with others, which leads to using stale data while updating keys. Fortunately, the fix is (relatively) easy: just combine your read/write chunk of operations into a single atomic unit, using either a Lua script, a transaction or, even easier, through a single RedisJSON command.
Here is an example using RedisJSON. Prepare your JSON key/document which will hold all the scores for the room first, using the JSON.SET command:
> JSON.SET room:foo $ '{ "roomId": "foo", "score": [] }'
OK
After that, use the JSON.ARRAPPEND command once you need to append an item to the score array:
> JSON.ARRAPPEND room:foo $.score '{ "player1": 123 }'
1
...
> JSON.ARRAPPEND room:foo $.score '{ "player2": 456 }'
2
Getting back the whole JSON document is as easy as running:
> JSON.GET room:foo
"{\"roomId\":\"foo\",\"score\":[{\"player1\":123},{\"player2\":456}]}"

AWS cron expression OK, lambda not triggered

I have set the following cron expression in AWS (CloudWatch trigger).
0 */5 7-12,1pm-11pm ? * MON,TUE,WED,THU,FRI
In expression generator, I get for very similar expresssion (7-23 intead of the hours )
At second :00, every 5 minutes starting at minute :00, every hour between 07am and 23pm, on every Monday, Tuesday, Wednesday, Thursday and Friday, every month
as expected.
However, it is not triggered. I don't see anything in the log.
Why is that? (trigger is enabled of course)
Thanks.
When you create CloudWatch Event Rule, or EventBridge Event Rule (is what AWS calls these days) and select Lambda function as target, there are 2 main points that you need to consider:
CRON SCHEDULE
You need to specify the cron schedule and this schedule timezone is UTC+0.
I assume that you are in different timezone and observe there is not any triggers since the next trigger time has not been reached yet.
RESOUCE-BASED POLICY
There is another chance that you need to check is about permissions, you need to concern is Lambda function's Resource Based Policy.
Go to AWS Console, you can check your Lambda's permission tab and review your permissions which is required to allow your Event Rule triggers.
{
"Version": "2012-10-17",
"Id": "default",
"Statement": [
{
"Sid": "AWSEvents",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:<FUNCTION_NAME>",
"Condition": {
"ArnLike": {
"AWS:SourceArn": "arn:aws:events:<REGION>:<ACCOUNT_ID>:rule/<RULE_NAME>"
}
}
}
]
}
Well, the expression wasn't right.
It considers minutes only and the */5 confused it too maybe. Not sure about the * at the end.
0/5 7-23 ? * MON-FRI * - this works
To debug triggers, there is "Rules" section in the log service, where you can see the next executions.
Since the CRON expression is in UTC time format. Could you please check if there is any time difference b/w your standard time and UTC time?
And change the expression accordingly.

Append an array to a json using jq in BASH

I have a json that looks like this:
{
"failedSet": [],
"successfulSet": [{
"event": {
"arn": "arn:aws:health:us-east-1::event/AWS_RDS_MAINTENANCE_SCHEDULED_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
"endTime": 1502841540.0,
"eventTypeCategory": "scheduledChange",
"eventTypeCode": "AWS_RDS_MAINTENANCE_SCHEDULED",
"lastUpdatedTime": 1501208541.93,
"region": "us-east-1",
"service": "RDS",
"startTime": 1502236800.0,
"statusCode": "open"
},
"eventDescription": {
"latestDescription": "We are contacting you to inform you that one or more of your Amazon RDS DB instances is scheduled to receive system upgrades during your maintenance window between August 8 5:00 PM and August 15 4:59 PM PDT. Please see the affected resource tab for a list of these resources. \r\n\r\nWhile the system upgrades are in progress, Single-AZ deployments will be unavailable for a few minutes during your maintenance window. Multi-AZ deployments will be unavailable for the amount of time it takes a failover to complete, usually about 60 seconds, also in your maintenance window. \r\n\r\nPlease ensure the maintenance windows for your affected instances are set appropriately to minimize the impact of these system upgrades. \r\n\r\nIf you have any questions or concerns, contact the AWS Support Team. The team is available on the community forums and by contacting AWS Premium Support. \r\n\r\nhttp://aws.amazon.com/support\r\n"
}
}]
}
I'm trying to add a new key/value under successfulSet[].event (key name as affectedEntities) using jq, I've seen some examples, like here and here, but none of those answers really show how to add a possible one key with multiple values (the reason why I say possible is because as of now, AWS is returning one value for the affected entity, but if there are more, then I'd like to list them).
EDIT: The value of the new key that I want to add is stored in a variable called $affected_entities and a sample of that value looks like this:
[
"arn:aws:acm:us-east-1:xxxxxxxxxxxxxx:certificate/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
]
The value could look like this:
[
"arn:aws:acm:us-east-1:xxxxxxxxxxxxxx:certificate/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx",
"arn:aws:acm:us-east-1:xxxxxxxxxxxxxx:certificate/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx",
...
...
...
]
You can use this jq,
jq '.successfulSet[].event += { "new_key" : "new_value" }' file.json
EDIT:
Try this:
jq --argjson argval "$new_value" '.successfulSet[].event += { "affected_entities" : $argval }' file.json
Test:
sat~$ new_value='[
"arn:aws:acm:us-east-1:xxxxxxxxxxxxxx:certificate/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
]'
sat~$ jq --argjson argval "$new_value" '.successfulSet[].event += { "affected_entities" : $argval }' file.json
Note that --argjson works with jq 1.5 and above.

Is there a way I can get historic performance data of various alerts in Nagios as json/xml?

I am looking to get performance data of various alerts setup in my Nagios Core/XI. I think it is stored in RRDs. Are there ways I can get access to it?
If you're using Nagios XI you can get this data a few different ways.
If you're using XI 5 or later, then the easiest way that springs to mind is the API. Log in to your XI server as an administrator, navigate to 'Help' menu, then select 'Objects Reference' on the left hand side navigation and find 'GET objects/rrdexport' from the Objects Reference navigation box (or just scroll down to near the bottom).
An example curl might look like this:
curl -XGET "http://nagiosxi/nagiosxi/api/v1/objects/rrdexport?apikey=YOURAPIKEY&pretty=1&host_name=localhost"
Your response should look something like:
{
"meta": {
"start": "1453838100",
"step": "300",
"end": "1453838400",
"rows": "2",
"columns": "4",
"legend": {
"entry": [
"rta",
"pl",
"rtmax",
"rtmin"
]
}
},
"data": {
"row": [
{
"t": "1453838100",
"v": [
"6.0373333333e-03",
"0.0000000000e+00",
"1.7536000000e-02",
"3.0000000000e-03"
]
},
{
"t": "1453838400",
"v": [
"6.0000000000e-03",
"0.0000000000e+00",
"1.7037333333e-02",
"3.0000000000e-03"
]
}
]
}
}
BUT WAIT, THERE IS ANOTHER WAY
This way will work no matter what version you're on, and would actually work if you were processing performance data with NPCD on a Core system as well.
Log in to your server via ssh or console and get your butt over to the /usr/local/nagios/share/perfdata directory. From here we're going to use the localhost object as an example..
$ cd /usr/local/nagios/share/perfdata/
$ ls
localhost
$ cd localhost/
$ ls
Current_Load.rrd Current_Users.xml HTTP.rrd PING.xml SSH.rrd Swap_Usage.xml
Current_Load.xml _HOST_.rrd HTTP.xml Root_Partition.rrd SSH.xml Total_Processes.rrd
Current_Users.rrd _HOST_.xml PING.rrd Root_Partition.xml Swap_Usage.rrd Total_Processes.xml
$ rrdtool dump _HOST_.rrd
Once you run the rrdtool dump command, there is going to be an awful lot of output, so I keep that as an exercise for you, the reader ;)
If you're trying to automate something of some kind, then you should note that the xml files contain meta data for the rrd files and could potentially be useful to parse first.
Also, if you're anything like me, you love reading technical manuals. Here is a great one to read: RRDTool documentation
Hope this helped!

Golang: healthd and healthtop of the library "gocraft/health"

Im using gocraft/health to check the health of my service and have the metrics of each endPoint.
Im usin The JSON polling sink to get the metrics.
sink := health.NewJsonPollingSink(time.Minute*5, time.Minute*5)
stream.AddSink(sink)
I want to use healthtop and healthd here Link they explain how.
I fixed the environment variables: export HEALTHD_MONITORED_HOSTPORTS=:5001 HEALTHD_SERVER_HOSTPORT=:5002 healthd
as they said
after they said "Now you can run it". how, they didn't give any command to do it.I didn't realy understand what they mean.
I navigated to src/github.com/gocraft/health/cmd/healthd. I found main.go when I run it I got that in the console
[openrtb#sd-69536 healthd]$ go run main.go
[2015-06-17T23:04:20.871743758Z]: job:general event:starting kvs:[health_host_port::5002 monitored_host_ports::5001,:5002 server_host_port::5002]
[2015-06-17T23:04:20.87810814Z]: job:poll status:success time:4 ms kvs:[host_port::5002]
[2015-06-17T23:04:20.881896459Z]: job:poll status:success time:8 ms kvs:[host_port::5001]
[2015-06-17T23:04:20.882338024Z]: job:recalculate status:success time:231 μs
[2015-06-17T23:04:23.275370787Z]: job:recalculate status:success time:6 μs
[2015-06-17T23:04:30.875230839Z]: job:poll status:success time:1573 μs kvs:[host_port::5002]
[2015-06-17T23:04:30.881415193Z]: job:poll status:success time:7 ms kvs:[host_port::5001]
.
.
but no reslute on the those endpoints
localhost:5002/jobs: Lists top jobs
localhost:5002/hosts: Lists all monitored hosts and their statuses
it gave me {"error": "not_found"}
excepte this localhost:5002/health I got this JSON responce
{
"instance_id": "sd-69536.1291",
"interval_duration": 3600000000000,
"aggregations": [
{
"interval_start": "2015-06-18T01:00:00+02:00",
"serial_number": 48,
"jobs": {
"general": {
"timers": {},
"events": {
"starting": 1
},
"event_errs": {},
"count": 0,
"nanos_sum": 0,
"nanos_sum_squares": 0,
"nanos_min": 0,
"nanos_max": 0,
"count_success": 0,
"count_validation_error": 0,
"count_panic": 0,
"count_error": 0,
"count_junk": 0
},
"poll": {
"timers": {},
"events": {},
"event_errs": {},
"count": 24,
"nanos_sum": 107049159,
"nanos_sum_squares": 6.06770682813009e+14,
"nanos_min": 1581783,
"nanos_max": 8259442,
"count_success": 24,
"count_validation_error": 0,
"count_panic": 0,
"count_error": 0,
"count_junk": 0
},
"recalculate": {
"timers": {},
"events": {},
"event_errs": {},
"count": 23,
"nanos_sum": 3501601,
"nanos_sum_squares": 6.75958305123e+11,
"nanos_min": 70639,
"nanos_max": 290877,
"count_success": 23,
"count_validation_error": 0,
"count_panic": 0,
"count_error": 0,
"count_junk": 0
}
},
"timers": {},
"events": {
"starting": 1
},
"event_errs": {}
}
]
}
but no idea what this result mean, because it doesn't have any relation with my
localhost:5001/health EndPoint that should normaly aggregate as they said.
What you downloaded is a binary so you can just invoke it with healthd if you're in the correct directory, they actually provide this example;
HEALTHD_MONITORED_HOSTPORTS=:5020 HEALTHD_SERVER_HOSTPORT=:5032 healthd
Which isn't setting env var as much as invoking healthd with those two values (export or something would be required to persist the change beyond the one command). healthtop more clearly states what it is but as you can see by their paths, they're both commands gocraft/health/cmd/healthtop. They have several examples of using healthtop from bash, not so explicit about healthd but it's the same.
If you ran that command (as you show in your question) then you may want to try healthtop jobs or something to that effect. I don't know a ton about this project and don't care to research it but from what I can tell healthd is just a service that collects results from various /health endpoints and makes them available in on API. It seems like they intend for you to use healthtop to on top of it to view reports.
Also note this;
Great! To get a sense of the type of data healthd serves, you can manually navigate to:
/jobs: Lists top jobs
/aggregations: Provides a time series of aggregations
/aggregations/overall: Squishes all time series aggregations into one aggregation.
/hosts: Lists all monitored hosts and their statuses.
However, viewing raw JSON is just to give you a sense of the data. See the next section...
I'm not sure what the domain is (localhost:5032 if you're running locally?) but you should probably just be able to go to localhost:5032/jobs and see the healthd is running and doing something. Also check your apps to confirm it's up and running. Don't expect any output from it directly, that's what healthtop is for.

Resources