Scheduled mapreduce job on Google Cloud Platform - hadoop

I'm developing a node.js application that basically stores user event logs in a database and shows insights about user actions.
For achieving this event logs must be analyzed by using a Mapreduce job which would run once a day automatically (every night).
I've found lots of tutorials about mapreduce on google cloud web site but I'm totally lost because there are several technologies and can't find a way to do it without using the command line and also there is no information about scheduling (I want that the whole analysis process to be entirely automated)
Please, could you provide me advice about what google technologies should I use or where I can find a good tutorial?
Thank you

You want to be looking at:
Dataproc (run Hadoop/Spark jobs out of the box)
Dataflow (develop 'pipelines' using the Dataflow/Beam programming model)

Related

ElasticSearch/ElasticCloud Alert Creation

I am a newbie in Elastic in general and currently I am trying to manage our alerts for CPU/Disk/Memory in Elastic Cloud. I can create the alerts manually just fine, but that takes a huge amount of time and if we migrate I want to be able to create the alerts in some automated way. In the past I have worked with Azure and created alerts with Az PowerShell and etc, so I am searching how to automate the alert creation for our infrastructure in Elastic Cloud. I went through the documentation for Alerts Link. But, im not sure I understand how to use the API to actually do this.
Is there a way to automate lets say creation of CPU alerts for 10 different hosts that we monitor with Elastic ? Is using the API the only way and are there any materials other than the official documentation that can help me achieve this? And am I even on the correct path? Thank you in advance.
Let me share knowledge of using Azure Monitor where you can connects the resources to Azure Monitor and manage the Alerts. Alerts can send you an email or call a web hook when some metric (for example database size or CPU usage) reaches the threshold. There are several ways to create Alerts- using Azure Portal, Command Line Interface, Powershell and Azure Monitor Rest API. Hope it will help you.
Even you can automate alerts using Azure Automation runbook with Mertic Alerts. where can automate the alerts according to the customized dimensional values and once the Alert criteria met it can even send an mail.

Laravel Monitoring Tool

I am new to Laravel and keep on learning its capability and I need some assistance on the website I am working on... So, I have a website that is somewhat having a timeout problem when connecting to a 3rd party app during a certain period... It could be an issue with this 3rd party having a maintenance/ downtime, but is there any way I can check/ monitor the Laravel site itself for scheduled tasks running during those affected time?
Or could it also be monitored using Apache logs ?
I tried searching online but it appears I need to install some cron monitoring tools.
Laravel has Telescope, a very good debug tool. You can see scheduled tasks, requests, events etc ...

Monitor Windows Events (e.g. MSSQLServer) in Azure?

I searched for about one week for the oppurtunity to monitor different windows events, for example the SQL-event(or service) in AzureRM virtual machines.
I tried it with different LogAnalytics queries, Runbooks, Powershell scripts to connect to the vm, etc. But everything I tried doesn't work.
Do you have any suggestions?
The solution should inform me, when a windows service stopped.
Best regards!
Azure Monitor, which now includes Log Analytics and Application Insights, provides sophisticated tools for collecting and analyzing telemetry that allow you to maximize the performance and availability of your cloud and on-premises resources and applications. It helps you understand how your applications are performing and proactively identifies issues affecting them and the resources they depend on.
Activity log alerts are the alerts that get activated when a new activity log event occurs that matches the conditions specified in the alert. You can follow the following document to set an activity log alert: https://learn.microsoft.com/en-us/azure/monitoring-and-diagnostics/alert-activity-log?toc=/azure/azure-monitor/toc.json

Schedule CronJobs with the PHP Buildpack

For my PHP Web App I am using the PHP Buildpack. Now I would like to schedule a Tasks that should be triggered every month. Normally I would use CronJobs for that.
How can I achieve that within the Swisscom Application Cloud?
Swisscom App Cloud is based on Open Source Cloud Foundry
Upstream Cloud Foundry doesn’t have a feature equivalent to cron jobs (task scheduler). Stay tuned, I guess this feature will be soon implemented, because lots of people migrating from Heroku to CF. Heroku offers a cron job feature. Subscribe to Swisscom App Cloud Newsletter to read announcements.
There are workarounds for scheduling tasks, see Scheduling tasks on Cloud Foundry on blog.pivotal.io for a Ruby/Rake based example. Sorry for PHP I didn't found example code. There is no elegant solution! You need to implement yourself some kind of workaround. Would be great if you publish your code to GitHub.
If you need cron jobs only in data store, for example MariaDB offers Events.
Events are named database objects containing SQL statements that are
to be executed at a later stage, either once off, or at regular
intervals.
They function very similarly to the Windows Task Scheduler or Unix
cron jobs.
We had a simular issue. As written by #Fyodor, there is no native solution in Cloud Foundry. We did some research and found vendors like https://www.iron.io/.
Finally, we ended up with a very simple solution.
We expose all our background jobs via an https interface.
As we anyhow use Jenkins for CI/CD and it has lots of scheduling capabilities, we use our existing Jenkins to trigger these jobs via a simple cURL call to the HTTP endpoints.

Logging user interactions with a program

We have an app and we want to log how the user is interacting with it. For example are they using the pages we expect them to. I dont want to log this via the app as it will be very hard for me to then get this information from the device. Each page interacts with webservices so I was planning to log that interaction.
I have had some thoughts on this
* as the webservice is being called add a logging table to the database - problem here could be performance impact
* use log4j async mode to log these details.
Does anyone have any other suggestion on how to do this? Im reading the Lean Startup at the moment (very good so far) and this sort of thing seems fundamental to it so Im wondering if there are any other tips to this.
Thanks
Since no one answered this for a couple months, I thought a couple pointers might help you...
Use mobile analytics tools
Fabric.io
Google Analytics for Mobile Apps
Flurry
Amazon Mobile Analytics
appsee
Have the server record what users access (that's the approach you're considering). To offload the overhead, there are a couple tactics you could employ (mix 'n match as you will):
Use async mechanisms (async operations in the server, such as Futures; log4j async mode; async databases; etc).
Use a separate database.
Use a NoSQL database only to write accesses. Later on you process that information in a separate analytics application.
Have the client (mobile app) record the actions and send them in bulk to the server once in a while (as frequently as you need / want / can afford).
Cheers

Resources