Is it possible to trigger a local cloud function when a firestore document is modified on production? The function below works when it is deployed on google cloud with a cloud trigger. How can I run it on my machine and specify a firestore trigger?
import functions_framework
#functions_framework.cloud_event
def hello_firestore(event, context):
resource_string = context.resource
# print out the resource string that triggered the function
print(f"Function triggered by change to: {resource_string}.")
# now print out the entire event object
print(str(event))
Related
I'm trying to enable thumbnail caching in my Superset instance. The official docs provide an example of caching the thumbnail images on Amazon S3. But I'm on the Azure cloud. How could I rework this block of code to work with an Azure blob (or other appropriate Azure storage) instead?
From the docs:
from flask import Flask
from s3cache.s3cache import S3Cache
...
class CeleryConfig(object):
broker_url = "redis://localhost:6379/0"
imports = ("superset.sql_lab", "superset.tasks", "superset.tasks.thumbnails")
result_backend = "redis://localhost:6379/0"
worker_prefetch_multiplier = 10
task_acks_late = True
CELERY_CONFIG = CeleryConfig
def init_thumbnail_cache(app: Flask) -> S3Cache:
return S3Cache("bucket_name", 'thumbs_cache/')
THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache
# Async selenium thumbnail task will use the following user
THUMBNAIL_SELENIUM_USER = "Admin"
I have the rest of my caching working with Redis. I see these Python packages that look relevant: https://github.com/alejoar/Flask-Azure-Storage and https://pypi.org/project/azure-storage-blob/ but I'm not sure where to go next.
I want to create an app that supports multiple users. Every user can upload multiple files and each file uploaded needs to be processed by a pipeline (for example processing1.exe -> processing2.py -> processing3.exe). The results must be made available to the user after processing.
The backend is in Laravel.
I have the following questions:
I tried running the processing1.exe binary on Azure Functions but I got "access denied". I guess this is because the default Windows image does not have the necessary dependencies installed (Windows SDK). From what I read there are other offerings of "serverless" in Azure like Logic Apps, Custom containers etc. Is there a way to use "serverless"/Azure Functions with that binary? What are my options?
Everytime a new file upload is detected (by Laravel), how should I trigger Azure to start processing it? Should I Azure Storage Queues, Azure Service Bus, Event Hub or Event Grid (or something else)?
Ideally the system shouldn't process all the files from users in a FIFO manner but instead use round-robin (that way if user1 uploads 1000 files and user2 uploads 3 files, user2 wouldn't have to wait for all of user1 files to finish).
Few of the workarounds for running the .exe file on Azure Functions:
Workaround 1:
There is one of my practical workarounds on running and getting .exe output in Azure Functions, please refer this SO Thread.
Workaround 2:
My .exe file contains SQL database query that insert record into database using Azure Function App:
cmd.CommandText = "insert into [dbo].[debug]([Name]) values('test')";
run.csx:
using System;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
System.Diagnostics.Process process = new System.Diagnostics.Process();
process.StartInfo.FileName = #"D:\home\site\wwwroot\TimerTriggerclass1\demofunction.exe";
process.StartInfo.Arguments = "";
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.Start();
string output = process.StandardOutput.ReadToEnd();
string err = process.StandardError.ReadToEnd();
process.WaitForExit();
log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
}
Before running the Function App in Azure, we need to upload the .exe file in the Azure Functions App > Functions (TimerTriggerclass1) > Upload File option.
Workaround 3:
using System;
using System.Diagnostics;
using System.Threading;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
System.Diagnostics.Process process = new System.Diagnostics.Process();
string WorkingDirectoryInfo =#"D:\home\site\wwwroot\TimerTriggerClass1";
string ExeLocation = #"D:\home\site\wwwroot\TimerTriggerClass1\MyApplication.exe";
Process proc = new Process();
ProcessStartInfo info = new ProcessStartInfo();
try
{
info.WorkingDirectory = WorkingDirectoryInfo;
info.FileName = ExeLocation;
info.Arguments = "";
info.WindowStyle = ProcessWindowStyle.Minimized;
info.UseShellExecute = false;
info.CreateNoWindow = true;
proc.StartInfo = info;
proc.Refresh();
proc.Start();
proc.WaitForInputIdle();
proc.WaitForExit();
}
catch
{
}
log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
}
Everytime a new file upload is detected (by Laravel), how should I trigger Azure to start processing it? Should I Azure Storage Queues, Azure Service Bus, Event Hub or Event Grid (or something else)? Ideally the system shouldn't process all the files from users in a FIFO manner but instead use round-robin (that way if user1 uploads 1000 files and user2 uploads 3 files, user2 wouldn't have to wait for all of user1 files to finish).
For this process, the pure serverless approach like Azure Functions with Blob Storage Trigger or Azure Logic Apps with Blob Storage Trigger is better.
Whenever the new file is uploaded, then the Blob Storage will trigger the Azure Function to process the files immediately.
It will not make wait of the user 2 until the user1 finishes all files processing. User 2 can get the processed result at the same time when files are uploaded.
In Comparison, Logic Apps also does the same functionality whenever the files are added or modified but Logic Apps has built-in processing methods and connectors like after processing, share or save it to some other storages or mails whereas if you need customized functionality before/on/after processing the files uploaded on Azure Storage, Azure functions is the best approach to write the custom optimized code.
I am trying to get all the instance(server name) ID based on the app. Let's say I have an app in the server. How do I know which apps below to which server. I want my code to find all the instance (server) that belongs to each app. Is there any way to look through the app in the ec2 console and figure out the servers are associated with the app. More of using tag method
import boto3
client = boto3.client('ec2')
my_instance = 'i-xxxxxxxx'
(Disclaimer: I work for AWS Resource Groups)
Seeing your comments that you use tags for all apps, you can use AWS Resource Groups to create a group - the example below assumes you used App:Something as tag, first creates a Resource Group, and then lists all the members of that group.
Using this group, you can for example get automatically a CloudWatch dashboard for those resources, or use this group as a target in RunCommand.
import json
import boto3
RG = boto3.client('resource-groups')
RG.create_group(
Name = 'Something-App-Instances',
Description = 'EC2 Instances for Something App',
ResourceQuery = {
'Type': 'TAG_FILTERS_1_0',
'Query': json.dumps({
'ResourceTypeFilters': ['AWS::EC2::Instance'],
'TagFilters': [{
'Key': 'App',
'Values': ['Something']
}]
})
},
Tags = {
'App': 'Something'
}
)
# List all resources in a group using a paginator
paginator = RG.get_paginator('list_group_resources')
resource_pages = paginator.paginate(GroupName = 'Something-App-Instances')
for page in resource_pages:
for resource in page['ResourceIdentifiers']:
print(resource['ResourceType'] + ': ' + resource['ResourceArn'])
Another option to just get the list without saving it as a group would be to directly use the Resource Groups Tagging API
What you install on an Amazon EC2 instance is totally up to you. You do this by running code on the instance itself. AWS is not involved in the decision of what you install on the instance, nor does it know what you installed on an instance.
Therefore, you will need to keep track of "what apps are installed on what server" yourself.
You might choose to take advantage of Tags on instances to add some metadata, such as the purpose of the server. You could also use AWS Systems Manager to run commands on instances (eg to install software) or even use AWS CodeDeploy to roll-out software to fleets of servers.
However, even with all of these deployment options, AWS cannot track what you have put on each individual server. You will need to do that yourself.
Update: You can use AWS Resource Groups to view/manage resources by tag.
Here's some sample Python code to list tags by instance:
import boto3
ec2_resource = boto3.resource('ec2', region_name='ap-southeast-2')
instances = ec2_resource.instances.all()
for instance in instances:
for tag in instance.tags:
print(instance.instance_id, tag['Key'], tag['Value'])
I'm using jovo framework (version 1.0.0) and I'm facing the following problem:
In app.js:
app.setHandler({
'LAUNCH': function() {
if(this.user().isNewUser()) {
this.tell('This will never be told on AWS Lambda.');
}
}
});
Running locally I can distinguish between (isNewUser === true) and (isNewUser === false) but as soon as I'm executing it as a lambda function on AWS isNewUser is always false. Why is that?
and additionally
NEW_USER': function() {
}
Isn't triggered either.
System Environment on local machine:
Windows 10 Home
NodeJS: v8.9.1
Lambda function:
NodeJS 6.10
I really appreciate any help you can provide.
Both 'NEW_USER' and this.user().isNewUser() need to have access to a database, where the number of sessions is stored for each user.
When you're prototyping locally, it uses the default File Persistence database integration, which saves the data to a local db/db.json file.
However, on AWS Lambda, the local db doesn't work, so you need to set up a DynamoDB configuration. Learn more here: Jovo Framework Docs > Database Integrations > DynamoDB.
Please remember to give your Lambda function role the right permission to access DynamoDB data: AWS Lambda Permissions Model.
I have an an AWS Lambda function that needs to connect to a remote TCP service. Is there any way to configure the Lambda function with the IP address of the remote service after the Lambda function has been deployed to AWS? Or do I have to bake the configuration into the packaged Lambda function before it's deployed?
I found a way that I use for supporting a test environment and a production environment that will help you.
For the test version of the function, I am calling it TEST-ConnectToRemoteTcpService and for the production version of the function I am naming the function PRODUCTION-ConnectToRemoteTcpService. This allows me pull out the environment name using a regular expression.
Then I am storing config/test.json and config/production.json in the zip file that I upload as code for the function. This zip file will be extracted into the directory process.env.LAMBDA_TASK_ROOT when the function runs. So I can load that file and get the config I need.
Some people don't like storing the config in the code zip file, which is fine - you can just load a file from S3 or use whatever strategy you like.
Code for reading the file from the zip:
const readConfiguration = () => {
return new Promise((resolve, reject) => {
let environment = /^(.*?)-.*/.exec(process.env.AWS_LAMBDA_FUNCTION_NAME)[1].toLowerCase();
console.log(`environment is ${environment}`);
fs.readFile(`${process.env.LAMBDA_TASK_ROOT}/config/${environment}.json`, 'utf8', function (err,data) {
if (err) {
reject(err);
} else {
var config = JSON.parse(data);
console.log(`configuration is ${data}`);
resolve(config);
}
});
});
};
Support for environment variables was added for AWS Lambda starting November 18, 2016. Adding a variable to an existing function can be done via command line as shown below or from the AWS Console.
aws lambda update-function-configuration \
--function-name MyFunction \
--environment Variables={REMOTE_SERVICE_IP=100.100.100.100}
Documentation can be found here.
You can invoke the Lambda function via SNS topic subscription and have it configure itself from the payload inside the SNS event.
Here's the official guide on how to do that Invoking Lambda via SNS.
A few options, depending on the use-case
If your config will not change then you can use S3 objects and access from Lambda or set your Lambda to trigger on new config changes. (Though this is the cheapest way, you are limited in what you can do compared to other alternatives)
If the config is changing constantly, then DynamoDB - Key/value is an alternative.
If DynamoDB is expensive for the frequent read/writes and not worth the value then you can have TCP service post config into a SQS queue. (or an SNS if you want to trigger when the service posts a new config)