What's the standard means of processing input to an AWS Lambda handler function, when the format of the incoming JSON varies depending on the type of trigger?
e.g. I have a Lambda function that gets called when an object is created in an S3 bucket, or when an hourly scheduled event fires. Obviously, the JSON passed to the handler is formatted differently.
Is it acceptable to overload Lambda handler functions, with the input type defined as S3Event for one signature and ScheduledEvent for the other? If not, are developers simply calling JsonConvert.DeserializeObject in try blocks? Or is the standard practice to establish multiple Lambda functions, one for each input type (yuck!)?
You should use one function per event.
Having multiple triggers for one Lambda will just make things way harder, as you'll end up with a bunch of if/else, switch statements or even Factory methods if you want to apply design patterns.
Now think of Lambda functions as small and maintainable. Think of pieces of code that should do one thing and should do it well. By the moment you start having multiple triggers, you kind of end up with a "Lambda Monolith", as it will have way too many responsibilities. Not only that, you strongly couple your Lambda functions with your events, meaning that once a new trigger is added, your Lambda code should change. This is just not scalable after two or three triggers.
Another drawback is that you are bound to using one language only if you architect it like that. For some use cases, Java may be the best option. But for others, it may be Node JS, Python, Go...
Essentially, your functions should be small enough to be easily maintainable and even rewritten if necessary. There's absolutely nothing wrong with creating one function per event, although, apparently, you strongly disapprove it. Think of every Lambda as a separate Microservice, which scales out independently, has its own CI/CD pipeline and its own suite of tests.
Another thing to consider is if you want to limit your Lambda concurrent executions depending on your trigger type. This would be unachievable via the "One-Lambda-Does-It-All" model.
Stick with one Lambda per trigger and you'll sleep better at night.
This is actually possible by doing the following:
Have the Lambda signature take a Stream rather than the Amazon event type, so we can get the raw JSON message.
Read the JSON contents of the stream as a string.
Deserialize the string to a custom type in order to identify the event source.
Use the event source information to deserialize the JSON a second time to the appropriate type for the event source.
For example:
public async Task FunctionHandler(Stream stream, ILambdaContext context)
{
using var streamReader = new StreamReader(stream);
var json = await streamReader.ReadToEndAsync();
var serializationOptions = new JsonSerializationOptions { PropertyNameCaseInsensitive = true };
var awsEvent = JsonSerializer.Deserialize<AwsEvent>(json, serializationOptions);
var eventSource = awsEvent?.Records.Select(e => e.EventSource).SingleOrDefault();
await (eventSource switch
{
"aws:s3" => HandleAsync(Deserialize<S3Event>(json, serializationOptions), context),
"aws:sqs" => HandleAsync(Deserialize<SQSEvent>(json, serializationOptions), context),
_ => throw new ArgumentOutOfRangeException(nameof (stream), $"Unsupported event source '{eventSource}'."),
});
}
public async Task HandlyAsync(S3Event #event) => ...
public async Task HandleAsync(SQSEvent #event) => ...
public sealed class AwsEvent
{
public List<Record> Records { get; set; }
public sealed class Record
{
public string EventSource { get; set; }
}
}
Related
I am wondering what is the use of asObservable:
As per docs:
An observable sequence that hides the identity of the
source sequence.
But why would you need to hide the sequence?
When to use Subject.prototype.asObservable()
The purpose of this is to prevent leaking the "observer side" of the Subject out of an API. Basically to prevent a leaky abstraction when you don't want people to be able to "next" into the resulting observable.
Example
(NOTE: This really isn't how you should make a data source like this into an Observable, instead you should use the new Observable constructor, See below).
const myAPI = {
getData: () => {
const subject = new Subject();
const source = new SomeWeirdDataSource();
source.onMessage = (data) => subject.next({ type: 'message', data });
source.onOtherMessage = (data) => subject.next({ type: 'othermessage', data });
return subject.asObservable();
}
};
Now when someone gets the observable result from myAPI.getData() they can't next values in to the result:
const result = myAPI.getData();
result.next('LOL hax!'); // throws an error because `next` doesn't exist
You should usually be using new Observable(), though
In the example above, we're probably creating something we didn't mean to. For one, getData() isn't lazy like most observables, it's going to create the underlying data source SomeWeirdDataSource (and presumably some side effects) immediately. This also means if you retry or repeat the resulting observable, it's not going to work like you think it will.
It's better to encapsulate the creation of your data source within your observable like so:
const myAPI = {
getData: () => return new Observable(subscriber => {
const source = new SomeWeirdDataSource();
source.onMessage = (data) => subscriber.next({ type: 'message', data });
source.onOtherMessage = (data) => subscriber.next({ type: 'othermessage', data });
return () => {
// Even better, now we can tear down the data source for cancellation!
source.destroy();
};
});
}
With the code above, any behavior, including making it "not lazy" can be composed on top of the observable using RxJS's existing operators.
A Subject can act both as an observer and an observable.
An Obervable has 2 methods.
subscribe
unsubscribe
Whenever you subscribe to an observable, you get an observer which has next, error and complete methods on it.
You'd need to hide the sequence because you don't want the stream source to be publicly available in every component. You can refer to #BenLesh's example, for the same.
P.S. : When I first-time came through Reactive Javascript, I was not able to understand asObservable. Because I had to make sure I understand the basics clearly and then go for asObservable. :)
In addition to this answer I would mention that in my opinion it depends on the language in use.
For untyped (or weakly typed) languages like JavaScript it might make sense to conceal the source object from the caller by creating a delegate object like asObservable() method does. Although if you think about it it won't prevent a caller from doing observable.source.next(...). So this technique doesn't prevent the Subject API from leaking, but it indeed makes it more hidden form the caller.
On the other hand, for strongly typed languages like TypeScript the method asObservable() doesn't seem to make much sense (if any).
Statically typed languages solve the API leakage problem by simply utilizing the type system (e.g. interfaces). For example, if your getData() method is defined as returning Observable<T> then you can safely return the original Subject, and the caller will get a compilation error if attempting to call getData().next() on it.
Think about this modified example:
let myAPI: { getData: () => Observable<any> }
myAPI = {
getData: () => {
const subject = new Subject()
// ... stuff ...
return subject
}
}
myAPI.getData().next() // <--- error TS2339: Property 'next' does not exist on type 'Observable<any>'
Of course, since it all compiles to JavaScript in the end of the day there might still be cases when you want to create a delegate. But my point is that the room for those cases is much smaller then when using vanilla JavaScript , and probably in majority of cases you don't need that method.
(Typescript Only) Use Types Instead of asObservable()
I like what Alex Vayda is saying about using types instead, so I'm going to add some additional information to clarify.
If you use asObservable(), then you are running the following code.
/**
* Creates a new Observable with this Subject as the source. You can do this
* to create customize Observer-side logic of the Subject and conceal it from
* code that uses the Observable.
* #return {Observable} Observable that the Subject casts to
*/
asObservable(): Observable<T> {
const observable = new Observable<T>();
(<any>observable).source = this;
return observable;
}
This is useful for Javascript, but not needed in Typescript. I'll explain why below.
Example
export class ExampleViewModel {
// I don't want the outside codeworld to be able to set this INPUT
// so I'm going to make it private. This means it's scoped to this class
// and only this class can set it.
private _exampleData = new BehaviorSubject<ExampleData>(null);
// I do however want the outside codeworld to be able to listen to
// this data source as an OUTPUT. Therefore, I make it public so that
// any code that has reference to this class can listen to this data
// source, but can't write to it because of a type cast.
// So I write this
public exampleData$ = this._exampleData as Observable<ExampleData>;
// and not this
// public exampleData$ = this._exampleData.asObservable();
}
Both do the samething, but one doesn't add additional code calls or memory allocation to your program.
❌this._exampleData.asObservable();❌
Requires additional memory allocation and computation at runtime.
✅this._exampleData as Observable<ExampleData>;✅
Handled by the type system and will NOT add additional code or memory allocation at runtime.
Conclusion
If your colleagues try this, referenceToExampleViewModel.exampleData$.next(new ExampleData());, then it will be caught at compile time and the type system won't let them, because exampleData$ has been casted to Observable<ExampleData> and is no longer of type BehaviorSubject<ExampleData>, but they will be able to listen (.subscribe()) to that source of data or extend it (.pipe()).
This is useful when you only want a particular service or class to be setting the source of information. It also helps with separating the input from the output, which makes debugging easier.
I'm trying to understand the proper way to use Windows.Foundation.Diagnostics.LoggingChannel. In particular I'd like to understand the purpose behind the Level property and when is this property set.
As described in the MSDN documentation of LoggingChannel, the Level property is read-only. So how can I set the level that a channel accepts messages at?
Currently what I have designed as a logger for my app is something like below:
public class Logger
{
public LoggingLevel LoggerLoggingLevel { get; set; }
private LoggingSession _session;
private LoggingChannel _channel;
public Logger()
{
_channel = new LoggingChannel("MyChannel");
_session = new LoggingSession("MySession");
_session.AddLoggingChannel(_channel);
}
public void LogMessage(string msg, LoggingLevel level)
{
if (level >= LoggerLoggingLevel)
{
_channel.LogMessage(msg, level);
}
}
.
.
.
}
// The consumer of the Logger class will instantiate an instance of it,
// sets the LoggerLoggingLevel, and then starts logging messages at various levels.
// At any point, the consumer can change LoggerLoggingLevel to start accepting
// messages at different levels.
IS this the right approach or is there a better way (for example by somehow setting the level of _channel and then passing the message & level to the channel, letting the channel decide whether it should filter out the message or accept and log it)?
LoggingChannel.Level tells you "somebody has expressed interest in receiving messages from you that are of severity 'Level' or higher". This property will be set automatically by the runtime when somebody subscribes to events from your LoggingChannel instance. (Within your app, you can subscribe to your app's events using the LoggingSession class; outside of your app, you can record your app's events using a tool like tracelog or xperf.)
In simple scenarios, you don't need to worry about the value of LoggingChannel.Level. The LoggingChannel.LogMessage function already checks the value of LoggingChannel.Level. It also checks the value of LoggingChannel.Enabled, which tells you whether anybody is subscribed to your events at any level. (Note that the value of LoggingChannel.Level is UNDEFINED and MEANINGLESS unless LoggingChannel.Enabled is true.) In normal use, you don't need to worry about LoggingChannel.Enabled or LoggingChannel.Level -- just call LogMessage and let LoggingChannel check the levels for you.
LoggingChannel exposes the Enabled and Level properties to support a more complex scenario where it is expensive to gather the data you are about to log. In this case, you would probably like to skip gathering the data if nobody is listening for your event. You would then write code like this:
if (channel.Enabled && channel.Level <= eventLevel)
{
string expensiveData = GatherExpensiveData();
channel.LogMessage(expensiveData, eventLevel);
}
Note that the Windows 10 version of LoggingChannel added a bunch of new methods to make life a bit easier. If your program will run on Windows 10 or later, you can use the IsEnabled method instead of separate checks for Enabled and Level:
if (channel.IsEnabled(eventLevel))
{
string expensiveData = GatherExpensiveData();
channel.LogMessage(expensiveData, eventLevel);
}
A bunch of other stuff was also added to LoggingChannel for Windows 10. You can now log complex events (strongly-typed fields) instead of just strings, you can define keywords and opcodes (look up ETW documentation for more information), and you can basically have your LoggingChannel act like a first-class ETW citizen.
I am some need help understanding the latest recommended approach to wire up and use reactiveui for a WPF project.
In doing research on the internet on reactiveui I came across various (few) posts spanning a long time period during which the library evolved with the unfortunate result that some of these how-to articles now refer to older ways of doing things which are no longer applicable
I am trying to understand the recommended way to wire up commands (usually to invoke web service which returns a DTO) and I’ve found multiple ways mentioned to do it.
My current understanding is that
// this is the first thing to do
MyCommand = ReactiveCommand.Create()
// variations to wire up the delegates / tasks to be invoked
MyCommand.CreateAsyncTask()
MyCommand.CreateAsyncFunc()
MyCommand.CreateAsyncAction()
// this seems to be only way to wire handler for receiving result
MyCommand.Subscribe
// not sure if these below are obsolete?
MyCommand.ExecuteAsync
MyCommand.RegisterAsyncTask()
Could someone try to explain which of these variations is the latest API and which are obsolete, with perhaps a few words about when to use each of them
The changes on the ReactiveCommand API are documented in this blog post:
http://log.paulbetts.org/whats-new-in-reactiveui-6-reactivecommandt/
The first option - ReactiveCommand.Create() - just creates a reactive command.
To define a command which asynchronously returns data from a service you would use :
MyCommand = ReactiveCommand.CreateAsyncTask(
canExec, // optional
async _ => await api.LoadSomeData(...));
You may use the Subscribe method to handle data when it is received:
this.Data = new ReactiveList<SomeDTO>();
MyCommand.Subscribe(items =>
{
this.Data.Clear();
foreach (var item in items)
this.Data.Add(item);
}
Though, the simplest thing is to use instead the ToProperty method like this:
this._data = MyCommand
.Select(items => new ReactiveList<SomeDTO>(items))
.ToProperty(this, x => x.Data);
where you have defined an output property for Data:
private readonly ObservableAsPropertyHelper<ReactiveList<SomeDTO>> _data;
public ReactiveList<SomeDTO> Data
{
get { return _data.Value; }
}
Is there a way to pass data to dependencies registered with either Execution Context Scope or Lifetime Scope in Simple Injector?
One of my dependencies requires a piece of data in order to be constructed in the dependency chain. During HTTP and WCF requests, this data is easy to get to. For HTTP requests, the data is always present in either the query string or as a Request.Form parameter (and thus is available from HttpContext.Current). For WCF requests, the data is always present in the OperationContext.Current.RequestContext.RequestMessage XML, and can be parsed out. I have many command handler implementations that depend on an interface implementation that needs this piece of data, and they work great during HTTP and WCF scoped lifestyles.
Now I would like to be able to execute one or more of these commands using the Task Parallel Library so that it will execute in a separate thread. It is not feasible to move the piece of data out into a configuration file, class, or any other static artifact. It must initially be passed to the application either via HTTP or WCF.
I know how to create a hybrid lifestyle using Simple Injector, and already have one set up as hybrid HTTP / WCF / Execution Context Scope (command interfaces are async, and return Task instead of void). I also know how to create a command handler decorator that will start a new Execution Context Scope when needed. The problem is, I don't know how or where (or if I can) "save" this piece of data so that is is available when the dependency chain needs it to construct one of the dependencies.
Is it possible? If so, how?
Update
Currently, I have an interface called IProvideHostWebUri with two implementations: HttpHostWebUriProvider and WcfHostWebUriProvider. The interface and registration look like this:
public interface IProvideHostWebUri
{
Uri HostWebUri { get; }
}
container.Register<IProvideHostWebUri>(() =>
{
if (HttpContext.Current != null)
return container.GetInstance<HttpHostWebUriProvider>();
if (OperationContext.Current != null)
return container.GetInstance<WcfHostWebUriProvider>();
throw new NotSupportedException(
"The IProvideHostWebUri service is currently only supported for HTTP and WCF requests.");
}, scopedLifestyle); // scopedLifestyle is the hybrid mentioned previously
So ultimately unless I gut this approach, my goal would be to create a third implementation of this interface which would then depend on some kind of context to obtain the Uri (which is just constructed from a string in the other 2 implementations).
#Steven's answer seems to be what I am looking for, but I am not sure how to make the ITenantContext implementation immutable and thread-safe. I don't think it will need to be made disposable, since it just contains a Uri value.
So what you are basically saying is that:
You have an initial request that contains some contextual information captured in the request 'header'.
During this request you want to kick off a background operation (on a different thread).
The contextual information from the initial request should stay available when running in the background thread.
The short answer is that Simple Injector does not contain anything that allows you to do so. The solution is in creating a piece of infrastructure that allows moving this contextual information along.
Say for instance you are processing command handlers (wild guess here ;-)), you can specify a decorator as follows:
public class BackgroundProcessingCommandHandlerDecorator<T> : ICommandHandler<T>
{
private readonly ITenantContext tenantContext;
private readonly Container container;
private readonly Func<ICommandHandler<T>> decorateeFactory;
public BackgroundProcessingCommandHandlerDecorator(ITenantContext tenantContext,
Container container, Func<ICommandHandler<T>> decorateeFactory) {
this.tenantContext = tenantContext;
this.container = container;
this.decorateeFactory = decorateeFactory;
}
public void Handle(T command) {
// Capture the contextual info in a local variable
// NOTE: This object must be immutable and thread-safe.
var tenant = this.tenantContext.CurrentTenant;
// Kick off a new background operation
Task.Factory.StartNew(() => {
using (container.BeginExecutionContextScope()) {
// Load a service that allows setting contextual information
var context = this.container.GetInstance<ITenantContextApplier>();
// Set the context for this thread, before resolving the handler
context.SetCurrentTenant(tenant);
// Resolve the handler
var decoratee = this.decorateeFactory.Invoke();
// And execute it.
decoratee.Handle(command);
}
});
}
}
Note that in the example I make use of an imaginary ITenantContext abstraction, assuming that you need to supply the commands with information about the current tenant, but any other sort of contextual information will obviously do as well.
The decorator is a small piece of infrastructure that allows you to process commands in the background and it is its responsibility to make sure all the required contextual information is moved to the background thread as well.
To be able to do this, the contextual information is captured and used as a closure in the background thread. I created an extra abstraction for this, namely ITenantContextApplier. Do note that the tenant context implementation can implement both the ITenantContext and the ITenantContextApplier interface. If however you define the ITenantContextApplier in your composition root, it will be impossible for the application to change the context, since it does not have a dependency on ITenantContextApplier.
Here's an example:
// Base library
public interface ITenantContext { }
// Business Layer
public class SomeCommandHandler : ICommandHandler<Some> {
public SomeCommandHandler(ITenantContext context) { ... }
}
// Composition Root
public static class CompositionRoot {
// Make the ITenantContextApplier private so nobody can see it.
// Do note that this is optional; there's no harm in making it public.
private interface ITenantContextApplier {
void SetCurrentTenant(Tenant tenant);
}
private class AspNetTenantContext : ITenantContextApplier, ITenantContext {
// Implement both interfaces
}
private class BackgroundProcessingCommandHandlerDecorator<T> { ... }
public static Container Bootstrap(Container container) {
container.RegisterPerWebRequest<ITenantContext, AspNetTenantContext>();
container.Register<ITenantContextApplier>(() =>
container.GetInstance<ITenantContext>() as ITenantContextApplier);
container.RegisterDecorator(typeof(ICommandHandler<>),
typeof(BackgroundProcessingCommandHandlerDecorator<>));
}
}
A different approach would be to just make the complete ITenantContext available to the background thread, but to be able to pull this off, you need to make sure that:
The implementation is immutable and thus thread-safe.
The implementation doesn't require disposing, because it will typically be disposed when the original request ends.
I'm using Linq and having trouble doing something that I believe should be trivial. I want to return data from one layer so it can be used independently of linq in another layer.
Suppose I have a Data Access Layer. It knows about the entity framework and how to interact with it. But, it doesn't care who accesses it. The one interesting requirement I have is that the queries in the entity framework return projected data that is not part of the Entity Model itself. Please don't ask me to change this part of the requirement and make POCOs for each return type, as it is not the best design given the problem I am trying to solve. Below is an example.
public class ChartData
{
public function <<returnType??>> GetData()
{
MyEntities context = new MyEntities();
var results = from context.vManyColumnsOfData as v
where v.CompanyName = "acme"
select new {Year = v.SalesYear, Income = v.Income};
return ??;
}
}
Then, I would like to have an ASP.Net UI layer be able to call into the Data Access Layer to get the data in order to bind it to a control. The UI layer should have no notion of where the data came from. It should only know that it has the data it needs to bind. Below is an example.
protected void chart_Load(object sender, EventArgs e)
{
// set some chart properties
chart.Skin = "Default";
...
// Set the data source
ChartData dataMgr = new ChartData();
<<returnType?>> data = dataMgr.GetData();
chart.DataSource = data;
chart.DataBind();
}
What is the best way to send linq projected data back to another layer?
If you don't need to use the projected type statically, just return IEnumerable<object>.
Please don't ask me to change this part of the requirement and make
POCOs for each return type, as it is not the best design given the
problem I am trying to solve.
I feel like I should rightly ignore this, as the best thing to do is to return a defined type. Anonymous types are useful when they are wholly contained within the method that creates them. Once you start passing them around, it is time to go ahead and give them the proper class treatment.
However, to live within your imposed limitations, you can return IEnumerable<object> from the method and use that or var at the callsite and rely upon the dynamic binding of the control to get at the data. It's not going to help you if you need to deal with the object programmatically, but it will serve fine for databinding.
You can not return an anonymous type, so basically for this you will need POCO's even though you don't want them.
"not the best design given the problem I am trying to solve"
Could you explain what you are trying to achieve a little more? It might be possible to return some type of list containing a dictionary of items (ie rows and columns). Think something like an untyped dataset (yuck)
Your GetData method can use IEnumerable (the "old" non-generic interface) as its return type.
Any dynamic resolution (e.g. ASP.NET or XAML bindings) should work as expected, which seems to be what you want to do.
However, if you want to use the results in your code, you will probably have to resort to .NET 4's dynamic keyword.
The following example can be run in LINQPad (in "C# Program" mode) and illustrates this:
void Main()
{
var v = GetData();
foreach (dynamic element in v)
{
((string)element.Name).Dump();
}
}
public IEnumerable GetData()
{
return from i in Enumerable.Range(1, 10)
select new
{
Name = "Item " + i,
Value = i
};
}
Keep in mind that, design-wise, coding like this will make most people frown and can affect performance.