Microsoft Cognitive Services offers a wide variety of capabilities to extract information from natural language. However I am not able to find how to use them in order to detect "relationships" where e.g. two (or more) specific "entities" are involved.
For example, detecting company acquisitions / merging.
These could be expressed in News articles as
"Company 1" has announced to acquire "Company2".
Certainly, there are several approaches to address that need, some that include entity detection first (e.g. Company1 and Company2 being companies) and then the relation (e.g. acquire ...).
Other approaches involve identifying first the "action" ( acquire ) and then through grammatical analysis find which is the "actor" and which the "object" of the action.
Machine learning approaches for semantic relation extraction has also been developed, in order to avoid humans to craft formal relation rules.
I would like to know if / how this use case can be performed with the Microsoft Cognitive Services.
Thankyou
Depends on tech used to examine response from the API https://dev.projectoxford.ai/docs/services
I use JQuery to parse the json response (webclient in asp.net code behind) from Luis/Cognitive Services API (I am not using the Bot Framework). I have a rules engine that I can configure for clients and save it, so that when the page loads, they fire functions based on the parsed json response. The rules engine includes various condition functions like contains, begins with, is, etc so I can test the users query for specific entities or virtually anything in the users query. It really comes down to a && or || javascript functions...
For example if intent=product in the json response, I then show a shopping cart widget. Or if entity=coffee black OR entity=double double then it triggers a widget to inject into the chat window (SHOW Shopping Cart). In short you either handle the AND/OR via the Bot Framework or via your tech of choice.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 15 days ago.
Improve this question
I have a monolith application that uses one database, and in my company, we decide to rewrite the application and use microservices in the backend.
At this time, we decided NOT to split the database because other applications and processes are using it, and it takes two years to change.
The difficulty in the process is to decompose and identify the right microservices.
I'll try to explain our system by start describing the UI. Please read carefully because I am trying to explain it in detail.
The system displays the stock market data. The Company or Fund or Fund manager in the market is posting everyday reports about the company's activities like status, information for investors, and more.
"breaking announcement" page
displays a list of today's priority reports. Each row contains the subject from the pdf document (the report) that the company is publishing and the company that belongs to the report:
When the user clicks on the row, we redirect to "report page" and which contains the report details:
In the database, we have entities such report, company, company_report, event, public_offers, upcoming_offering, and more.
So to get the list, we run an inner join query like this:
Select ... From report r inner join
company_report cr on r.reportid=cr.reportid
inner join company c on cr.company_cd=c.company_cd
Where ....
Most of our server endpoints are not changing anything but are only used to retrieve the data.
So I'll create this endpoint /reports/breaking-announcement to get the list, and it returns an object like that:
[{ reportId, subject, createAt, updateAt, pdfUrl, company: { id, name } }]
today's companies report page acts like "breaking announcement" page. but the page displays all the reports from today (not necessarily with priority).
disclosures are reports
On this page, we also have a search to get all reports by cretiria for example to get reports by company name. to do that we have autocomplete so the user types the company name or id.
In other to do that we think it should be API endpoint /companies/by-autocomplete and the response will [{ companyId, companyName, isCompany }].
eft page same as before, but this time we display the Funds report's (not a companys reports).
The list contained the fund name and the subject of the report. each click on the row leads to the report detail page (same page).
On this page we have a search by criteria such date-from date-to, name or id of the funds by autocomplete. and endpoint (/funds/by-autocomplete returns [{ fundId, fundName, ...}]).
foreign etf page same as before, list of items. each item is like before:
<fund name>
<subject of the report>
The query is different.
Okay, this was a very long description. thank you for reading.
Now I want to detect what are the microservices for this application.
I endup with:
Report microservice - responsible for getting and handling all the reports in the system.
which have a endpoints like getall, getbyid, get like getbreakingannouncement, getcompanytodayreports, getfunds, getforeignfunds. the report microservice will make a request to company or funds microservice to join the data from the company and build to the response.
company microservice:
handle all companies data. I mean endpoints such getall, getByIds (for report service), getByAutocomplete.
funds microservice:
handle all funds data. I mean endpoints such getall, getByIds (for report service), getByAutocomplete.
There are other services, such as a notification service or email service. but those are not business services. I want to split up my business logic into microservices in order to deploy and maintain them easily.
I'm not sure I decomposing right. maybe I do. but is fit the microservice ideas? it's fit the Pattern: Decompose by business capability
? if not what are the business capability in my system?
I don't think a query-oriented decomposition of your current application monolith will lead to a good microservice (MS) design. Two of your proposed microserivces have the same end-point query API which suggests to me that you are viewing your first-generation microservices as just entity-servers.
Your idea to perform joins on cross MS query operations indicates these first gen "microservices" are closely coupled and hence fall short of a genuine MS architecture.
One technique to verify an MS design is to ask yourself, "how would the whole system cope if one MS is unavailable for 3 minutes?". Solving that design challenge leads down a path towards decoupled message-base interactions between the microservices. And this in turn leads to interactions between Microservices being expressed as business operations where one MS raises messages that trigger a mutation in the state of another MS.
Maybe you should reduce the scope of your MS ambitions and instead look at Schema Stitching in GraphQL. Reading between the lines of your question I think a more realistic first step towards a distributed system would be to create specialised query services with a GraphQL endpoint.
At this time, we decided NOT to split the database because other applications and processes are using it, and it takes two years to change.
I'll try to stop you right here. In general case shared database is a huge antipattern in microservices architecture and should be avoided as much as possible. There are multiple problems here - less transparent dependencies between services which can cause high coupling with all the consequences in development and deployment, increasing chance to eventually end up with distributed monolith instead of microservices, etc.
Other applications and processes using it should not stop you from moving away from it - there are things which allow to mitigate that - you just sync data between services and "legacy" database (asynchronously using basically the same approaches like you will use in your microservices - for example transaction log tailing for example using something like debezium). It have it's own costs but I would argue that it is usually better to pay them upfront then have to pay bigger percentages on the tech debt.
I endup with: ....
I would argue that this split looks more like decomposition by subdomain then by business capability. Which is actually can be quite fine and suits microservices architecture also.
Based on your description I see at least the following business capabilities in your system that can be defined:
View (manage?) breaking announcements
View (manage?) reports
Search (reports?)
Potentially "today's reports" and "Funds reports" can be considered as separate business capabilities.
I want to split up my business logic into microservices in order to deploy and maintain them easily.
Then again - I highly recommend to reconsider not moving away from shared database.
I'm not sure I decomposing right
Without whole overview of the system including amount of data, data flows, resources available for development and competences in the teams, amount of incoming new business requirements, potential vectors of change, etc. it is hard to actually tell.
P.S.
Note that despite the microservices architecture having a lot of popularity it is not always a right solution for a concrete project to go full-blown microservices. If you have quite small team and/or do not handle high loads/large amount of data with various access patterns then potentially you do not need microservices. You still can leverage a lot of approaches used in the microservices architecture though.
I am learning to develop microservices using DDD, CQRS, and ES. It is HTTP RESTful service. The microservices is about online shop. There are several domains like products, orders, suppliers, customers, and so on. The domains built in separate services. How to do the validation if the command payload relates to other domains?
For example, here is the addOrderItemCommand payload in the order service (command-side).
{
"customerId": "CUST111",
"productId": "SKU222",
"orderId":"SO333"
}
How to validate the command above? How to know that the customer is really exists in database (query-side customer service) and still active? How to know that the product is exists in database and the status of the product is published? How to know whether the customer eligible to get the promo price from the related product?
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation. Because, we have developed an event processor outside the command-service that listen from the event and apply the event to the materalized view.
Thank you.
As there are more than one bounded contexts that need to be queried for the validation to pass you need to consider eventual consistency. That being said, there is always a chance that the process as a whole can be in an invalid state for a "small" amount of time. For example, the user could be deactivated after the command is accepted and before the order is shipped. An online shop is a complex system and exceptions could appear in any of its subsystems. However, being implemented as an event-driven system helps; every time the ordering process enters an invalid state you can take compensatory actions/commands. For example, if the user is deactivated in the meantime you can cancel all its standing orders, release the reserved products, announce the potential customers that have those products in the wishlist that they are not available and so on.
There are many kinds of validation in DDD but I follow the general rule that the validation should be done as early as possible but without compromising data consistency. So, in order to be early you could query the readmodel to reject the commands that couldn't possible be valid and in order for the system to be consistent you need to make another check just before the order is shipped.
Now let's talk about your specific questions:
How to know that the customer is really exists in database (query-side customer service) and still active?
You can query the readmodel to verify that the user exists and it is still active. You should do this as a command that comes from an invalid user is a strong indication of some kind of attack and you don't want those kind of commands passing through your system. However, even if a command passes this check, it does not necessarily mean that the order will be shipped as other exceptions could be raised in between.
How to know that the product is exists in database and the status of the product is published?
Again, you can query the readmodel in order to notify the user that the product is not available at the moment. Or, depending on your business, you could allow the command to pass if you know that those products will be available in less than 24 hours based on some previous statistics (for example you know that TV sets arrive daily in your stock). Or you could let the customer choose whether it waits or not. In this case, if the products are not in stock at the final phase of the ordering (the shipping) you notify the customer that the products are not in stock anymore.
How to know whether the customer eligible to get the promo price from the related product?
You will probably have to query another bounded context like Promotions BC to check this. This depends on how promotions are validated/used.
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation.
This depends on how resilient you want your system to be and how fast you want to reject invalid commands.
Synchronous call are simpler to implement but they lead to a less resilient system (you should be aware of cascade failures and use technics like circuit breaker to stop them).
Asynchronous (i.e. using events) calls are harder to implement but make you system more resilient. In order to have async calls, the ordering system can subscribe to other systems for events and maintain a private state that can be queried for validation purposes as the commands arrive. In this way, the ordering system continues to work even of the link to inventory or customer management systems are down.
In any case, it really depends on your business and none of us can tell you exaclty what to do.
As always everything depends on the specifics of the domain but as a general principle cross domain validation should be done via the read model.
In this case, I would maintain a read model within each microservice for use in validation. Of course, that brings with it the question of eventual consistency.
How you handle that should come from your understanding of the domain. Factors such as the length of the eventual consistency compared to the frequency of updates should be considered. The cost of getting it wrong for the business compared to the cost of development to minimise the problem. In many cases, just recording the fact there has been a problem is more than adequate for the business.
I have a blog post dedicated to validation which you can find here: How To Validate Commands in a CQRS Application
I am trying to understand how to use the FHIR Questionnaire resource, and have a specific question regarding this.
My project is specifically regarding how a citizen in our country could be responding to Questionnaires via a web app, which are then submitted to the FHIR server as QuestionnaireAnswers, to be read/analyzed by a health professional.
A FHIR-based system could have lots of Questionnaires (Qs), groups of Qs or even specific Qs would be targeted towards certain users or groups of users. The display of the questionnare to the citizen could also be based on a Care-plan of a sort, for example certain Questionnaires needing filling-in in the weeks after surgery. The Questionnaires could also be regular ones that need to be filled in every day or week permanently, to support data collection on the state of a chronic disease.
What I'm wondering is if FHIR has a resource which fits into organizing the 'logistics' of displaying the right form to the right person. I can see CarePlan, which seems to partly fit. Or is this something that would typically be handled out-of-FHIR-scope by specific server implementations?
So, to summarize:
Which resource or mechanism would a health professional use to set up that a patient should answer certain Questionnaires, either regularly or as part of for example a follow-up after a surgery. So this would include setting up the schedule for the form(s) to be filled in, and possibly configure what would happen if the form wasn't filled in as required.
Which resource (possibly the same) or mechanism would be used for the patient's web app to retrieve the relevant Questionnaire(s) at a given point in time?
At the moment, the best resource for saying "please capture data of type X on schedule Y" would be DiagnosticOrder, though the description probably doesn't make that clear. (If you'd be willing to click the "Propose a change" link and submit a change request for us to clarify, that'd be great.) If you wanted to order multiple questionnaires, then CarePlan would be a way to group that.
The process of taking a complex schedule (or set of schedules) and turning that into a simple list of "do this now" requests that might be more suitable for a mobile application to deal with is scheduled for DSTU 2.1. Until then, you have a few options for the mobile app:
- have it look at the CarePlan and complex DiagnosticOrder schedule and figure things out itself
- have a server generate a List of mini 1-time DiagnosticOrders and/or Orders identifying the specific "answer" times
- roll your own mechanism using the Other/Basic resource
Depending on your timelines, you might want to stay tuned to discussions by the Patient Care and Orders and Observations work groups as they start dealing with the issues around workflow management starting next month in Atlanta.
This is more of an architectural question. Let's assume we have different types of users logging into a system, and we have a 'customer' entity. Depending on the permissions of the user, I may want to return different sub-sets of 'customer' properties. I also might want to allow edits to only certain properties.
Any suggestions on what path to go down? Here are the options I've thought of thusfar:
For each permission level, extend the model - and return the furthest descendant based upon the user permissions. On the input side, accept the furthest descendant and only cast it to the correct descendant. (Seems like a lot of implicit logic - doesn't seem very elegant)
Create different methods (cluttered API, implies more functionality than I might want to expose)
Any other suggestions?
Thanks
What you're describing is a clear-cut use case for XACML. XACML is the eXtensible Access Control Markup Language. It lets you define fine-grained access control using attributes (about the user, the resource, the environment...).
It's policy-based which means you can write things like:
users can view customer records that are in the same region as the user
users can edit customer records they are directly assigned to
auditors can view customer records for the entire business unit except for sensitive fields
There are several XACML engines out there (WSO2, Heras AF for Java; Axiomatics for .NET).
I've developed quite a few ASP .NET web apps in the .NET 4.0 framework and managed to apply authorization at the presentation tier, the WCF tier, and the data tier. Feel free to ping me for additional information.
I'm doing a fairly complex emberjs application, and tying it to a backend of APIs.
The API calls are not usually tied to any particular model, but may return objects of various types in different sections of the response, e.g. a call to Events API would return events, but also return media assets and individuals involved in those events.
I've just started with the project, and I'd like to get some expert guidance on how best to separate concerns to have a clean maintainable code base.
The way I am approaching this is:
Models: essentially handle records with their fields, and other computed properties. However, models are not responsible for making requests.
e.g. Individual, Event, Picture, Post etc.
Stores: They are essentially caches. For example, an eventStore would store all events received from the server so far (from possibly different requests) in an array, and also in an hash of events indexed by id.
e.g. individualStore, eventStore etc.
Controllers: They tie to a set of related API calls, e.g. eventsController would be responsible for fetching events or a particular event, or creating a new event etc. They would 'route' the response to different stores for later retrieval. They don't keep the response once it has been sent to stores.
e.g. eventsController, userSearchController etc.
Views: They are tied to a particular view. In general, my application may have several views at different places, e.g. latestEventsView on the Dashboard in addition to having a separate events page.
Templates: are what they are.
Quite often, my templates require to be bound directly to the stores (e.g. peopleView wants to list all the individuals in the individualStore in a list, sorted by some order).
And sometimes, they bind to a computed property
alivePeople: function () { ... }.property('App.individualStore.content.#each'),
The various filtering and sorting options 'chosen' in the view, should return different lists from the store. You can see my last question at what is the right emberjs way to switch between various filtering options?
Who should do this filtering, the view themselves or the stores?
Is this kind of binding across layers okay, or a code smell? Is the separation of concerns good, or am I missing something? Shouldn't controllers be doing something more here? Should my views directly bind to stores?
Any particular special case of MVC more suited to my needs?
Update 17 April 2012
My research goes on, particularly from http://vimeo.com/user7276077/videos and http://jzajpt.github.com/2012/01/17/emberjs-app-architecture.html and http://jzajpt.github.com/2012/01/24/emberjs-app-architecture-data.html
Some issues with my design that I've figured out are:
controllers making requests (stores or models or something else should do it, not controllers)
statecharts are missing -- they are important for view-controller interactions (after sometime you realize your interactions are no more simple)
This is a good example of state charts in action: https://github.com/DominikGuzei/ember-routing-statechart-example
UPDATE 9th JANUARY 2013
Yes, it's been long but this question is lately getting lots of views, and that's why I'd like to edit it so that people may get a sense.
Ember's landscape has changed a lot since this question was framed, and the new guides are much improved. EmberJS has come up with conventions (like Rails) and the MVC is much more well defined now.
Anybody still confused should read all the guides, and watch some videos:
Seattle Ember.js Meetup
At the moment, I'm upgrading my application to Ember.js 1.0.0-pre2.
You should think of your application in terms of states. Have a look at this
Initially, only a route and a template are required to describe
something and finally display it in the browser, that's what the new
API of Emberjs tries to enforce. As your requirements get more
elaborate you can throw in a view, a controller or an object. Each
though answers a specific need.
Consider a view if you need to handle any browser events or wrap
any 3rd party javascript lib you're using for animation, styling ..
Consider an Object if you need to capture domain specific
information, most likely mimics backend information.
A controller is merely a proxy for the domain object and may encapsulate logic that doesn't pertain necessarily to the object.
That's all what's to it. If you learn how to design your application in terms of states, the rest will fall into the right place, providing you're using the latest api, enforcing the rules i mentioned previously.
Since the release of Ember 1.0.0-pre4 with the new router implementation I've seen two good references describing a standardised EmberJS app structure.
Those of you familiar with Rails would find it fairly familiar.
https://github.com/trek/ember-todos-with-build-tools-tests-and-other-modern-conveniences
http://reefpoints.dockyard.com/ember/2013/01/07/building-an-ember-app-with-rails-api-part-1.html
The ember-rails project at https://github.com/emberjs/ember-rails includes a Rails generator for creating an EmberJS application directory structure that is essentially the same as the structure described in the two links above.
The EmberJS guides also now describe the new routing structure. http://emberjs.com/guides/
UPDATE 21/08/2013
If you are using Rails then the ember-rails gem is great. I've used it with a lot of success.
There are two efforts within the ember community to assist in providing a standardised ember application layout. Apparently they are going to be merged but for now check out:
https://github.com/rpflorence/ember-tools
https://github.com/stefanpenner/ember-app-kit
See also this http://addyosmani.com/largescalejavascript/ It is not about EmberJs in particular but it's a great article that gives you an idea how to write larg scale javascript apps.