I'm building an OData service with Web API. Database access is handled via Entity Framework.
I am mapping my EF entities to another set of classes using AutoMapper. Those mapped classes are what is being exposed via the OData service.
I have a model that looks like this:
Assignment(int AssignmentId, int EmployeeId, Employee Employee)
Employee(int EmployeeId, ICollection<Skill> Skills)
Skill(int SkillId, string SkillName)
My OData endpoint exposes an IQueryable<Assignment>. Everything works fine with simple OData queries ($top, $select, $filter, etc). By default, Assignment.Employee is not returned from the service; I am fine with that.
When I attempt to $expand the Employee, I get this error:
{"error":{"code":"","message":"An error has occurred.","innererror":{"message":"Cannot compare 'member 'Skills' of type '[...].Employee''. Only primitive types, enumeration types and entity types are supported.","type":"System.NotSupportedException"[...]}
[...] are not actually part of the error, just pieces of the message I've removed.
At this point I have not requested Skills, and my expectation was that Employee.Skills would not be expanded because I haven't explicitly requested to expand it. I am unsure what Skills is being compared to that is throwing EF off.
The best I can tell is that OData is applying an additional projection on top of any projections I have applied, and it is that projection that EF is having issues with.
Has anyone had experience/success with using OData/EF to navigate properties that had more than one level of depth to it?
I have tried removing AutoMapper and writing the expressions by hand, but I still run into the same error, so I don't believe AutoMapper is causing any issues here.
Related
How can I apply a filter accent-insensitive? In OData the "eq" operator is case and accent sensitive. The case is easy to fix, because the "tolower" but relative to the accent I'm not getting a simple solution. I know contains is supposed to be accent-insensitive but if I use contains filtering by "São José" I am only getting these responses "São José" and "São José dos Campos", it is missing "Sao Jose".
The following example filtering by "Florianopolis" is expected to return "Florianópolis", but it does not:
url: api/cidades/get?$filter=contains(nome, 'Florianopolis')
[HttpGet]
[EnableQuery]
public ActionResult<IQueryable<CidadeDTO>> Get()
{
try
{
return Ok(_mapper.Map<IEnumerable<CidadeDTO>>(_db.Cidades));
}
catch (System.Exception e)
{
return BadRequest(e.GetBaseException().Message);
}
}
It should bring aswell, like entity framework.
If your OData model was mapped directly to EF models AND an IQueryable<T> expression was passed into OK() then the query is explicitly passed through to the database engine as SQL:
SELECT * FROM Cidades WHERE nome LIKE '%Florianopolis%'
When that occurs, the Collation settings in the database connection will determine the comparison matching logic.
When your database collation is case and accent insensitive, but your data is still filtered as if it was not, then this is an indication that an IEnumerable<T> has been passed into OK() and the comparison logic is being evaluated in C# which by default in insensitive to both case and accent. Unfortunately this means that it is very likely that the entire data table has been loaded into memory first so that the filter can be applied.
In your case the OData model is mapped to DTO expressions that are mapped to the EF models via AutoMapper and that is where the generated query can break down. By calling Map() you are loading ALL records from the EF table and leaving the $filter criteria to be applied by the EnableQueryAttribute
For OData query conventions to be applied automatically you must return an IQueryable<T> from your method, or atleast pass an IQueryable<T> into the OK() response handler. With AutoMapper, you can use the Queryable Extensions to satisfy the IQueryable<T> requirement:
Queryable Extensions
When using an ORM such as NHibernate or Entity Framework with AutoMapper’s standard mapper.Map functions, you may notice that the ORM will query all the fields of all the objects within a graph when AutoMapper is attempting to map the results to a destination type.
...
ProjectTo must be the last call in the chain. ORMs work with entities, not DTOs. So apply any filtering and sorting on entities and, as the last step, project to DTOs.
In OData the last requirement (about ProjectTo) is still problematic because the EnableQueryAttribute will append the query options to the IQueryable<T> response, which will still end up materializing the entire table into memory first (IEnumerable<T>) and then apply the filter, which is still incredibly inefficient. It is this behaviour that is generally observed when someone complains about poor performance from an OData implementation, it is not always AutoMapper, but usually the pattern that the data source is loaded into memory in its entirety and then filtered. Following the default guidance for AutoMapper will lead you in this direction.
Instead we need to use an additional package: AutoMapper.Extensions.ExpressionMapping that will give us access to the UseAsDataSource extension method.
UseAsDataSource
Mapping expressions to one another is a tedious and produces long ugly code.
UseAsDataSource().For<DTO>() makes this translation clean by not having to explicitly map expressions. It also calls ProjectTo<TDO>() for you as well, where applicable.
This changes your implementation to the following:
[HttpGet]
[EnableQuery]
public ActionResult<IQueryable<CidadeDTO>> Get()
{
return Ok(_db.Cidades.UseAsDataSource().For<CidadeDTO>());
}
Don't fall into the trap of assuming that AutoMapper is necessary or best practice for an OData API implementation. If you are not using the unique features that AutoMapper provides then adding an additional abstraction layer can end up over-complicating your solution.
I'm not against AutoMapper, I use it a lot for Integrations, ETL, GraphQL and non-DDD style data schemas where the DTO models are significantly different to the underlying EF/data storage models. But it is a maintenance and performance overhead that a simple DDD data model and OData API based solution can easily do without.
Don't hire an excavator when a simple shovel will do the job.
AutoMapper is a convention based ORM that can be useful when you want to change the structure between implementation layers in your code, traditionally you might map Business Domain models that may represent aggregates or have flattened structures to highly normalised Database models.
OData is also a convention based ORM. It was designed to facilitate many of the same operations that AutoAmpper provides with the exception of Flattening and Unflattening models. These operations are deferred to the EF engine. The types exposed via OData mapping are DTOs
If your DTO models are the same relational structure as your EF models, then you would generally not use AutoMapper at all, the OData Edm mapping is optimised specifically to manage this type of workload and is designed to be and has been integrated directly into the serialization layer, making the Edm truely Data Transfer Objects that only exist over the wire and in the client.
This did the job
[HttpGet]
public ActionResult<IQueryable<PessoaDTO>> Get(ODataQueryOptions<Pessoa> options)
{
try
{
var queryResult = options.ApplyTo(_db.Pessoas);
return Ok(queryResult);
}
catch (System.Exception e)
{
return BadRequest(e.GetBaseException().Message);
}
}
A little background of my problem. I have a set of the following services:
AdapterService - intended for loading certain products from an external system
ApiGateway - accepts requests from UI. In particular, now there is only one request that receives product data to display product in UI from Product Service
ProductService - data storage service for various products. The service itself does not specifically know what kind of product it specifically stores. All types of products are created dynamically by other services that are responsible for these products. Product data is stored as a key-value map (technically it is a json string in DB column)
There is a schema for service interations
So, services in BLUE zone are mine (they can be changed in any way). RED zone describes services of another team (they can't be changed).
Whats the problem
To load product from external system I want to use SpecialProductDto which will store product data. I can use some validation features like Spring annotations and so on. Then to load the product from Adapter Service to ProductService I must transform SpecialProductDto to Map<String, Object> because ProductSerivcie requires it via API.
When I would get product info for UI through ApiGateway, I will need to call ProductService api for getting product that return attribues in Map<String, Object> and then transform this data to some UIReponse which contains some part of product data (because I dont need all product information, just only name and price for example).
But I also want to use SpecialProductDto in my ApiGateway service, because it seems working with Map<String, Object> is error prone... I practically need to fetch data blindly from Map to construct UIResponse. And what if some attribute names will be changed? With Map I only will know it when the request would be made from UI but using special DTO I get such exception in compilation time.
Question
So, what is the best practiсe or maybe patterт should I use in such situation? At the moment I see the following solutions:
Duplicate DTOs in both AdapterService and ApiGateway services. So, any changes in one class must be supported in another
Use Map<String, Object> at my own peril and risk, hoping that nothing will change there
Share SpecialProductDTO between ApiGateway and AdapterSerivce in some separate library and service (seems to be antipattern because of sharing someting can make a lot of problems)
Сan anyone help?
In my opinion, there's nothing wrong on duplicating DTOs.
Also, there's nothing wrong on providing the DTO in a separate library to be imported on each project, you will only be sharing the ProductService's contract and that's it. It does not cause any tight coupling between the Api Gateway and the Adapter. If the contract changes, then it must be changed on all of it's consumers (api gateway and adapter), simple as that.
About using Maps: usually I don't recommend this, because, like you said, you will not be able to take advantages of the built-in Bean Validations that Spring (and other frameworks) provides, but not only that, you'll also, depending on the situation, be using lots of casts and type conversions, which is not good and can be prevented by using DTOs.
Also, be aware that a DTO, in my opinion, should not be named with the suffix of 'DTO'. That's because a name like SpecialProductDTO doesn't clearly states where this object is being used or should be used.
Instead, prefer a something like CreateSpecialProductRequest - this indicates that this object is used when creating a Special Product. Another example is CreateSpecialProductResponse which just represents the response (if needed) after a Special Product creation. Take a look at this StackOverflow answer: Java data transfer object naming convention?
I’m trying to go against a Web API Patch method that accepts a Delta. My entity has a navigational property that is a collection. The domain model is trivial, I have a team (entity) and a collection of members (navigational property).
I’m using HttpClient directly instead of the OData client.
The problem I’m running into is that when I send a patch request with the payload = a Team, the Delta deserialized in my controller is null when I include the navigational property collection in my payload.
Eg (ignore that the missing quotes, this is typed in}:
{ Name: Foo } -> serializes to Delta successfully.
{Name: Foo, Members : [] } -> fails to serialize and Delta is null.
Is this supported? I could not find concrete documentation online about whether you can supply navigational properties as an entire collection on patch (not looking for merge support, just want full replace of the collection.)
I also tried tweaking my client to directly send a Delta payload, but the default JsonMediaTypeFormatter is unable to serialize this into a valid request body (always produces empty Json), and the ODataMediaTypeFormatter throws an exception saying it cannot write an object of type Delta, although I have initialized it with every ODataPayloadKind.
I’m sure I’m either doing something wrong or missing something basic, assuming using Delta + patch with HttpClient is not this challenging.
For OData spec, it says:
11.4.3 Update an Entity
...
The entity MUST NOT contain related entities as inline content. It MAY contain binding information for navigation properties. For single-valued navigation properties this replaces the relationship. For collection-valued navigation properties this adds to the relationship.
That is you can't put the navigation properties in the Patch request content.
From Web API OData, it has these codes and these codes. It means that it doesn't allow to patch navigation property form entity.
However, you can shoot your target by following steps:
Patch the principal entity (team)
Patch the dependent entities (members)
use the $ref to update the link
You can refer to my simple test project.
I like OData and I was particularly pleased to its adoption by the ASP.NET Web API.
I've created a few services for internal applications to consume, but never for public consumption. The primary reason is that the open nature of OData seems to make it very hard to make "safe" against abuse.
Most specifically, I'm worried that given the power to run arbitrary queries, a user could express a complex query which stresses the operational system to the point where the experience is bad for all other users.
In a WebApi controller, an OData endpoint is exposed as follows:
public class OrderController
{
[Queryable]
public IQueryable<Orders> Get()
{
// Compose and return the IQueryable<Orders>
}
}
This gives full control over the process of composition and execution of the query, but does so though the complex IQuerable<T> interface. It makes it trivial to give the user a subset of the information, e.g. append a Where to only include the records they have permission to access.
Is there an IQueryable<T> implementation that can wrap an existing IQuerable<T> instance to provide restrictions on the queries a user can run? I'm most interested in restricting the complexity of the query, but I also want to be able to prevent a user traversing associations to resources they shouldn't have access to.
I think you'll be glad to learn that in RTM, we've added options to let you customize what kind of querying you want to expose to users. So you can do this for example:
[Queryable(
AllowedFunctions = AllowedFunctions.AllStringFunctions,
AllowedLogicalOperators = AllowedLogicalOperators.Equal,
AllowedOrderByProperties = "ID")]
and restrict your query in a few common ways. If you want to restrict your query even further, there are validation hooks you can plug into, like ODataQueryValidator or by overriding the ValidateQuery method on the [Queryable] attribute.
You can use our nightly builds to get access to these features, or build the latest bits yourself.
Instead of using the Queryable attribute (which you are missing), you can not use this attribute and instead manually accept the ODataQueryOptions parameter, which gives you access to the various filter, top, etc options to permit validation of them.
Any function being called by a OData query can be passed an argument ODataQueryOptions like:
public IQueryable<T> Get(ODataQueryOptions options)`
{
//Use the following vars to fetch the values, and check if
//they are as you expect them to be. etc.
options.Top.RawValue;
options.Filter.Value;
options.Filter.ApplyTo();
}
In this case you can skip the [Queryable] attribute and use ApplyTo to manually apply the various queries over the result. :)
I'm working on a CRM-style MVC web app, which has the following (simplified) schema:
Contacts Table
ContactId
Forename
Surname
Etc.
Tags Table
TagId
Value
ContactTags Table
ContactId
TagId
I've then generated POCO objects from the *.edmx file, interacting with a DbContext-based entities context, hiding the ContactTags table so that the relationship between Contact and Tag entities is modelled as a many-to-many association. I've then restricted access to the raw Contact.Tags navigation property, setting it to be internal rather than public, and exposed a ReadOnlyCollection that can be used outside the Domain layer for displaying tags, but restricting data operations on the collection to a Contact.EditTags() method.
Upon writing UI code to display the list of tags on a contact, I found that the Tags navigation property isn't being lazy-loaded. After scratching my head and googling about a bit, I found another question at EF CTP4 lazy loading not playing ball which matched my problem. The author of the question found that when he changed the internal property to be public, it started working, and sure enough this is what happened for me too - I've changed the Tags navigation property to be public, and it's now working.
I'm uncomfortable with this from an object modelling/data encapsulation point of view, since the UI shouldn't be given access to the raw Tags collection which would enable controller code to call Tags.Add(), Tags.Remove, etc.
Does anyone know if this is a bug, or a deliberate design decision by the EF team? Is it possible to get an internal navigation property to be lazy-loaded? I know we could eager-load, but we'd like to avoid that if at all possible.
Lazy loading on POCOs needs creation of POCO proxies. Proxies are only created when the model classes fulfill certain requirements. One of these requirements for proxies which enable lazy loading is this one:
Each navigation property must be
declared as public, virtual
(Overridable in Visual Basic), and not
sealed (NotOverridable in Visual
Basic) get accessor.
Quote from here: http://msdn.microsoft.com/en-us/library/dd468057.aspx