I'm pulling data from a third party api. The api runs multiple times in a day. So, if the same data is present in the table it should ignore that record, else if there are any changes it should update that record or insert a new record if anything new shows up in the json received.
I'm using the below code for inserting any new data.
var input = JsonConvert.DeserializeObject<List<DeserializeLookup>>(resultJson).ToList();
var entryset = input.Select(y => new Lookup
{
lookupType = "JOBCODE",
code = y.Code,
description = y.Description,
isNew = true,
lastUpdatedDate = DateTime.UtcNow
}).ToList();
await _context.Lookup.AddRangeAsync(entryset);
await _context.SaveChangesAsync();
But, after the first run, when the api runs again it's again inserting the same data in the table. As a result, duplicate entries are getting into table. To handle the same, I used a foreach loop as below before inserting data to the table.
foreach (var item in input)
{
if (!_context.Lookup.Any(r =>
r.code== item.Code))
{
//above insert code
}
}
But, the same doesn't work as expected. Also, the api takes a lot of time to run when I put a foreach loop. Is there a solution to this in .net core 3.1
List<DeserializeLookup> newList=new();
foreach (var item in input)
{
if (!_context.Lookup.Any(r =>
r.code== item.Code))
{
newList.add(item);
//above insert code
}
}
await _context.Lookup.AddRangeAsync(newList);
await _context.SaveChangesAsync();
It will be better if you try this way
I’m on my phone so forgive me for not being able to format the code in my response. The solution to your problem is something I actually just encountered myself while syncing data from an azure function and third party app and into a sql database.
Depending on your table schema, you would need one column with a unique identifier. Make this column a primary key (first step to preventing duplicates). Here’s a resource for that: https://www.w3schools.com/sql/sql_primarykey.ASP
The next step you want to take care of is your stored procedure. You’ll need to perform what’s commonly referred to as an UPSERT. To do this you’ll need to merge a table with the incoming data...on a specified column (whichever is your primary key).
That would look something like this:
MERGE
Table_1 AS T1
USING
Incoming_Data AS source
ON
T1.column1 = source.column1
/// you can use an AND / OR operator in here for matching on additional values or combinations
WHEN MATCHED THEN
UPDATE SET T1.column2= source.column2
//// etc for more columns
WHEN NOT MATCHED THEN
INSERT (column1, column2, column3) VALUES (source.column1, source.column2, source.column3);
First of all, you should decouple the format in which you get your data from your actual data handling. In your case: get rid of the JSon before you actually interpret the data.
Alas, I haven't got a clue what your data represents, so Let's assume your data is a sequence of Customer Orders. When you get new data, you want to Add all new orders, and you want to update changed orders.
So somewhere you have a method with input your json data, and as output a sequence of Orders:
IEnumerable<Order> InterpretJsonData(string jsonData)
{
...
}
You know Json better than I do, besides this conversion is a bit beside your question.
You wrote:
So, if the same data is present in the table it should ignore that record, else if there are any changes it should update that record or insert a new record
You need an Equality Comparer
To detect whether there are Added or Changed Customer Orders, you need something to detect whether Order A equals Order B. There must be at least one unique field by which you can identify an Order, even if all other values are of the Order are changed.
This unique value is usually called the primary key, or the Id. I assume your Orders have an Id.
So if your new Order data contains an Id that was not available before, then you are certain that the Order was Added.
If your new Order data has an Id that was already in previously processed Orders, then you have to check the other values to detect whether it was changed.
For this you need Equality comparers: one that says that two Orders are equal if they have the same Id, and one that says checks all values for equality.
A standard pattern is to derive your comparer from class EqualityComparer<Order>
class OrderComparer : EqualityComparer<Order>
{
public static IEqualityComparer<Order> ByValue = new OrderComparer();
... // TODO implement
}
Fist I'll show you how to use this to detect additions and changes, then I'll show you how to implement it.
Somewhere you have access to the already processed Orders:
IEnumerable<Order> GetProcessedOrders() {...}
var jsondata = FetchNewJsonOrderData();
// convert the jsonData into a sequence of Orders
IEnumerable<Order> orders = this.InterpretJsonData(jsondata);
To detect which Orders are added or changed, you could make a Dictonary of the already Processed orders and check the orders one-by-one if they are changed:
IEqualityComparer<Order> comparer = OrderComparer.ByValue;
Dictionary<int, Order> processedOrders = this.GetProcessedOrders()
.ToDictionary(order => order.Id);
foreach (Order order in Orders)
{
if(processedOrders.TryGetValue(order.Id, out Order originalOrder)
{
// order already existed. Is it changed?
if(!comparer.Equals(order, originalOrder))
{
// unequal!
this.ProcessChangedOrder(order);
// remember the changed values of this Order
processedOrder[order.Id] = Order;
}
// else: no changes, nothing to do
}
else
{
// Added!
this.ProcessAddedOrder(order);
processedOrder.Add(order.Id, order);
}
}
Immediately after Processing the changed / added order, I remember the new value, because the same Order might be changed again.
If you want this in a LINQ fashion, you have to GroupJoin the Orders with the ProcessedOrders, to get "Orders with their zero or more Previously processed Orders" (there will probably be zero or one Previously processed order).
var ordersWithTPreviouslyProcessedOrder = orders.GroupJoin(this.GetProcessedOrders(),
order => order.Id, // from every Order take the Id
processedOrder => processedOrder.Id, // from every previously processed Order take the Id
// parameter resultSelector: from every Order, with its zero or more previously
// processed Orders make one new:
(order, previouslyProcessedOrders) => new
{
Order = order,
ProcessedOrder = previouslyProcessedOrders.FirstOrDefault(),
})
.ToList();
I use GroupJoin instead of Join, because this way I also get the "Orders that have no previously processed orders" (= new orders). If you would use a simple Join, you would not get them.
I do a ToList, so that in the next statements the group join is not done twice:
var addedOrders = ordersWithTPreviouslyProcessedOrder
.Where(orderCombi => orderCombi.ProcessedOrder == null);
var changedOrders = ordersWithTPreviouslyProcessedOrder
.Where(orderCombi => !comparer.Equals(orderCombi.Order, orderCombi.PreviousOrder);
Implementation of "Compare by Value"
// equal if all values equal
protected override bool Equals(bool x, bool y)
{
if (x == null) return y == null; // true if both null, false if x null but y not null
if (y == null) return false; // because x not null
if (Object.ReferenceEquals(x, y) return true;
if (x.GetType() != y.GetType()) return false;
// compare all properties one by one:
return x.Id == y.Id
&& x.Date == y.Date
&& ...
}
For GetHashCode is one rule: if X equals Y then they must have the same hash code. If not equal, then there is no rule, but it is more efficient for lookups if they have different hash codes. Make a tradeoff between calculation speed and hash code uniqueness.
In this case: If two Orders are equal, then I am certain that they have the same Id. For speed I don't check the other properties.
protected override int GetHashCode(Order x)
{
if (x == null)
return 34339d98; // just a hash code for all null Orders
else
return x.Id.GetHashCode();
}
I am doing some coding in dapper and I get the error No mapping exists from object type to a known managed provider native type this error occurs on the myfriends var for dapper . I am using dapper to get a list of INT values from a table then comparing them against another.. this is the code that gives me that error
int myid = Convert.ToInt32(User.Identity.Name);
// The var myfriend is giving me that error above
var myfriends = sqlConnection.Query<friend>("Select otherfriendsID from friends where profileID=#myidd", new { myidd = myid }).ToList();
var profiles = sqlConnection.Query<profile>("Select top 40 * from profiles where photo is not null AND profileID=#friendship order by profileID desc", new {friendship=myfriends}).ToList();
however if I use entity everything works fine for instance this code below works..
var myfriends = (from s in db.friends where s.profileID == myid select s.otherfriendsID).ToList();
What could be going on here..
myfriends is a List<friend>. You then pass that in as a query parameter, i.e.
new {friendship=myfriends}
with:
AND profileID=#friendship
Now... what is #friendship ? How should it pass in a List<friend> here? What does that even mean to pass in a list of objects (each of which could have multiple properties) as a single parameter? (note: I'm ignoring table-valued-parameters for the purposes of this question)
So: how many myfriends do you expect? 0? 1? any number? This could be, for example:
var profileIds = myfriends.Select(x => x.ProfileId);
...
new {profileIds}
...
AND profileID in #profileIds
or maybe:
new {profileId = myfriends.Single().ProfileId}
...
AND profileID = #profileId
i have following query to select the last record in the database
using (Entities ent = new Entities())
{
Loaction last = (from x in ent.Locations select x).Last();
Location add = new Location();
add.id = last.id+1;
//some more stuff
}
the following error is returned when calling the method containing these lines via "Direct event" on ext.net:
LINQ to Entities does not recognize the method 'Prototype.DataAccess.Location Last[Location]
(System.Linq.IQueryable`1[Prototype.DataAccess.Location])' method,
and this method cannot be translated into a store expression.
table structure is following:
int ident IDENTITY NOT NULL,
int id NOT NULL,
varchar(50) name NULL,
//some other varchar fields
Last is not supported by Linq to Entities. Do OrderByDescending (usually by ID or some date column) and select first. Something like:
Location last = (from l in ent.Locations
orderby l.ID descending
select l).First();
Or
Location last = ent.Locations.OrderByDescending(l => l.ID).First();
See MSDN article Supported and Unsupported LINQ Methods (LINQ to Entities) for reference.
I've an 'product' object that has to update a record in db.
Through a 'productCode' I retrieve the object to update using Linq.
Which is the most elegant way to overwrite alle the property but not the key one and then save changes?
You will need to access each property anyways, as you fetch a record manually. my recommendation for easiest, fastest and cleanest code:
code = productCode_toChange
using(Entity ent = new Entity())
{
var update = (from x in ent.Products where x.productCode == code select x).First();
update.property1 = product.property1;
update.property2 = product.property2;
// and so on for each property you change
ent.SaveChanges();
}
you could of course try:
using(Entity ent = new Entity())
{
var update = (from x in ent.Products where x.productCode == code select x).First();
update = product;
ent.SaveChanges();
}
but i can almost guarantee this will not work, as product will certainly have an id-property it will try to write to update, which will throw an exception, given the case product is the LINQ-generated type for the table-instance.
Be careful to check for type-conformity. also note, that you the SQL-Type of product code should not be text, as this type is incomparable
var queryResult = this.kindergardenDataContext.Groups.Where(g => g.group_id == groupToAdd.group_id && g.group_enabled==true);
if (queryResult != null && queryResult.Count() != 0)
{
Group groupToEdit = queryResult.First();
groupToEdit.group_name = groupToAdd.group_name;
groupToEdit.group_create_date = groupToAdd.group_create_date;
groupToEdit.Kindergarden = groupToAdd.Kindergarden;
groupToEdit.GroupGuardian = groupToAdd.GroupGuardian;
groupToEdit.Room = groupToAdd.Room;
}
this.kindergardenDataContext.SubmitChanges();
Hi. Im beginner in LINQ. Here is ok, but foreigns key kindergarden, groupguardian, room has been duplicated in database (added new rows in database in kindergarden, groupguardian, room table). How can I set good reference to update row in group table.
Unless you retrieved groupToAdd.Kindergarden, groupToAdd.GroupGuardian, andgroupToAdd.Roomfromthis.kindergardenDataContext` or attached them with the right IDs set, that is the expected behviour. It doesn't know that you mean the same object instances that already resides in the database.