EFCore Getting Differences between two tables - linq

I am trying to get the differences between two tables, let's call them YestedayEmployees and TodaysEmployees. I am currently getting all both tables and then checking each item to see if the employee status changed or if they were deleted (they appear in yesterday's but not in today's table). This was fine when it was a smaller number of records but as the records increase, the network and compute overhead on my server are becoming a problem. Is there a way to do this as a linq query in EFCore? (or even two one for the deleted and one for the changed)

Please refer the following query statement:
Test data (you could get the table value from database using EF core, more detail information about using EF core with asp.net MVC, check this link):
List<Employee> todayEmployees = new List<Employee>()
{
new Employee(){ EmpID=1001, EmpName="David", Status="OT" },
new Employee(){ EmpID=1002, EmpName="Tom", Status="Off-line" },
new Employee(){ EmpID=1003, EmpName="Jason", Status="OT" },
new Employee(){ EmpID = 1004, EmpName="Dick", Status="Off-line" },
new Employee(){ EmpID = 1005, EmpName="Cece", Status="OT" },
new Employee(){ EmpID = 1006, EmpName="Dillion", Status="OT" },
new Employee(){ EmpID = 1007, EmpName="Jeffery", Status="Off-Line" }
};
List<Employee> yesterdayEmployees = new List<Employee>()
{
new Employee(){ EmpID=1001, EmpName="David", Status="OT" },
new Employee(){ EmpID=1002, EmpName="Tom", Status="OT" },
new Employee(){ EmpID=1003, EmpName="Jason", Status="OT"},
new Employee(){ EmpID = 1004, EmpName="Dick", Status="OT" },
new Employee(){ EmpID = 1005, EmpName="Cece", Status="Off-Line" }
};
To get the Employee which status is changed, we could use Join clause and where clause to compare the employee status:
// get the employees which changes status
var result = (from t in todayEmployees
join y in yesterdayEmployees
on t.EmpID equals y.EmpID
where (t.Status != y.Status)
select t).ToList();
Output:
//get the employees status change information
var query3 = (from t in todayEmployees
join y in yesterdayEmployees
on t.EmpID equals y.EmpID
where (t.Status != y.Status)
select new EmployeeViewModel()
{
EmpID = t.EmpID,
EmpName = t.EmpName,
StatusChangeLog = "change status from " + t.Status + " to " + y.Status
}).ToList();
Output:
To get the Employees which in the TodayEmployees Table, but not exist in the YesterdayEmployee Table, we could use the contains method to determines whether a sequence contains a specified element.
//get the employees, which in TodayEmployees Table, but not exist in the YesterdayEmployee
var query4 = (from t in todayEmployees
where !(from y in yesterdayEmployees select y.EmpID).Contains(t.EmpID)
select t).ToList();
var query5 = todayEmployees.Where(c => !yesterdayEmployees.Select(y => y.EmpID).Contains(c.EmpID)).Select(t => t).ToList();
Output:

Related

.net core linq select overload index does not work

I got a linq lambda select code that works before I added the Select index overload. Before, I got the list of records but I need the index which I use to assign a unique Id to each record. When I add with ToList(), I get an exception with no error/inner exception. Only way I can get the code to not throw an error is to use .AsEnumberable() but I need a list. I read many post that .ToList() works with the overload but I have been unsuccessful.
Here is my code and my attempt to fix this
var emps = this.DbContext.Employees
.GroupJoin(this.DbContext.Depts,
employee => employee.EmployeeId,
dept => dept.EmployeeId,
(employee, dept) => new { employee, dept }
)
.SelectMany(
employee_dept_left => employee_dept_left.dept.DefaultIfEmpty(),
(employee_dept_left, dept) => new { employee_dept_left, dept }
)
.Join(this.DbContext.Divs,
emp_emp_dept => emp_emp_dept.employee_dept_left.employee.DivId,
division => division.DivId,
(emp_emp_dept, division) => new { emp_emp_dept, division }
)
.Where(s => !string.IsNullOrEmpty(filter.selectedDiv))
.GroupBy(grouped => new
{
grouped.emp_emp_dept.employee_dept_left.employee.EmployeeId,
grouped.emp_emp_dept.employee_dept_left.employee.LastNm,
grouped.emp_emp_dept.employee_dept_left.employee.FirstNm,
grouped.emp_emp_dept.employee_dept_left.employee.DivId
})
.Select((joined, index) => new EmployeeViewModel
{
Id = index,
EmployeeId = joined.Key.EmployeeId,
LastNm = joined.Key.LastNm.Trim(),
FirstNm = joined.Key.FirstNm.Trim(),
DivisionId = joined.Key.DivId,
}).ToList();
The error message says
Could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
I tried using .AsEnumerable() instead of .ToList():
List<EmployeeViewModel> test = emps.Cast<EmployeeViewModel>().ToList();
but this throws an exception.
Any help is greatly appreciated.
Thanks in advance
Problem that this Select is not currently translatable to the SQL. You can make additional Select to solve issue with AsEnumerable().
...
.Select(joined => new
{
EmployeeId = joined.Key.EmployeeId,
LastNm = joined.Key.LastNm.Trim(),
FirstNm = joined.Key.FirstNm.Trim(),
DivisionId = joined.Key.DivisionId,
})
.AsEnumerable()
.Select((x, index) => new EmployeeViewModel
{
Id = index,
EmployeeId = x.EmployeeId,
LastNm = x.LastNm,
FirstNm = x.FirstNm,
DivisionId = x.DivisionId,
}).ToList();
And note that query is more readable in Query syntax when there are joins.
var query =
from employee in this.DbTracsContext.Employees
join dept in his.DbTracsContext.Depts on employee.EmployeeId equals dept.EmployeeId into employee_dept_left
from dept in employee_dept_left.DefaultIfEmpty()
join division in this.DbTracsContext.Depts on employee.DivisionId equals division.DivisionId
where string.IsNullOrEmpty(filter.DivisionSelection) || filter.DivisionSelection == "0" || employee.DivisionId == filter.DivisionSelection
group employee by new { employee.EmployeeId, employee.LastNm, employee.FirstNm, employee.DivisionId } into g
select new
{
EmployeeId = g.Key.EmployeeId,
LastNm = g.Key.LastNm.Trim(),
FirstNm = g.Key.FirstNm.Trim(),
DivisionId = g.Key.DivisionId,
};
var emps = query
.AsEnumerable()
.Select((x, index) => new EmployeeViewModel
{
Id = index,
EmployeeId = x.EmployeeId,
LastNm = x.LastNm,
FirstNm = x.FirstNm,
DivisionId = x.DivisionId,
}).ToList();

Dynamic LINQ Where query with joining other tables

I have several tables (the exact number is not known when the program is build) looking like this (the number of rows and columns may differ from table to table):
The source data is stored in a data set. Now I want to generate a new table where all data of all ids is stored (the picture shows only the result for id 10 and 20 but the target table should contain the data for all ids):
The equivalent SQLite statement for that looks like this:
SELECT * FROM Dataset
JOIN Datensatz2 ON (Dataset.ID=Datensatz2.ID)
JOIN Datensatz3 ON (Datensatz3.ID=Dataset.ID)
JOIN Datensatz4 ON (Datensatz4.ID=Dataset.ID)
WHERE Dataset.Id=10
UNION
SELECT * FROM Dataset
JOIN Datensatz2 ON (Dataset.ID=Datensatz2.ID)
JOIN Datensatz3 ON (Datensatz3.ID=Dataset.ID)
JOIN Datensatz4 ON (Datensatz4.ID=Dataset.ID)
WHERE Dataset.Id=20
...
The double id columns will be removed afterwards so donĀ“t worry about that. The questions is now how to convert it as a dynamic LINQ query?
There are plenty of open question but maybe this helps to solve it. Since the tables are already stored in a DataSet you could use Linq-To-DataSet and Enumerable.GroupBy to group by ID:
var idTables = ds.Tables.Cast<DataTable>().Where(t => t.Columns.Contains("Id"));
if(!idTables.Any()){ MessageBox.Show("No id-tables"); return; }
var idRowGroups = idTables.SelectMany(t => t.AsEnumerable())
.GroupBy(row => row.Field<int>("Id"))
.Select(grp => new { ID = grp.Key, Rows = grp });
foreach(var idGroup in idRowGroups)
{
Console.WriteLine("ID:{0} Rows:{1}"
, idGroup.ID
, String.Join(" | ", idGroup.Rows.Select(row => String.Join(",", row.ItemArray))));
}
Sample data:
var ds = new DataSet();
DataTable t1 = new DataTable();
t1.Columns.Add("Id", typeof(int));
t1.Columns.Add("Data", typeof(int));
t1.Rows.Add(1, 1);
t1.Rows.Add(2, 10);
t1.Rows.Add(3, 100);
t1.Rows.Add(4, 1000);
ds.Tables.Add(t1);
t1 = new DataTable();
t1.Columns.Add("Id", typeof(int));
t1.Columns.Add("Data", typeof(int));
t1.Rows.Add(4, 5);
t1.Rows.Add(5, 50);
t1.Rows.Add(7, 500);
t1.Rows.Add(3, 5997);
ds.Tables.Add(t1);
t1 = new DataTable();
t1.Columns.Add("Id", typeof(int));
t1.Columns.Add("Data1", typeof(int));
t1.Columns.Add("Data2", typeof(int));
t1.Rows.Add(1, 5, 0);
t1.Rows.Add(3, 7, 1);
t1.Rows.Add(5, 9, 11);
t1.Rows.Add(7, 11, 222);
ds.Tables.Add(t1);
Output:
ID:1 Rows:1,1 | 1,5,0
ID:2 Rows:2,10
ID:3 Rows:3,100 | 3,5997 | 3,7,1
ID:4 Rows:4,1000 | 4,5
ID:5 Rows:5,50 | 5,9,11
ID:7 Rows:7,500 | 7,11,222
Ok, I finally made it but it seems to be much too complicated. If someone is able to help me improve the solution he is very welcome.
DataSet dsResult = new DataSet();
var idTables = ds.Tables.Cast<DataTable>().Where(t => t.Columns.Contains("ID"));
if (!idTables.Any()) { MessageBox.Show("No id-tables"); return; }
var idRowGroups = idTables.SelectMany(t => t.AsEnumerable())
.GroupBy(row => row.Field<Int64>("ID"))
.Select(grp => new { ID = grp.Key, Rows = grp });
foreach (var idGroup in idRowGroups)
{
var liste = idGroup.Rows.ToList();
for (int i = 0; i < liste.Count; i++)
{
if (!dsResult.Tables.Contains(liste[i].Table.TableName))
{
dsResult.Tables.Add(liste[i].Table.TableName);
foreach (DataColumn dtCol in liste[i].Table.Columns)
{
if (dsResult.Tables[liste[i].Table.TableName].Columns.Contains("ID"))
dsResult.Tables[liste[i].Table.TableName].Columns.Add(dtCol.ColumnName+i.ToString());
else
{
dsResult.Tables[liste[i].Table.TableName].Columns.Add(dtCol.ColumnName);
}
dsResult.Tables[liste[i].Table.TableName].Columns[dtCol.ColumnName].DataType = dtCol.DataType;
}
}
DataRow dRow = dsResult.Tables[liste[i].Table.TableName].NewRow();
dRow.ItemArray = liste[i].ItemArray;
dsResult.Tables[liste[i].Table.TableName].Rows.Add(dRow);
}
IEnumerable<IEnumerable<DataRow>> allTablesRows = dsResult.Tables.Cast<DataTable>()
.Select(table => table.AsEnumerable())
.CartesianProduct();
int k = 0;
foreach (var rows in allTablesRows)
{
DataRow zRow = dsErgebnis.Tables[2].NewRow();
foreach (DataRow dRow in rows)
{
for (int i = 0; i < dRow.ItemArray.Length; i++)
{
zRow[k] = dRow.ItemArray[i];
k++;
}
}
k = 0;
dsErgebnis.Tables[2].Rows.Add(zRow);
}
dsResult.Clear();
}
First I filter the content by ID.
Then I put the result in a new tables (all rows with the ID 10 I found in 'Datasatz2' in dataset 'ds' for example I put to a new table 'Datasatz2' in the dataset 'dsResult').
At least I build the cartesian product of all tables and store it in dtaset dsErgebnis.
The result is what I expect but as mentioned before I am not satisfied with the solution.

LINQ EF QUERY (view difference from condition & dynamic)

I need to make a query to filter records, when get distinct records, get these records information by difference conditions. Also I need these to be dynamic(quantity filter in first select)
Let me show you an example:
I have 2 tables:
tblCustomers:
id customerName
1 John
2 Philip
3 Steve
tblOrders
id customerId ordId payment
1 1 100 True
2 1 101 True
3 1 102 False
4 2 101 True
5 2 102 True
6 2 103 False
7 3 101 True
My condition is:
where (orderId = 101 and orderId = 102)
but get all records of this customer that payment = true I mean my condition is different from what I need to see.
I want to receive all records with payment=True without care of orderId
I must get:
john 100
john 101
Philip 101
Philip 102
Clearing: I need two step - first filter customer who has orderId=101&102, in second step i want to show these selected customers' orderId which payment is true. so for example in first step i get john(who has order id =101&102) then show john 100 - john 101 (which payment istrue). consider tblorder.id=1 isn't in first query but I must show in final result.
#Raphael direct me to better expression:I want to see all payment true order for the customers that have orders (101 & 102). but orderids may be more than 2 (thanks #Raphael).
2nd problem is: it must be dynamic. Sometimes I have more than 10 orderId that must be checked - sometimes less. I mean my query must be flexible.
In SQL Server select command, I can prepare a string variable and use but in linq I can't do it.
From what I understood from your post and the comments, you need all customers, where the orderId is 101 or 102 and the payment is true.
You need the where clause with the orderIds to be dynamic so you can change the Ids to be checked against outside of the query.
List<int> IDList = new List<int>();
IDList.Add(101);
IDList.Add(102);
IDList.Add(110);
//...
var result = from cust in tblCustomers
join order in tblOrders on cust.id equals order.customerId
where IDList.Contains(order.ordId) && order.payment == true
select new {
Name = cust.customerName
OrderId = order.ordId
payment = order.payment
//...
}
With this you can store all orderIds which need to be checked against in a list, which in turn you can edit from your code.
EDIT
I really haven't found a clean solution to your problem, so I took a detour, which isn't very clean but should work. In my example I created 2 classes, Customer & Order and filled it with your data from above. Then I took my first query and attached a groupBy to it and a where-clause comparing the length of the grouping with the length of the list
var result = (from cust in Customers
join order in Orders on cust.Id equals order.customerId
where IDList.Contains(order.orderId) &&
order.payment == true
select new {
Name = cust.Name,
OrderId = order.orderId,
Payment = order.payment
//...
}).GroupBy (r => r.Name)
.Where (r => r.Count() == IDList.Count());
Output:
Name OrderId Payment
Philip 101 True
Philip 102 True
If you want/need it, I can provide you with the whole Linqpad query, so you can see my whole code and what I have done. Speaking of Linqpad: ignore the result.Dump() line. It won't work on visual Studio.
void Main()
{
List<Customer> customers = new List<Customer>
{
new Customer { Id = 1, Name = "John" },
new Customer { Id = 2, Name = "Philip" },
new Customer { Id = 3, Name = "Steve" }
};
List<Order> orders = new List<Order>
{
new Order { Id = 1, CustomerId = 1, OrderId = 100, Payment = true },
new Order { Id = 2, CustomerId = 1, OrderId = 101, Payment = true },
new Order { Id = 3, CustomerId = 1, OrderId = 102, Payment = false },
new Order { Id = 4, CustomerId = 2, OrderId = 101, Payment = true },
new Order { Id = 5, CustomerId = 2, OrderId = 102, Payment = true },
new Order { Id = 6, CustomerId = 2, OrderId = 103, Payment = false },
new Order { Id = 7, CustomerId = 3, OrderId = 101, Payment = true }
};
List<int> orderIds = new List<int> { 101, 102 };
var customersWithRelevantOrders =
from ord in orders
group ord by ord.CustomerId into customerOrders
where orderIds.All (
i => customerOrders.Select (co => co.OrderId).Contains(i))
select customerOrders.Key;
var paymentTrueOrdersForTheseCustomers =
from ord in orders
join cust in customers on ord.CustomerId equals cust.Id
where ord.Payment
where customersWithRelevantOrders.Contains(cust.Id)
select new
{
Name = cust.Name,
OrderId = ord.OrderId
};
}
public class Customer
{
public int Id { get; set; }
public string Name { get; set; }
}
public class Order
{
public int Id { get; set; }
public int CustomerId { get; set; }
public int OrderId { get; set; }
public bool Payment { get; set; }
}

group by base on 2 element

equal this
select id,name, count(*) from table group by id, name
what is in linq???
In case of entity framework it is better to return computed projection directly from SQL:
var query = from x in context.YourEntities
group x by new { x.ID, x.Name } into y
select new
{
y.Key.ID,
y.Key.Name,
y.Count()
};
This will do Count in database and reduce amount of transferred data.
var groups = table.GroupBy(elt => new {ID = elt.ID, Name = elt.name});
foreach (var group in groups)
{
var ID = group.Key.ID;
var name = group.Key.Name;
var count = group.Count();
...
}

How to select same column's name from different table in Linq?

I've 2 tables with same column's name, for example, both table A and table B has column's name "Test". I want to select column Test from both table A and B to entity class. How can I do this?
It sounds like you want the two entities of TableA and TableB merged into a new object. You can use the .Select() extension method to create a new anonymous type, or into a class that you already have defined.
The requirement here is that you've got to find a common attribute between TableA and TableB. Here I assume you've got something like ID to match them together.
Anonymous Type
var mergedTests = from a in db.TableA
join b in db.TableB on a.CommonID equals b.CommonID
select new
{ TestFromA = a.Test, TestFromB = b.Test }
.ToList();
Existing Class
List<MyCustomTests> mergedTests = from a in db.TableA
join b in db.TableB on a.CommonID equals b.CommonID
select new MyCustomTests
{ TestName= a.Test, ShortName= b.Test }
.ToList();
class Program
{
static void Main(string[] args)
{
var A = new Data[] {
new Data { Test = 1, Relation = 1 },
new Data { Test = 2, Relation = 2 },
new Data { Test = 3, Relation = 3 },
new Data { Test = 4, Relation = 4 },
new Data { Test = 5, Relation = 5 },
};
var B = new Data[] {
new Data { Test = 2, Relation = 2 },
new Data { Test = 3, Relation = 3 },
new Data { Test = 5, Relation = 5 },
};
var res = from a in A
join b in B on a.Relation equals b.Relation
select new { TestA = a.Test, TestB = b.Test };
}
}
class Data
{
public int Test;
public int Relation;
}

Resources