I have a big table if I use a normal query it has a timeout exception. So I want to select top 1000 then output it, the next step is to retrieve from 1001 to 2000 and log it and so on.
I am not sure how to add a parameter in my query.
int pageNumer = 0;
var query = DBContext.MyTable.Where(c=>c.FacilityID == facilityID)
.OrderBy(c=>c.FilePath)
.Skip(pageNumer*1000)
.Take(1000);
foreach(var x in query)
{
// Console.WriteLine(x.Name);
}
// I want pageNumber is incremented until it goes to the bottom of the table.
// I don't know how many records in the table.
Try this out:
int pageNumber = 0;
bool hasHitEnd = false;
while (!hasHitEnd)
{
var query = DBContext.MyTable.Where(c=>c.FacilityID == facilityID)
.OrderBy(c=>c.FilePath)
.Skip(pageNumber*1000)
.Take(1000);
foreach(var x in query)
{
// Do something
}
if (query.Count < 1000)
{
hasHitEnd = true;
}
pageNumber++;
}
Related
In my C# application i am using linq. I need a help what is the syntax for if-elseif- using linq in single line. Data, RangeDate are the inputs. Here is the code:
var Date1 = RangeData.ToList();
int record =0;
foreach (var tr in Date1)
{
int id =0;
if (tr.Item1 != null && tr.Item1.port != null)
{
id = tr.Item1.port.id;
}
else if (tr.Item2 != null && tr.Item2.port != null)
{
id = tr.Item2.port.id;
}
if (id >0)
{
if(Data.Trygetvalue(id, out cdat)
{
// Do some operation. (var cdata = SumData(id, tr.item2.port.Date)
record ++;
}
}
}
I think your code example is false, your record variable is initialized to 0 on each loop so increment it is useless .
I suppose that you want to count records in your list which have an id, you can achieve this with one single Count() :
var record = Date1.Count(o => (o.Item1?.port?.id ?? o.Item2?.port?.id) > 0);
You can use following code:
var count = RangeData.Select(x => new { Id = x.Item1?.port?.id ?? x.Item2?.port?.id ?? 0, Item = x })
.Count(x =>
{
int? cdate = null; // change int to your desired type over here
if (x.Id > 0 && Data.Trygetvalue(x.Id, out cdat))
{
// Do some operation. (var cdata = SumData(x.Id, x.Item.Item2.port.Date)
return true;
}
return false;
});
Edit:
#D Stanley is completely right, LINQ is wrong tool over here. You can refactor few bits of your code though:
var Date1 = RangeData.ToList();
int record =0;
foreach (var tr in Date1)
{
int? cdat = null; // change int to your desired type over here
int id = tr.Item1?.port?.id ?? tr.Item2?.port?.id ?? 0;
if (id >0 && Data.Trygetvalue(id, out cdat))
{
// Do some operation. (var cdata = SumData(id, tr.Item2.port.Date)
record ++;
}
}
Linq is not the right tool here. Linq is for converting or querying a collection. You are looping over a collection and "doing some operation". Depending on what that operation is, trying to shoehorn it into a Linq statement will be harder to understand to an outside reader, difficult to debug, and hard to maintain.
There is absolutely nothing wrong with the loop that you have. As you can tell from the other answers, it's difficult to wedge all of the information you have into a "single-line" statement just to use Linq.
I have an MVC application with a dynamic table on one of the pages, which the users defines how many columns the table has, the columns order and where to get the data from for each field.
I have written some very bad code in order to keep it dynamic and now I would like it to be more efficient.
My problem is that I don't know how to define the columns I should get back into my IEnumerable on runtime. My main issue is that I don't know how many columns I might have.
I have a reference to a class which gets the field's text. I also have a dictionary of each field's order with the exact property It should get the data from.
My code should look something like that:
var docsRes3 = from d in docs
select new[]
{
for (int i=0; i<numOfCols; i++)
{
gen.getFieldText(d, res.FieldSourceDic[i]);
}
};
where:
docs = List from which I would like to get only specific fields
res.FieldSourceDic = Dictionary in which the key is the order of the column and the value is the property
gen.getFieldText = The function which gets the entity and the property and returns the value
Obviously, it doesn't work.
I also tried
StringBuilder fieldsSB = new StringBuilder();
for (int i = 0; i < numOfCols; i++)
{
string field = "d." + res.FieldSourceDic[i] + ".ToString()";
if (!string.IsNullOrEmpty(fieldsSB.ToString()))
{
fieldsSB.Append(",");
}
fieldsSB.Append(field);
}
var docsRes2 = from d in docs
select new[] { fieldsSB.ToString() };
It also didn't work.
The only thing that worked for me so far was:
List<string[]> docsRes = new List<string[]>();
foreach (NewOriginDocumentManagment d in docs)
{
string[] row = new string[numOfCols];
for (int i = 0; i < numOfCols; i++)
{
row[i] = gen.getFieldText(d, res.FieldSourceDic[i]);
}
docsRes.Add(row);
}
Any idea how can I pass the linq the list of fields and it'll cut the needed data out of it efficiently?
Thanks, Hoe I was clear about what I need....
Try following:
var docsRes3 = from d in docs
select (
from k in res.FieldSourceDic.Keys.Take(numOfCols)
select gen.getFieldText(d, res.FieldSourceDic[k]));
I got my answer with some help from the following link:
http://www.codeproject.com/Questions/141367/Dynamic-Columns-from-List-using-LINQ
First I created a string array of all properties:
//Creats a string of all properties as defined in the XML
//Columns order must be started at 0. No skips are allowed
StringBuilder fieldsSB = new StringBuilder();
for (int i = 0; i < numOfCols; i++)
{
string field = res.FieldSourceDic[i];
if (!string.IsNullOrEmpty(fieldsSB.ToString()))
{
fieldsSB.Append(",");
}
fieldsSB.Append(field);
}
var cols = fieldsSB.ToString().Split(',');
//Gets the data for each row dynamically
var docsRes = docs.Select(d => GetProps(d, cols));
than I created the GetProps function, which is using my own function as described in the question:
private static dynamic GetProps(object d, IEnumerable<string> props)
{
if (d == null)
{
return null;
}
DynamicGridGenerator gen = new DynamicGridGenerator();
List<string> res = new List<string>();
foreach (var p in props)
{
res.Add(gen.getFieldText(d, p));
}
return res;
}
I have a table named industries. In this my fields are
workfor_id,
workfor_usr_id,
workfor_industry_id.
With the same values of workfor_id, I have different workfor_industry_id's.
foreach (var k in us){
var ind = dbContext.industries.Where(i => i.workfor_id ==
k.id).Select(i => i).FirstOrDefault();
string ind2 = k.industry;
var industryParts = ind2.Split(',');
var o = (industryParts.Length);
for (c = 0; c < o; c++){
ind.workfor_id = Convert.ToInt16(k.id);
ind.workfor_industry_id = Convert.ToInt16(k.industryid); }
}
To update workfor_industry_id field I have implemented inner loop inside the foreach loop to get the values of workfor_industry_id's.here same record is over loading with different workfor_industry_id's.
can you tell me how to implement this.
UPDATED
This update adds a little more error checking and assumes that -1 is never a valid value for industry_id
short GetShort(string value) {
short returnValue;
value = (value ?? string.Empty).Replace("\"",null);
return short.TryParse(value, out returnValue) ? returnValue : (short)-1;
}
foreach (var k in us){
var id=Convert.ToInt16(k.id);
var toRemove=from i in dbContext.industries
where i.workfor_id == k.id
select i;
var toAdd = from x in (k.industry ?? string.Empty).Split(',')
select new Industry {
workfor_id=id,
workfor_industry_id=GetShort(x)
};
dbContext.industries.DeleteAllOnSubmit(toRemove);
dbContext.industries.InsertAllOnSubmit(toAdd.Where(x=>x.workfor_industry_id != -1));
}
dbContext.SubmitChanges();
I have table that contains more than 12 millions of rows.
I need to index this rows using Lucene.NET (I need to perform initial indexing).
So I try to index in batch manner, by reading batch packets from sql (1000 rows per batch).
Here is how it looks:
public void BuildInitialBookSearchIndex()
{
FSDirectory directory = null;
IndexWriter writer = null;
var type = typeof(Book);
var info = new DirectoryInfo(GetIndexDirectory());
//if (info.Exists)
//{
// info.Delete(true);
//}
try
{
directory = FSDirectory.GetDirectory(Path.Combine(info.FullName, type.Name), true);
writer = new IndexWriter(directory, new StandardAnalyzer(), true);
}
finally
{
if (directory != null)
{
directory.Close();
}
if (writer != null)
{
writer.Close();
}
}
var fullTextSession = Search.CreateFullTextSession(Session);
var currentIndex = 0;
const int batchSize = 1000;
while (true)
{
var entities = Session
.CreateCriteria<BookAdditionalInfo>()
.CreateAlias("Book", "b")
.SetFirstResult(currentIndex)
.SetMaxResults(batchSize)
.List();
using (var tx = Session.BeginTransaction())
{
foreach (var entity in entities)
{
fullTextSession.Index(entity);
}
currentIndex += batchSize;
Session.Flush();
tx.Commit();
Session.Clear();
}
if (entities.Count < batchSize)
break;
}
}
But, the operation times out when current index is bigger then 6-7 million. NHibernate Pagging throws time out.
Any suggestions, any other way in NHibernate to index this 12 millions of rows?
EDIT:
Probably I will implement the most peasant solution.
Because BookId is cluster index in my table and select occurs very fast by BookId, I am going to find max BookId and going through all records and index all of them them.
for (long = 0; long < maxBookId; long++)
{
// get book by bookId
// if book exist, index it
}
If you have any other suggestion, please reply yo this question.
Instead of paging your whole data set, you could try to divide and conquer it. You said you had an index on book id, just change your criteria to return batches of books according to bounds of bookid :
var entities = Session
.CreateCriteria<BookAdditionalInfo>()
.CreateAlias("Book", "b")
.Add(Restrictions.Gte("BookId", low))
.Add(Restrictions.Lt("BookId", high))
.List();
Where low and high are set like 0-1000, 1001-2000, etc
I have created a JTable, the table contains 4 rows, and for a specific row, I have overridden the sorting and its working fine only on that column.
Status Scheduled Date Scheduled time Status
false 30/01/2012 02:00:00 Scheduled
false 29/01/2012 14:58:00 Scheduled
false 29/01/2012 15:50:00 Scheduled
For Scheduled Date, which I try to sort, it would sort, but the respecitve rows are not being updated.
Here is my code for sorting
public static void sortColumn(DefaultTableModel model, int colIndex,
boolean sortingOrder) {
Vector<?> data = model.getDataVector();
Object[] colData = new Object[model.getRowCount()];
SortedSet<Object> dataCollected = null;
List<Date> dateCollected;
boolean dateFlag = false;
dateCollected = new ArrayList<Date>();
// Copy the column data in an array
for (int i = 0; i < colData.length; i++) {
Object tempData = ((Vector<?>) data.get(i)).get(colIndex);
if ((colIndex == 1 || colIndex == 4)
&& tempData.toString().contains("/")) {
String[] _scheduledDate1 = ((String) tempData).split("/");
Calendar _cal1 = Calendar.getInstance();
_cal1.set(Integer.parseInt(_scheduledDate1[2]),
Integer.parseInt(_scheduledDate1[1]) - 1,
Integer.parseInt(_scheduledDate1[0]));
dateCollected.add(_cal1.getTime());
dateFlag = true;
} else {
colData[i] = ((Vector<?>) data.get(i)).get(colIndex);
}
}
// DateCompare compare = new DateCompare();
if (!dateFlag) {
dataCollected = new TreeSet<Object>();
dataCollected.add(colData);
dateFlag = false;
}
// Copy the sorted values back into the table model
if ((colIndex == 1 || colIndex == 4) && dateFlag) {
SimpleDateFormat sdf = new SimpleDateFormat("dd/MM/yyyy");
sortOrder = !sortOrder;
if (sortOrder) {
Collections.sort(dateCollected);
} else {
Collections.sort(dateCollected, Collections.reverseOrder());
}
colData = dateCollected.toArray();
for (int i = 0; i < colData.length; i++) {
((Vector<Object>) data.get(i)).set(colIndex,
sdf.format(((Date) colData[i]).getTime()));
}
} else {
for (int i = 0; i < colData.length; i++) {
((Vector<Object>) data.get(i)).set(colIndex, colData[i]);
}
}
model.fireTableStructureChanged();
}
How to I get the entire row update accordingly?
I found the problem, my object against I was comparing was wrong, I've change the code for the same it all works fine.
I implemented QuickSort algorithm to sort the vector on the specific column I need.