How can I merge two Linq IEnumerable<T> queries without running them? - linq

Question

How do I merge a List<T> of TPL-based tasks for later execution?
public async IEnumerable<Task<string>> CreateTasks(){ /* stuff*/ }
My assumption is .Concat() ...
void MainTestApp() // Full sample available upon request.
{
List<string> nothingList = new List<string>();
nothingList.Add("whatever");
cts = new CancellationTokenSource();
delayedExecution =
from str in nothingList
select AccessTheWebAsync("", cts.Token);
delayedExecution2 =
from str in nothingList
select AccessTheWebAsync("1", cts.Token);
delayedExecution = delayedExecution.Concat(delayedExecution2);
}
/// SNIP
async Task AccessTheWebAsync(string nothing, CancellationToken ct)
{
// return a Task
}
I want to make sure that this won't spawn any task or evaluate anything. In fact, I suppose I'm asking "what logically executes an IQueryable to something that returns data"?
Background
Since I'm doing recursion and I don't want to execute this until the correct time, what is the correct way to merge the results if called multiple times?
If it matters I'm thinking of running this command to launch all the tasks var AllRunningDataTasks = results.ToList(); followed by this code:
while (AllRunningDataTasks.Count > 0)
{
// Identify the first task that completes.
Task<TableResult> firstFinishedTask = await Task.WhenAny(AllRunningDataTasks);
// ***Remove the selected task from the list so that you don't
// process it more than once.
AllRunningDataTasks.Remove(firstFinishedTask);
// TODO: Await the completed task.
var taskOfTableResult = await firstFinishedTask;
// Todo: (doen't work)
TrustState thisState = (TrustState)firstFinishedTask.AsyncState;
// TODO: Update the concurrent dictionary with data
// thisState.QueryStartPoint + thisState.ThingToSearchFor
Interlocked.Decrement(ref thisState.RunningDirectQueries);
Interlocked.Increment(ref thisState.CompletedDirectQueries);
if (thisState.RunningDirectQueries == 0)
{
thisState.TimeCompleted = DateTime.UtcNow;
}
}

Answer 1

To answer the specific question "what logically executes an IQueryable to something that returns data"? That would be anything that either forces the production of at least one value, or forces the discovery of whether a value is available.
For example, ToList, ToArray, First, Single, SingleOrDefault, and Count will all force evaluation. (Although First will not evaluate the entire collection - it'll retrieve the first item and then stop.) These all have to at least start retrieving values, because there's no way for any of them to return what they return without doing so. In the case of ToList and ToArray, these return fully-populated non-lazy collections, which is why they have to evaluate everything. The methods that return a single item need to at least ask for the first item, and the Single ones will then go on to check that nothing else comes out if evaluation continues (and throw an exception if there turns out to be more).
Using foreach to iterate over the query will also force evaluation. (And again, it's for the same reason: you're asking it for actual values from the collection so it has to provide them.)
Concat does not evaluate immediately because it doesn't need to - it's only when you ask the concatenated sequence for a value that it needs to ask its inputs for values.
BTW, although you asked about IQueryable you're not using that in the examples here. This can matter, because there are some differences in how that works compared to the LINQ to Objects implementation (which you get for plain IEnumerable<T>) that you're actually getting. I don't think it makes a difference in this example, but it makes me wonder if something might have changed between your original code, and the version you posted for illustration here? It can matter because different LINQ providers can do things different ways. The IEnumerable<T> flavour of Concat definitely uses deferred evaluation, and although I'd expect that to be true for most other LINQ implementations, it's not absolutely a given.
If you need to use the results multiple times, and you want to ensure that you only evaluate them once, but that you don't evaluate them until you actually need them, then the usual approach is to call ToList at the point where you definitely need to evaluate, and then hold onto the resulting List<T> so you can use it again. Once you've got the data in List<T> (or array) form you can use that list as many times as you like.
By the way, your first question has an issue:
"How do I merge a List of TPL-based tasks for later execution?"
In general, if you already have a TPL task then you can't stop it from executing. (There is an exception to this. If you construct a Task directly instead of using one of the more normal ways of creating it, it won't actually run until you tell it to. But in general, APIs that return tasks return live ones, i.e., they may well already be running, or even complete, by the time you get your hands on them.)
The "later execution" in your example comes from the fact that you don't actually have a list of tasks at all to start with. (If you did in fact have a List<T> of tasks, "later execution" would not be an option.) What you have is a couple of enumerables which, if you were to evaluate them, would create tasks. The act of creating the task is indivisible from the act of starting it in any TAP-style API that returns a Task.
Based on the rest of what you wrote, I think what you are really asking is:
"How do I merge multiple IEnumerable<Task<T>> objects into a single IEnumerable<Task<T>> in a way that defers evaluation of the underlying enumerables until the combined enumerable itself is evaluated?"
Concat should work for that.

Answer 2

The following is a hacky way to get data merged... I don't like how I had to use "nothingList" in Main or a few other aspects of this sample, but it seems to get the job done and allow me to merge pending tasks.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
// Add a using directive and a reference for System.Net.Http.
using System.Net.Http;
// Add the following using directive.
using System.Threading;
namespace ProcessTasksAsTheyFinish
{
public partial class MainWindow : Window
{
// Declare a System.Threading.CancellationTokenSource.
CancellationTokenSource cts;
List<IEnumerable<Task>> launchList = new List<IEnumerable<Task>>();
public MainWindow()
{
InitializeComponent();
List<string> nothingList = new List<string>();
nothingList.Add("whatever");
cts = new CancellationTokenSource();
delayedExecution =
from str in nothingList
select AccessTheWebAsync("", cts.Token);
List<string> nothingList2 = new List<string>();
nothingList2.Add("whatever");
delayedExecution2 =
from str in nothingList2
select AccessTheWebAsync("1", cts.Token);
launchList.Add(delayedExecution);
launchList.Add(delayedExecution2);
delayedExecution = delayedExecution.Concat(delayedExecution2);
}
IEnumerable<Task> delayedExecution = null;
IEnumerable<Task> delayedExecution2 = null;
private async void startButton_Click(object sender, RoutedEventArgs e)
{
resultsTextBox.Clear();
// Instantiate the CancellationTokenSource.
try
{
// ***Set up the CancellationTokenSource to cancel after 25 seconds.
//cts.CancelAfter(250000);
var test = delayedExecution;// AccessTheWebAsync("", cts.Token);
var testList = test.ToList();
while (testList.Count() > 0)
{
var firstFinishedTask = await Task.WhenAny(testList);
testList.Remove(firstFinishedTask);
await firstFinishedTask;
}
resultsTextBox.Text += "\r\nDownloads complete.";
}
catch (OperationCanceledException tee)
{
resultsTextBox.Text += "\r\nDownloads canceled.\r\n";
}
catch (Exception)
{
resultsTextBox.Text += "\r\nDownloads failed.\r\n";
}
cts = null;
}
private void cancelButton_Click(object sender, RoutedEventArgs e)
{
if (cts != null)
{
cts.Cancel();
}
}
async Task<string> AccessTheWebAsync(string nothing, CancellationToken ct)
{
// CHANGE THIS VALUE TO CONTROL THE TESTING
bool delayConversionOfQueryToList = false;
HttpClient client = new HttpClient();
// Make a list of web addresses.
List<string> urlList = null;
if (nothing == "1")
{
urlList = SetUpURLList2();
}
else urlList = SetUpURLList();
// ***Create a query that, when executed, returns a collection of tasks.
IEnumerable<Task<int>> downloadTasksQuery =
from url in urlList select ProcessURL(url, client, ct);
// DEBUG!!!
if (delayConversionOfQueryToList == true)
{
await Task.Delay(10000);
resultsTextBox.Text += String.Format("\r\nDelay of IQueryable complete. Tip: Did you see any IsRunning messages?");
}
// ***Use ToList to execute the query and start the tasks.
List<Task<int>> downloadTasks = downloadTasksQuery.ToList();
// DEBUG!!!
if (delayConversionOfQueryToList == false)
{
await Task.Delay(10000);
resultsTextBox.Text += String.Format("\r\nDelay of .ToList() complete. Tip: Did you see any IsRunning messages?");
}
// ***Add a loop to process the tasks one at a time until none remain.
while (downloadTasks.Count() > 0)
{
// Identify the first task that completes.
Task<int> firstFinishedTask = await Task.WhenAny(downloadTasks);
resultsTextBox.Text += String.Format("\r\nID {0}", firstFinishedTask.Id);
// ***Remove the selected task from the list so that you don't
// process it more than once.
downloadTasks.Remove(firstFinishedTask);
// Await the completed task.
int length = await firstFinishedTask;
resultsTextBox.Text += String.Format("\r\nLength of the download: {0}", length);
}
return nothing;
}
private List<string> SetUpURLList()
{
List<string> urls = new List<string>
{
"http://msdn.microsoft.com",
"http://msdn.microsoft.com/library/windows/apps/br211380.aspx",
"http://msdn.microsoft.com/en-us/library/hh290136.aspx",
"http://msdn.microsoft.com/en-us/library/dd470362.aspx",
"http://msdn.microsoft.com/en-us/library/aa578028.aspx",
"http://msdn.microsoft.com/en-us/library/ms404677.aspx",
"http://msdn.microsoft.com/en-us/library/ff730837.aspx"
};
return urls;
}
private List<string> SetUpURLList2()
{
List<string> urls = new List<string>
{
"http://www.google.com",
};
return urls;
}
async Task<int> ProcessURL(string url, HttpClient client, CancellationToken ct)
{
resultsTextBox.Text += String.Format("\r\nIS RUNNING {0}", url);
// GetAsync returns a Task<HttpResponseMessage>.
HttpResponseMessage response = await client.GetAsync(url, ct);
// Retrieve the website contents from the HttpResponseMessage.
byte[] urlContents = await response.Content.ReadAsByteArrayAsync();
// Thread.Sleep(3000);
// await Task.Delay(1000, ct);
return urlContents.Length;
}
}
}
// Sample Output:
IS RUNNING http://msdn.microsoft.com
IS RUNNING http://msdn.microsoft.com/library/windows/apps/br211380.aspx
IS RUNNING http://msdn.microsoft.com/en-us/library/hh290136.aspx
IS RUNNING http://msdn.microsoft.com/en-us/library/dd470362.aspx
IS RUNNING http://msdn.microsoft.com/en-us/library/aa578028.aspx
IS RUNNING http://msdn.microsoft.com/en-us/library/ms404677.aspx
IS RUNNING http://msdn.microsoft.com/en-us/library/ff730837.aspx
IS RUNNING http://www.google.com
Delay of .ToList() complete. Tip: Did you see any IsRunning messages?
ID 1
Length of the download: 48933
ID 2
Length of the download: 375328
ID 3
Length of the download: 220428
ID 4
Length of the download: 222256
ID 5
Length of the download: 229330
ID 6
Length of the download: 136544
ID 7
Length of the download: 207171
Delay of .ToList() complete. Tip: Did you see any IsRunning messages?
ID 8
Length of the download: 43945
Downloads complete.