I have the following TPL Dataflow that never completes when using a predicate to filter the items passed from the TransformBlock to the ActionBlock.
If the predicate returns false for any of the items, then the dataflow hangs.
Please can someone offer some insight as to what's happening and how to resolve this?
// define blocks
var getBlock = new TransformBlock<int, int>(i =>
{
Console.WriteLine($"getBlock: {i}");
return ++i;
});
var writeBlock = new ActionBlock<int>(i =>
{
Console.WriteLine($"writeBlock: {i}");
});
// link blocks
getBlock.LinkTo(writeBlock, new DataflowLinkOptions
{
PropagateCompletion = true
}, i => i == 12); // <-- this predicate prevents the completion of writeBlock
// push to block
var items = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
foreach (var i in items)
{
getBlock.Post(i);
}
// wait for all operations to complete
getBlock.Complete();
await writeBlock.Completion; // <-- application hangs here
The getBlock is not completing because the items posted to it have nowhere to go. If you have a predicate add a null target so that any items that don't match have a place to exit the pipeline.
getBlock.LinkTo(writeBlock, new DataflowLinkOptions
{
PropagateCompletion = true
}, i => i == 12)
getBlock.LinkTo(DataflowBlock.NullTarget<int>());
Related
Consider the following:
import { timer, of, Subject } from 'rxjs';
import { map, mergeMap } from 'rxjs/operators';
(function(){
var sub$ = new Subject<number>();
var complete = false;
var out$ = sub$.pipe(
mergeMap(n => {
complete = n === 7;
if (complete) {
return of(n);
} else {
return timer(1000).pipe(map(() => n));
}
})
);
out$.subscribe({
next: (n) => console.log(n)
});
for (var i = 1; i < 8; i++) {
sub$.next(i);
}
sub$.complete();
})();
Even though the values broadcasted by the subject, in order, are 1, 2, 3, 4, 5, 6, 7, the output of the above code is:
7
1
2
3
4
5
6
How can this pipe be organized differently to assure order despite internal delays?
You can use concatMap instead of mergeMap to achieve this https://rxjs.dev/api/index/function/concatMap
use timer(1000).pipe(map(() => n)) instead of of(n).
or use endWith operator
We have following problem. We parsing files (producer) and convert the data into a c# data format. Afterwards we need to merge all of this data together.
As this can be done in parallel we startet to implement a producer consumer pattern but stucking a bit in how to merge the results in a optimized manner.
Producer produces 5 data elements (named as follows):
1, 2, 3, 4, 5
Merges which will be done but the order does not matter. As soon as there are 2 elements created, they can be merged.
Example:
(1)and(2), (3)and(4), (12)and(34), (1234)and(5)
Data data = new Data();
BlockingCollection<Data> collection = new BlockingCollection<Data>();
Task consumer = Task.Factory.StartNew(() =>
{
while (!collection.IsCompleted)
{
var item = collection.Take();
data.Merge(item);
}
});
Task producer = Task.Factory.StartNew(() =>
{
Parallel.ForEach(files, file =>
{
collection.Add(new Data(file));
});
collection.CompleteAdding();
});
Task.WaitAll(consumer, producer);
//here we got the data merged with all files
return data;
This code works but has a problem. In our case the producer is much faster than the consumer. So we need parallel consumers who are waiting for two items to be at the queue. Then they should take them, merge them together and put them back to the queue. Is there any known pattern for such a merge issue?
We have found a quite nice solution for this.
Data data = new Data();
BlockingCollection<Data> collection = new BlockingCollection<Data>();
List<Task<Data>> tasks = new List<Task<Data>>();
Enumerable.Range(0, 5).ForEach(t => {
Task consumer = Task.Factory.StartNew(() =>
{
Data result = null;
foreach (Data data in collection.GetConsumingEnumerable())
{
if (result == null)
{
result = data;
}
else
{
result.Merge(data);
}
}
return result;
});
tasks.Add(consumer);
});
Task producer = Task.Factory.StartNew(() =>
{
Parallel.ForEach(files, file =>
{
collection.Add(new Data(file));
});
collection.CompleteAdding();
});
Task.WaitAll(consumers.Concat(new []{ producer}));
List<Data> datas = consumers.Select(t => t.Result).Where(t => t != null).ToList();
Data finalResult = datas.First();
foreach (Data toBeMerged in datas.Skip(1))
{
finalResult.Merge(toBeMerged);
}
return finalResult;
I have a list of project numbers that I need to process. A project could have about 8000 items and I need to get the data for each item in the project and then push this data into a list of servers. Can anybody please tell me the following..
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
2) Am I doing the await on all actionBlocks correctly?
3) How do I make the database call async?
Here is the database code
public MemcachedDTO GetIR(MemcachedDTO dtoItem)
{
string[] Tables = new string[] { "iowa", "la" };
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["test"].ConnectionString))
{
using (SqlCommand command = new SqlCommand("test", connection))
{
DataSet Result = new DataSet();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#ProjectId", SqlDbType.VarChar);
command.Parameters["#ProjectId"].Value = dtoItem.ProjectId;
connection.Open();
Result.EnforceConstraints = false;
Result.Load(command.ExecuteReader(CommandBehavior.CloseConnection), LoadOption.OverwriteChanges, Tables);
dtoItem.test = Result;
}
}
return dtoItem;
}
Update:
I have updated the code to the below. It just hangs when I run it and only writes 1/4 of the data to the server? Can you please let me know what I am doing wrong?
public static ITargetBlock<T> CreateGuaranteedBroadcastBlock<T>(IEnumerable<ITargetBlock<T>> targets, DataflowBlockOptions options)
{
var targetsList = targets.ToList();
var block = new ActionBlock<T>(
async item =>
{
foreach (var target in targetsList)
{
await target.SendAsync(item);
}
}, new ExecutionDataflowBlockOptions
{
CancellationToken = options.CancellationToken
});
block.Completion.ContinueWith(task =>
{
foreach (var target in targetsList)
{
if (task.Exception != null)
target.Fault(task.Exception);
else
target.Complete();
}
});
return block;
}
[HttpGet]
public async Task< HttpResponseMessage> ReloadItem(string projectQuery)
{
try
{
var linkCompletion = new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 2
};
var cts = new CancellationTokenSource();
var dbOptions = new DataflowBlockOptions { CancellationToken = cts.Token };
IList<string> projectIds = projectQuery.Split(',').ToList();
IEnumerable<string> serverList = ConfigurationManager.AppSettings["ServerList"].Split(',').Cast<string>();
var iR = new TransformBlock<MemcachedDTO, MemcachedDTO>(
dto => dto.GetIR(dto), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3 });
List<ActionBlock<MemcachedDTO>> actionList = new List<ActionBlock<MemcachedDTO>>();
List<MemcachedDTO> dtoList = new List<MemcachedDTO>();
foreach (string pid in projectIds)
{
IList<MemcachedDTO> dtoTemp = new List<MemcachedDTO>();
dtoTemp = MemcachedDTO.GetItemIdsByProject(pid);
dtoList.AddRange(dtoTemp);
}
foreach (string s in serverList)
{
var action = new ActionBlock<MemcachedDTO>(
async dto => await PostEachServerAsync(dto, s, "setitemcache"));
actionList.Add(action);
}
var bBlock = CreateGuaranteedBroadcastBlock(actionList, dbOptions);
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
iR.Complete();
iR.LinkTo(bBlock);
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
return Request.CreateResponse(HttpStatusCode.OK, new { message = projectIds.ToString() + " reload success" });
}
catch (Exception ex)
{
return Request.CreateResponse(HttpStatusCode.InternalServerError, new { message = ex.Message.ToString() });
}
}
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
Yes in the code below you set BoundedCapacity to one, if at anytime your BroadcastBlock cannot pass along an item it will drop it. Additionally a BroadcastBlock will only propagate Completion to one TargetBlock, do not use PropagateCompletion=true here. If you want all blocks to complete you need to handle Completion manually. This can be done by setting the ContinueWith on the BroadcastBlock to pass Completion to all of the connected targets.
var action = new ActionBlock<MemcachedDTO>(dto => PostEachServerAsync(dto, s, "set"), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = 1 });
broadcast.LinkTo(action, linkCompletion);
actionList.Add(action);
Option: Instead of the BroadcastBlock use a properly bounded BufferBlock. When your downstream blocks are bound to one item they cannot receive additional items until they finish processing what they have. That will allow the BufferBlock to offer its items to another, possibly idle, ActionBlock.
When you add items into a throttled flow, i.e. a flow with a BoundedCapacity less than Unbounded. You need to be using the SendAsync method or at least handling the return of Post. I'd recommend simply using SendAsync:
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
That will force your method signature to become:
public async Task<HttpResponseMessage> ReloadItem(string projectQuery)
2) Am I doing the await on all actionBlocks correctly?
The previous change will permit you to loose the blocking Wait call in favor of a await Task.WhenAlll
iR.Complete();
actionList.ForEach(x => x.Completion.Wait());
To:
iR.Complete();
await bufferBlock.Completion.ContinueWith(tsk => actionList.ForEach(x => x.Complete());
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
3) How do I make the database call async?
I'm going to leave this open because it should be a separate question unrelated to TPL-Dataflow, but in short use an async Api to access your Db and async will naturally grow through your code base. This should get you started.
BufferBlock vs BroadcastBlock
After re-reading your previous question and the answer from #VMAtm. It seems you want each item sent to All five servers, in that case you will need a BroadcastBlock. You would use a BufferBlock to distribute the messages relatively evenly to a flexible pool of servers that each could handle a message. None the less, you will still need to take control of propagating completion and faults to all the connected ActionBlocks by awaiting the completion of the BroadcastBlock.
To Prevent BroadcastBlock Dropped Messages
In general you two options, set your ActionBlocks to be unbound, which is their default value:
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = Unbounded });
Or broadcast messages your self from any variety of your own construction. Here is an example implementation from #i3arnon. And another from #svick
I'm new to TPL Dataflow and I'm looking for a construct which will allow splitting up a list of source messages for evenly distributed parallel processing while maintaining order of the messages message through individual pipelines. Is there a specific Block or concept within the DataFlow API that can be used to accomplish this or is it more of a matter providing glue code or custom Blocks between existing Blocks?
For those familiar with Akka.NET I'm looking for functionality similar to the ConsistentHashing router which allow sending messages to a single router which then forwards these messages on to individual routees to be handled.
Synchronous example:
var count = 100000;
var processingGroups = 5;
var source = Enumerable.Range(1, count);
// Distribute source elements consistently and evenly into a specified set of groups (ex. 5) so that.
var distributed = source.GroupBy(s => s % processingGroups);
// Within each of the 5 processing groups go through each item and add 1 to it
var transformed = distributed.Select(d => d.Select(i => i + 3).ToArray());
List<int[]> result = transformed.ToList();
Check.That(result.Count).IsEqualTo(processingGroups);
for (int i = 0; i < result.Count; i++)
{
var outputGroup = result[i];
var expectedRange = Enumerable.Range(i + 1, count/processingGroups).Select((e, index) => e + (index * (processingGroups - 1)) + 3);
Check.That(outputGroup).ContainsExactly(expectedRange);
}
In general I don't think what you're looking for is pre-made in Dataflow as it may be with a ConsistentHashing router. However, by adding an id to the pieces of data you wish to flow you can process them in any order, in parallel and reorder them when the processing finishes.
public class Message {
public int MessageId { get; set; }
public int GroupId { get; set; }
public int Value { get; set; }
}
public class MessageProcessing
{
public void abc() {
var count = 10000;
var groups = 5;
var source = Enumerable.Range(0, count);
//buffer all input
var buffer = new BufferBlock<IEnumerable<int>>();
//split each input enumerable into processing groups
var messsageProducer = new TransformManyBlock<IEnumerable<int>, Message>(ints =>
ints.Select((i, index) => new Message() { MessageId = index, GroupId = index % groups, Value = i }).ToList());
//process each message, one action block may process any group id in any order
var processMessage = new TransformBlock<Message, Message>(msg =>
{
msg.Value++;
return msg;
}, new ExecutionDataflowBlockOptions() {
MaxDegreeOfParallelism = groups
});
//output of processed message values
int[] output = new int[count];
//insert messages into array in the order the started in
var regroup = new ActionBlock<Message>(msg => output[msg.MessageId] = msg.Value,
new ExecutionDataflowBlockOptions() {
MaxDegreeOfParallelism = 1
});
}
}
In the example the GroupId of a message isn't used but it could be used in a more complete example for coordinating groups of messages. Also, handling follow up posts to the bufferblock could be done by changing the output array to a List and setting up a corresponding list element each time an enumerable of integers is posted to the buffer block. Depending on your exact use, you may need to support multiple users of the output, and this can be folded back into the flow.
You can dynamically create a pipeline with linking the blocks between each other based on predicate:
var count = 100;
var processingGroups = 5;
var source = Enumerable.Range(1, count);
var buffer = new BufferBlock<int>();
var consumer1 = new ActionBlock<int>(i => { });
var consumer2 = new ActionBlock<int>(i => { });
var consumer3 = new ActionBlock<int>(i => { });
var consumer4 = new ActionBlock<int>(i => { Console.WriteLine(i); });
var consumer5 = new ActionBlock<int>(i => { });
buffer.LinkTo(consumer1, i => i % 5 == 1);
buffer.LinkTo(consumer2, i => i % 5 == 2);
buffer.LinkTo(consumer3, i => i % 5 == 3);
buffer.LinkTo(consumer4, i => i % 5 == 4);
buffer.LinkTo(consumer5);
foreach (var i in source)
{
buffer.Post(i);
// consider async option if you able to do it
// await buffer.SendAsync(i);
}
buffer.Complete();
Console.ReadLine();
The code above will write only numbers from 4th group, processing other groups silently, but I hope you got the idea. There is a general practice to link a block for at least one consumer without filtering for messages not being dropped if they aren't accepted by any consumers, and you can do this if you don't have a default handler (NullTarget<int> simply ignores all the messages it got):
buffer.LinkTo(DataflowBlock.NullTarget<int>());
The downside of this is a continuation of it's advantages: you have to provide predicates, as there is no built-in structures for this. However, it still could be done.
As described here TypeScript introduces a foreach loop:
var someArray = [9, 2, 5];
for (var item of someArray) {
console.log(item); // 9,2,5
}
But isn't there any index/key? I would expect something like:
for (var item, key of someArray) { ... }
.forEach already has this ability:
const someArray = [9, 2, 5];
someArray.forEach((value, index) => {
console.log(index); // 0, 1, 2
console.log(value); // 9, 2, 5
});
But if you want the abilities of for...of, then you can map the array to the index and value:
for (const { index, value } of someArray.map((value, index) => ({ index, value }))) {
console.log(index); // 0, 1, 2
console.log(value); // 9, 2, 5
}
That's a little long, so it may help to put it in a reusable function:
function toEntries<T>(a: T[]) {
return a.map((value, index) => [index, value] as const);
}
for (const [index, value] of toEntries(someArray)) {
// ..etc..
}
Iterable Version
This will work when targeting ES3 or ES5 if you compile with the --downlevelIteration compiler option.
function* toEntries<T>(values: T[] | IterableIterator<T>) {
let index = 0;
for (const value of values) {
yield [index, value] as const;
index++;
}
}
Array.prototype.entries() - ES6+
If you are able to target ES6+ environments then you can use the .entries() method as outlined in Arnavion's answer.
See: Array.prototype.entries()
for (const [key, item] of someArray.entries()) { ... }
In TS this requires targeting ES2015 since it requires the runtime to support iterators, which ES5 runtimes don't. You can of course use something like Babel to make the output work on ES5 runtimes.
"Old school javascript" to the rescue (for those who aren't familiar/in love of functional programming)
for (let i = 0; i < someArray.length ; i++) {
let item = someArray[i];
}
You can use the for..in TypeScript operator to access the index when dealing with collections.
var test = [7,8,9];
for (var i in test) {
console.log(i + ': ' + test[i]);
}
Output:
0: 7
1: 8
2: 9
See Demo
Or another old school solution:
var someArray = [9, 2, 5];
let i = 0;
for (var item of someArray) {
console.log(item); // 9,2,5
i++;
}