I have a BufferBlock linked to two Target Blocks. The dataflow does not complete. I have followed the suggestions from this post, but I can't get the completion propagation right.
Any help would be appreciated.
// define blocks
var bufferBlock = new BufferBlock<int>();
var actionBlock1 = new TransformBlock<int, int>(i =>
{
Console.WriteLine($"actionBlock1: {i}");
return i;
});
var actionBlock2 = new ActionBlock<int>(i =>
{
Console.WriteLine($"actionBlock2: {i}");
});
// link blocks
bufferBlock.LinkTo(actionBlock1, i => i == 1);
bufferBlock.LinkTo(actionBlock2, i => i == 2);
bufferBlock.LinkTo(DataflowBlock.NullTarget<int>());
// push to block
var items = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
foreach (var i in items)
{
bufferBlock.Post(i);
}
// wait for all operations to complete
bufferBlock.Complete();
// NB: block will only propagate completion to one block regardless of how many blocks are linked. This even applies to the BroadcastBlock that broadcasts messages, it will not broadcast completion. In that case you can configure a continuation on the Completion Task
// see https://stackoverflow.com/questions/47310402/tpl-complete-vs-completion/47311537#47311537
var bufferBlockCompletion = bufferBlock.Completion.ContinueWith(tsk =>
{
if (!tsk.IsFaulted)
{
actionBlock1.Complete();
actionBlock2.Complete();
}
else
{
((IDataflowBlock)actionBlock1).Fault(tsk.Exception);
((IDataflowBlock)actionBlock2).Fault(tsk.Exception);
}
});
await Task.WhenAll(bufferBlockCompletion, actionBlock1.Completion, actionBlock2.Completion);
actonBlock1 is a TransformBlock that is not linked to anything. Any items that the block produces will remain in its output buffer, in this cas only the number 1. With items stuck in the output the block can never complete. You can fix that a couple of different ways depending on what exactly you need.
1) Change the TransformBlock to an ActionBlock
2) Link the TransformBlock to a NullTarget or another block.
Related
So I was solving a code challenge where the input was given as array of connections:
[
[2, 9],
[7, 2],
[7, 9],
[9, 5],
]
where the format of the array is [fromNode,toNode]. I tried to solve the problem with time complexity of O(N^2) however it seemed to have failed some unknown test cases that I couldn't think of.
Here's what I did:
const buildConnections = ({ toID, fromID, allConnections }) => {
const directConnectionNodes = {}; // Nodes which are directly connected to `toNode`
const mutualConnections = {}; // Nodes and their respective connections (Directly or Indirectly)
allConnections.forEach(([from, to]) => {
// O(N)
if (to == toID) {
directConnectionNodes[from] = true; // Adding the node to direct connections
} else {
const keys = Object.keys(mutualConnections);
// This step is basically to update existing node if the connections are updated
// Example First pair is [3,5] which would result in {5:{3}}
// Second pair is [2,3] which would result in {3:{2} & 5:{...allPairsOf3}=5:{3,2}}
// => Since 3 had a new connection with 2
keys.forEach((key) => {
// O(N*N)
const value = mutualConnections[key];
if (value[to] !== undefined) {
mutualConnections[key] = {
// O(N*N*N)
...mutualConnections[key],
[from]: true,
};
}
});
if (mutualConnections[from]) {
mutualConnections[to] = {
...mutualConnections[from],
[from]: true,
};
} else mutualConnections[to] = { [from]: true };
}
});
const directConnectionNodeKeys = Object.keys(directConnectionNodes);
const results = [];
// Here I check for disconnecting each node that could possibly connect the two nodes together
directConnectionNodeKeys.forEach((key) => {
if (
key == fromID ||
(mutualConnections[key] && mutualConnections[key][fromID])
)
results.push(key);
});
console.log(results.join(' '));
}
I had two concerns here:
First thing is that the algorithm I wrote assuming that there's a big graph with lots of edges and vertices, could go O(NMN) which is like (N^3) which is pretty bad, can someone help me figure out a more optimized way to do so?
Second thing is that what possible edge cases could I be missing?
I´ve accomplished the react drag and drop functionality into my project so i can reorder a row in a react table´s list. The problem is i have a column named 'Sequence', witch shows me the order of the elements, that i can´t update its values.
Example:
before (the rows are draggable):
Sequence | Name
1 Jack
2 Angel
after ( i need to update the values of Sequence wherea i change their position after dropping a specific draggable row, in this case i dragged Jack at the first position and dropped it at the second position) :
Sequence | Name
1 Angel
2 Jack
React/Redux it´s allowing me to change the index order of this array of elements, without getting the 'A state mutation was detected between dispatches' error message, but is not allowing me to update the Sequence values with a new order values.
This is what i have tried so far:
// within the parent class component
// item is an array of objects from child
UpdateSequence(startIndex, endIndex, item) {
// the state.Data is already an array of object
const result = this.state.Data;
const [removed] = result.splice(startIndex, 1);
result.splice(endIndex, 0, removed);
// this is working without the mutation state error
this.setState({ Data: result })
let positionDiff = 0;
let direction = null;
let newIndex = 0;
positionDiff = endIndex - startIndex;
if (startIndex > endIndex) {
direction = "up";
}
else if (startIndex < endIndex) {
direction = "down";
}
if (positionDiff !== 0) {
for (var x = 0; x <= Math.abs(positionDiff); x++) {
if (x === 0) {
newIndex = startIndex + positionDiff - x;
this.setState(prevState => ({
Data: {
...prevState.Data,
[prevState.Data[newIndex].Sequence]: Data[newIndex].Sequence + positionDiff
},
}));
}
else {
if (direction === "down") {
newIndex = startIndex + positionDiff - x;
this.setState(prevState => ({
Data: {
...prevState.Data,
[prevState.Data[newIndex].Sequence]: Data[newIndex].Sequence - 1
},
}));
}
else if (direction === "up") {
Data= startIndex + positionDiff + x;
this.setState(prevState => ({
Data: {
...prevState.Data,
[prevState.Data[newIndex].Sequence]: Data[newIndex].Sequence + 1
},
}));
}
}
}
// so when i call save action i am stepping into the 'A state mutation was detected between dispatches' error message.
this.props.actions.saveSequence(this.state.Data)
.then(() => {
this.props.actions.loadData();
})
.catch(error => {
toastr['error'](error, 'error....');
})
}
Calling the action 'saveSequence' whenever i try to update the element of the array, 'Sequence', i am getting the 'A state mutation was detected between dispatches' error message.
Any help will be greatfull! Thank you!
note: The logic applied to reorder the Sequence is ok.
While I don't know redux particularly well, I am noticing that you are directly modifying state, which seems like a likely culprit.
const result = this.state.Data;
const [removed] = result.splice(startIndex, 1);
splice is a destructive method that modifies its input, and its input is a reference to something in this.state.
To demonstrate:
> state = {Data: [1,2,3]}
{ Data: [ 1, 2, 3 ] }
> result = state.Data.splice(0,1)
[ 1 ]
> state
{ Data: [ 2, 3 ] }
Notice that state has been modified. This might be what Redux is detecting, and a general React no-no.
To avoid modifying state, the easy way out is to clone the data you are looking to modify
const result = this.state.Data.slice()
Note that this does a shallow copy, so if Data has non-primitive values, you have to watch out for doing destructive edits on those values too. (Look up deep vs shallow copy if you want to find out more.) However, since you are only reordering things, I believe you're safe.
Well, i figured it out changing this part of code:
//code....
const result = item;
const [removed] = result.splice(startIndex, 1);
// i created a new empty copy of the const 'removed', called 'copy' and update the Sequence property of the array like this below. (this code with the sequence number is just a sample of what i came up to fix it )
let copy;
copy = {
...removed,
Sequence: 1000,
};
result.splice(endIndex, 0, copy);
After i didn´t setState for it, so i commented this line:
// this.setState({ Data: result })
//...code
and the end of it was putting the result to the save action as a parameter , and not the state.
this.props.actions.saveSequence(result)
Works and now i have i fully drag and drop functionality saving the new order sequence into the database with no more 'A state mutation was detected between dispatches' error message!
I have some Subject. And one Observer subscribed to it. How to omit all Observer invocations if it is already processing one?
var subject = new Subject();
var observer = {
next: x => {
//... some long processing is here
console.log('Observer got a next value: ' + x)
}
};
subject.subscribe(observer);
subject.next(0);
subject.next(1);// <-- if 0 value is not processed in the observer then skip it
subject.next(2);// <-- if 0 value is not processed in the observer then skip it
I of cause can introduce some flag, set it in Observer before execution and clear it after. And apply filter operator, like this:
var subject = new Subject();
var flag = true;
var observer = {
next: x => {
flag = false;
//... some long processing is here
console.log('Observer got a next value: ' + x)
flag = true;
}
};
subject.filter(() => flag).subscribe(observer);
subject.next(0);
subject.next(1);// <-- if previous value is not processed in the observer then skip it
subject.next(2);// <-- if 0 value is not processed in the observer then skip it
But I believe that exists more elegant and efficient way to achieve that.
Use the exhaustMap operator instead of trying roll your own backpressure. It is designed to ignore new events while waiting for the current one to complete.
const clicks = fromEvent(document, 'click');
const result = clicks.pipe(
exhaustMap((ev) => interval(1000).pipe(take(5))),
);
result.subscribe(x => console.log(x));
I have some code that saves data using Breeze and reports progress over multiple saves that is working reasonably well.
However, sometimes a save will timeout, and I'd like to retry it once automatically. (Currently the user is shown an error and has to retry manually)
I am struggling to find an appropriate way to do this, but I am confused by promises, so I'd appreciate some help.
Here is my code:
//I'm using Breeze, but because the save takes so long, I
//want to break the changes down into chunks and report progress
//as each chunk is saved....
var surveys = EntityQuery
.from('PropertySurveys')
.using(manager)
.executeLocally();
var promises = [];
var fails = [];
var so = new SaveOptions({ allowConcurrentSaves: false});
var count = 0;
//...so I iterate through the surveys, creating a promise for each survey...
for (var i = 0, len = surveys.length; i < len; i++) {
var query = EntityQuery.from('AnsweredQuestions')
.where('PropertySurveyID', '==', surveys[i].ID)
.expand('ActualAnswers');
var graph = manager.getEntityGraph(query)
var changes = graph.filter(function (entity) {
return !entity.entityAspect.entityState.isUnchanged();
});
if (changes.length > 0) {
promises.push(manager
.saveChanges(changes, so)
.then(function () {
//reporting progress
count++;
logger.info('Uploaded ' + count + ' of ' + promises.length);
},
function () {
//could I retry the fail here?
fails.push(changes);
}
));
}
}
//....then I use $q.all to execute the promises
return $q.all(promises).then(function () {
if (fails.length > 0) {
//could I retry the fails here?
saveFail();
}
else {
saveSuccess();
}
});
Edit
To clarify why I have been attempting this:
I have an http interceptor that sets a timeout on all http requests. When a request times out, the timeout is adjusted upwards, the user is displayed an error message, telling them they can retry with a longer wait if they wish.
Sending all the changes in one http request is looking like it could take several minutes, so I decided to break the changes down into several http requests, reporting progress as each request succeeds.
Now, some requests in the batch might timeout and some might not.
Then I had the bright idea that I would set a low timeout for the http request to start with and automatically increase it. But the batch is sent asynchronously with the same timeout setting and the time is adjusted for each failure. That is no good.
To solve this I wanted to move the timeout adjustment after the batch completes, then also retry all requests.
To be honest I'm not so sure an automatic timeout adjustment and retry is such a great idea in the first place. And even if it was, it would probably be better in a situation where http requests were made one after another - which I've also been looking at: https://stackoverflow.com/a/25730751/150342
Orchestrating retries downstream of $q.all() is possible but would be very messy indeed. It's far simpler to perform retries before aggregating the promises.
You could exploit closures and retry-counters but it's cleaner to build a catch chain :
function retry(fn, n) {
/*
* Description: perform an arbitrary asynchronous function,
* and, on error, retry up to n times.
* Returns: promise
*/
var p = fn(); // first try
for(var i=0; i<n; i++) {
p = p.catch(function(error) {
// possibly log error here to make it observable
return fn(); // retry
});
}
return p;
}
Now, amend your for loop :
use Function.prototype.bind() to define each save as a function with bound-in parameters.
pass that function to retry().
push the promise returned by retry().then(...) onto the promises array.
var query, graph, changes, saveFn;
for (var i = 0, len = surveys.length; i < len; i++) {
query = ...; // as before
graph = ...; // as before
changes = ...; // as before
if (changes.length > 0) {
saveFn = manager.saveChanges.bind(manager, changes, so); // this is what needs to be tried/retried
promises.push(retry(saveFn, 1).then(function() {
// as before
}, function () {
// as before
}));
}
}
return $q.all(promises)... // as before
EDIT
It's not clear why you might want to retry downsteam of $q.all(). If it's a matter of introducing some delay before retrying, the simplest way would be to do within the pattern above.
However, if retrying downstream of $q.all() is a firm requirement, here's a cleanish recursive solution that allows any number of retries, with minimal need for outer vars :
var surveys = //as before
var limit = 2;
function save(changes) {
return manager.saveChanges(changes, so).then(function () {
return true; // true signifies success
}, function (error) {
logger.error('Save Failed');
return changes; // retry (subject to limit)
});
}
function saveChanges(changes_array, tries) {
tries = tries || 0;
if(tries >= limit) {
throw new Error('After ' + tries + ' tries, ' + changes_array.length + ' changes objects were still unsaved.');
}
if(changes_array.length > 0) {
logger.info('Starting try number ' + (tries+1) + ' comprising ' + changes_array.length + ' changes objects');
return $q.all(changes_array.map(save)).then(function(results) {
var successes = results.filter(function() { return item === true; };
var failures = results.filter(function() { return item !== true; }
logger.info('Uploaded ' + successes.length + ' of ' + changes_array.length);
return saveChanges(failures), tries + 1); // recursive call.
});
} else {
return $q(); // return a resolved promise
}
}
//using reduce to populate an array of changes
//the second parameter passed to the reduce method is the initial value
//for memo - in this case an empty array
var changes_array = surveys.reduce(function (memo, survey) {
//memo is the return value from the previous call to the function
var query = EntityQuery.from('AnsweredQuestions')
.where('PropertySurveyID', '==', survey.ID)
.expand('ActualAnswers');
var graph = manager.getEntityGraph(query)
var changes = graph.filter(function (entity) {
return !entity.entityAspect.entityState.isUnchanged();
});
if (changes.length > 0) {
memo.push(changes)
}
return memo;
}, []);
return saveChanges(changes_array).then(saveSuccess, saveFail);
Progress reporting is slightly different here. With a little more thought it could be made more like in your own answer.
This is a very rough idea of how to solve it.
var promises = [];
var LIMIT = 3 // 3 tris per promise.
data.forEach(function(chunk) {
promises.push(tryOrFail({
data: chunk,
retries: 0
}));
});
function tryOrFail(data) {
if (data.tries === LIMIT) return $q.reject();
++data.tries;
return processChunk(data.chunk)
.catch(function() {
//Some error handling here
++data.tries;
return tryOrFail(data);
});
}
$q.all(promises) //...
Two useful answers here, but having worked through this I have concluded that immediate retries is not really going to work for me.
I want to wait for the first batch to complete, then if the failures are because of timeouts, increase the timeout allowance, before retrying failures.
So I took Juan Stiza's example and modified it to do what I want. i.e. retry failures with $q.all
My code now looks like this:
var surveys = //as before
var successes = 0;
var retries = 0;
var failedChanges = [];
//The saveChanges also keeps a track of retries, successes and fails
//it resolves first time through, and rejects second time
//it might be better written as two functions - a save and a retry
function saveChanges(data) {
if (data.retrying) {
retries++;
logger.info('Retrying ' + retries + ' of ' + failedChanges.length);
}
return manager
.saveChanges(data.changes, so)
.then(function () {
successes++;
logger.info('Uploaded ' + successes + ' of ' + promises.length);
},
function (error) {
if (!data.retrying) {
//store the changes and resolve the promise
//so that saveChanges can be called again after the call to $q.all
failedChanges.push(data.changes);
return; //resolved
}
logger.error('Retry Failed');
return $q.reject();
});
}
//using map instead of a for loop to call saveChanges
//and store the returned promises in an array
var promises = surveys.map(function (survey) {
var changes = //as before
return saveChanges({ changes: changes, retrying: false });
});
logger.info('Starting data upload');
return $q.all(promises).then(function () {
if (failedChanges.length > 0) {
var retries = failedChanges.map(function (data) {
return saveChanges({ changes: data, retrying: true });
});
return $q.all(retries).then(saveSuccess, saveFail);
}
else {
saveSuccess();
}
});
I have decent large data set of around 1100 records. This data set is mapped to an observable array which is then bound to a view. Since these records are updated frequently, the observable array is updated every time using the ko.mapping.fromJS helper.
This particular command takes around 40s to process all the rows. The user interface just locks for that period of time.
Here is the code -
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
ko.mapping.fromJS(data,transactionList)
Is there a workaround for this? Or should I just opt of web workers to improve performances?
Knockout.viewmodel is a replacement for knockout.mapping that is significantly faster at creating viewmodels for large object arrays like this. You should notice a significant performance increase.
http://coderenaissance.github.com/knockout.viewmodel/
I have also thought of a workaround as follows, this uses less amount of code-
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
// Instead of - ko.mapping.fromJS(data,transactionList)
var i = 0;
//clear the list completely first
transactionList.destroyAll();
//Set an interval of 0 and keep pushing the content to the list one by one.
var interval = setInterval(function () {if (i == data.length - 1 ) {
clearInterval(interval);}
transactionList.push(ko.mapping.fromJS(data[i++]));
}, 0);
I had the same problem with mapping plugin. Knockout team says that mapping plugin is not intended to work with large arrays. If you have to load such big data to the page then likely you have improper design of the system.
The best way to fix this is to use server pagination instead of loading all the data on page load. If you don't want to change design of your application there are some workarounds which maybe help you:
Map your array manually:
var data = storage.transactions();
var mappedData = ko.utils.arrayMap(data , function(item){
return ko.mapping.fromJS(item);
});
var transactionList = ko.observableArray(mappedData);
Map array asynchronously. I have written a function that processes array by portions in another thread and reports progress to the user:
function processArrayAsync(array, itemFunc, afterStepFunc, finishFunc) {
var itemsPerStep = 20;
var processor = new function () {
var self = this;
self.array = array;
self.processedCount = 0;
self.itemFunc = itemFunc;
self.afterStepFunc = afterStepFunc;
self.finishFunc = finishFunc;
self.step = function () {
var tillCount = Math.min(self.processedCount + itemsPerStep, self.array.length);
for (; self.processedCount < tillCount; self.processedCount++) {
self.itemFunc(self.array[self.processedCount], self.processedCount);
}
self.afterStepFunc(self.processedCount);
if (self.processedCount < self.array.length - 1)
setTimeout(self.step, 1);
else
self.finishFunc();
};
};
processor.step();
};
Your code:
var data = storage.transactions();
var transactionList = ko.observableArray([]);
processArrayAsync(data,
function (item) { // Step function
var transaction = ko.mapping.fromJS(item);
transactionList().push(transaction);
},
function (processedCount) {
var percent = Math.ceil(processedCount * 100 / data.length);
// Show progress to the user.
ShowMessage(percent);
},
function () { // Final function
// This function will fire when all data are mapped. Do some work (i.e. Apply bindings).
});
Also you can try alternative mapping library: knockout.wrap. It should be faster than mapping plugin.
I have chosen the second option.
Mapping is not magic. In most of the cases this simple recursive function can be sufficient:
function MyMapJS(a_what, a_path)
{
a_path = a_path || [];
if (a_what != null && a_what.constructor == Object)
{
var result = {};
for (var key in a_what)
result[key] = MyMapJS(a_what[key], a_path.concat(key));
return result;
}
if (a_what != null && a_what.constructor == Array)
{
var result = ko.observableArray();
for (var index in a_what)
result.push(MyMapJS(a_what[index], a_path.concat(index)));
return result;
}
// Write your condition here:
switch (a_path[a_path.length-1])
{
case 'mapThisProperty':
case 'andAlsoThisOne':
result = ko.observable(a_what);
break;
default:
result = a_what;
break;
}
return result;
}
The code above makes observables from the mapThisProperty and andAlsoThisOne properties at any level of the object hierarchy; other properties are left constant. You can express more complex conditions using a_path.length for the level (depth) the value is at, or using more elements of a_path. For example:
if (a_path.length >= 2
&& a_path[a_path.length-1] == 'mapThisProperty'
&& a_path[a_path.length-2] == 'insideThisProperty')
result = ko.observable(a_what);
You can use typeOf a_what in the condition, e.g. to make all strings observable.
You can ignore some properties, and insert new ones at certain levels.
Or, you can even omit a_path. Etc.
The advantages are:
Customizable (more easily than knockout.mapping).
Short enough to copy-paste it and write individual mappings for different objects if needed.
Smaller code, knockout.mapping-latest.js is not included into your page.
Should be faster as it does only what is absolutely necessary.