InfluxDB 2.0 Flux - How to Handle Division by Zero - flux

Hi I am trying to do a simple success rate calculation between metrics. By dividing the number of successful requests by the number of attempts. The problem is some intervals are empty where both metrics are 0. When I write the query I get the below “cannot divide by zero” runtime error. In SQL there a NULLIF function to avoid this. Is there something similar in flux or is there an alternate method to avoid division by zero?
Error: runtime error #7:6-7:90: map: failed to evaluate map function: cannot divide by zero
My Sample Query:
from(bucket: "my_db")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "HTTPRequests")
|> filter(fn: (r) => r["_field"] == "RequestAttempt" or r["_field"] == "RequestSuccess_E")
|> filter(fn: (r) => r["host"] == "host-a")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> map(fn: (r) => ({ r with Request_SR: r.RequestSuccess_E/r.RequestAttempt }))
Thanks in advance.

Influx Team Answered my Q. This worked for my case.
You could use some simple conditional logic to check the values of ReqeustSuccess and RequestAttempt. You can check if they’re null or 0. You do need to provide a default value for when the operands or null or zero because you can’t map a null value.
from(bucket: "my_db")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "HTTPRequests")
|> filter(fn: (r) => r["_field"] == "RequestAttempt" or r["_field"] == "RequestSuccess_E")
|> filter(fn: (r) => r["host"] == "host-a")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> map(fn: (r) => ({ r with
Request_SR:
if not exists r.RequestSuccess_E or r.RequestSuccess_E == 0.0 then 0.0
else if not exists r.RequestAttempt or r.RequestAttempt == 0.0 then 0.0
else r.RequestSuccess_E / r.RequestAttempt
}))

Related

F# Array.Parallel.map does not provide parallel processing

I have to simulate a discrete environment in F#, to be called by Python, for a reinforcement learning problem. I had a function with primitive types (mainly float) to make the exchange of data smoother. Now I am in the position to run this function many times with different data, so to run it in parallel seems a good idea.
I have the following code:
type AscentStrategy = |Strategy of seq<float>
let simulateAscent env ascentLimiter initState (sequenceOfDepths:seq<float>) =
//let infinitSeqOfConstantValues = (fun _ -> constantDepth) |> Seq.initInfinite
sequenceOfDepths
|> Seq.scan ( fun ( nextState, rew, isTerminal, _ ) depth -> getNextEnvResponseAndBoundForNextAction(env, nextState , depth , ascentLimiter) ) ( initState, 0.0 , false, 0.0)
|> SeqExtension.takeWhileWithLast (fun (_ , _, isTerminalState, _) -> not isTerminalState)
|> Seq.toArray
and then
let simulateStrategy ({MaxPDCS = maxPDCS ; MaxSimTime = maximumSimulationTime ; PenaltyForExceedingRisk = penaltyForExceedingRisk ;
RewardForDelivering = rewardForDelivering ; PenaltyForExceedingTime = penaltyForExceedingTime ; IntegrationTime = integrationTime
ControlToIntegrationTimeRatio = controlToIntegrationTimeRatio; DescentRate = descentRate; MaximumDepth = maximumDepth ;
BottomTime = bottomTime ; LegDiscreteTime = legDiscreteTime } : SimulationParameters) (Strategy ascentStrategy : AscentStrategy) =
let env, initState , ascentLimiter , _ = getEnvInitStateAndAscentLimiter ( maxPDCS , maximumSimulationTime ,
penaltyForExceedingRisk , rewardForDelivering , penaltyForExceedingTime ,
integrationTime ,
controlToIntegrationTimeRatio,
descentRate ,
maximumDepth ,
bottomTime ,
legDiscreteTime )
ascentStrategy
|> simulateAscent env ascentLimiter initState
finally I call the function for testing:
let commonSimulationParameters = {MaxPDCS = 0.32 ; MaxSimTime = 2000.0 ; PenaltyForExceedingRisk = 1.0 ; RewardForDelivering = 10.0; PenaltyForExceedingTime = 0.5 ;
IntegrationTime = 0.1; ControlToIntegrationTimeRatio = 10; DescentRate = 60.0; MaximumDepth = 20.0 ; BottomTime = 10.0; LegDiscreteTime = 0.1}
printfn"insert number of elements"
let maxInputsString = Console.ReadLine()
let maxInputs = maxInputsString |> Double.Parse
let inputsStrategies = [|0.0 .. maxInputs|] |> Array.map (fun x -> Seq.initInfinite (fun _ -> x ) )
let testParallel = inputsStrategies
|> Array.Parallel.map (fun x -> (simulateStrategy commonSimulationParameters ( Strategy x )) )
I have compared this with Array.map and, while it is faster and uses 70% of the CPU on my laptop, still does not seem to use the whole processing power. I have run it on a machine with many more cores ( ~50) and it barely increases the CPU usage (it gets up to 3/4% of total usage with 50ish independent inputs). I think there must be a deadlock generated somewhere, but how can I detect and get rid of it?
Also, why does this happen? One of the advantages of functional programming, as I see it, is also to be able to parallelize easily.
PS: SeqExtension.takeWhileWithLast is a function I have found on SO, kindly provided by Tomas Petricek in one of his brilliant answers, if needed I can post it.
PPS: env is the environment, whose type is defined as:
type Environment<'S, 'A ,'I> = |Environment of (State<'S> -> Action<'A> -> EnvironmentOutput<'S ,'I>)
I have tried the same with Async.Parallel and ParallelSeq, reporting the same problem.
Would a message-based solution solve the problem>? I am looking into it, although I am not familiar at all, but would it be a good way of getting the code parallel, using MailboxProcessor?
Following my question,
I have tried also this great library for parallel code, based on streams of data. https://nessos.github.io/Streams/.
I have added the following code:
let nessosResult = inputsStrategies
|> ParStream.ofArray
|> ParStream.map simulateStrategy
|> ParStream.toArray
I have defined an ad hoc type for inputStrategy (basic the old tuple I had) so that simulateStrategy accepts only one input. Unfortunately the problem seems very well hidden somewhere. I attach a graph with CPU usage. Time spent on my machine for the different cases is: ~8.8 sec (sequential); ~6.2 sec (Array.Parallel.map); ~ 6.1 sec (Nessos.Streams)
I have found that server garbage collection is necessary to get the best parallel performance on .NET. Something like this in your app.config:
<configuration>
<runtime>
<gcServer enabled="true" />
</runtime>
</configuration>

Use Time.now to get time zoned date in Elm 0.18

posixDateTask : Task.Task x Time.Date.Date
posixDateTask =
let
timeZoneDate now =
Time.Date.date
(Time.ZonedDateTime.year (Time.ZonedDateTime.fromDateTime (TimeZones.canada_pacific ()) now))
(Time.ZonedDateTime.month (Time.ZonedDateTime.fromDateTime (TimeZones.canada_pacific ()) now))
(Time.ZonedDateTime.day (Time.ZonedDateTime.fromDateTime (TimeZones.canada_pacific ()) now))
in
Time.now
|> Task.map timeZoneDate
This gives me an error:
|> Task.map timeZoneDate
(|>) is expecting the right side to be a:
Task.Task x Time.Time -> a
But the right side is:
Task.Task x Time.DateTime.DateTime -> Task.Task x Time.Date.Date
How should I change to return a Task.Task x Time.Date.Date type.
I don't know where is the Time.DateTime.DateTime come from.
Time.now returns a task with a Time.Time, which is what you pass to timeZoneDate as now. You then pass now to Time.ZonedDateTime.fromDateTime, which expects a Time.DateTime.DateTime (which shouldn't be entirely surprising given its name.) You therefore have to convert now from Time.Time to Time.DateTime.DateTime. It seems you can do that using Time.DateTime.fromTimestamp.
Based on that, something along the liens of this should work:
posixDateTask : Task.Task x Time.Date.Date
posixDateTask =
let
timeZoneDate now =
let
dateTime =
Time.DateTime.fromTimeStamp now
timeZone =
TimeZones.canada_pacific ()
zonedDateTime =
Time.ZonedDateTime.fromDateTime timeZone dateTime
in
Time.Date.date
(Time.ZonedDateTime.year zonedDateTime)
(Time.ZonedDateTime.month zonedDateTime)
(Time.ZonedDateTime.day zonedDateTime)
in
Time.now
|> Task.map timeZoneDate

map v.s. flatmap when promise is involved

I am studying
operator map v.s. flatmap
how to add promise into observable chain.
Then I constructed 4 different versions of var source as below.
version 1, 3 works as expected, while version 2, 4 fail oddly.
My code has also been added in => js bin
Could someone tell what is wrong with my code?
Thanks,
Xi
console.clear();
var p = new Promise((resolve, reject) => {
setTimeout( () => {
resolve('resolved!');
} , 1000);
});
var source = Rx.Observable.interval(200).take(3)
.flatMap(x => Rx.Observable.timer(500).map(() => x)) //version 1, works OK
// .flatMap(x => Rx.Observable.timer(500).map((x) => x)) // version 2, not OK, returns => 0, 0, 0
// .map(x => p.then( s => console.log(s))); // version 3, works OK
// .flatMap(x => p.then( s => console.log(s))); // version 4, not OK, error occurs
source.subscribe(x => console.log(x.toString()));
.flatMap(x => Rx.Observable.timer(500).map((x) => x))
returns "0", "0", "0" because timer emits 0 after 500 ms and map takes that value as input x and returns it with (x) => x. In the previous line, x was not redeclared in the map, so it came from flatMap.
.flatMap(x => p.then( s => console.log(s)));
gives an error because a promise emits the return value of the then callback. That's console.log(s) which being a statement evaluates to undefined. So the flatMap gives an Observable of undefined, undefined, undefined. When the first reaches the subscribe it tries to do undefined.toString and errors out.

Enum.reduce with Ecto.Multi

I want to reduce a map of the form
%{"overall" => "2016-08-27", "2" => "2016-06-27", "8" => "2015-08-27"} with these two functions
def set_agreed_date(id, agreed_date) do
Repo.get(Job, id)
|> change(agreed_date: agreed_date)
# |> Repo.update # removed per Dogbert's comment
end
def update(conn, %{"id" => job_id, "agreed_dates" => agreed_dates}, user) do
update = Enum.reduce(agreed_dates, Multi.new, fn({k, v}, acc) ->
{:ok, d} = Ecto.Date.cast v
case k do
"overall" ->
Multi.update(acc, "overall", set_agreed_date(job_id, d))
_ ->
Multi.update(acc, k, ShipToController.set_agreed_date(k, d))
end
end)
case Repo.transaction(update) do
{:ok, ?? not sure what I will get here ??} -> ...
but I am getting
[error] #PID<0.972.0> running Api.Endpoint terminated
Server: localhost:4000 (http)
Request: PUT /api/abc/14
** (exit) an exception was raised:
** (FunctionClauseError) no function clause matching in Ecto.Multi.add_operation/3
(ecto) lib/ecto/multi.ex:331: Ecto.Multi.add_operation(%Ecto.Multi{names: #MapSet<[]>, operations: []}, "overall", {:changeset, #Ecto.Changeset<action: :update, changes: %{agreed_date: #Ecto.Date<2016-08-27>}, errors: [], data: #Api.Job<>, valid?: true>, []})
(stdlib) lists.erl:1263: :lists.foldl/3
(api) web/controllers/controller.ex:71: Api.Controller.update/3
(api) web/controllers/controller.ex:1: Api.Controller.action/2
(api) web/controllers/controller.ex:1: Api.Controller.phoenix_controller_pipeline/2
(api) lib/api/endpoint.ex:1: Api.Endpoint.instrument/4
(api) lib/phoenix/router.ex:261: Api.Router.dispatch/2
(api) web/router.ex:1: Api.Router.do_call/2
(api) lib/api/endpoint.ex:1: Api.Endpoint.phoenix_pipeline/1
(api) lib/plug/debugger.ex:93: Api.Endpoint."call (overridable 3)"/2
(api) lib/api/endpoint.ex:1: Api.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

RxJs - parse file, group lines by topics, but I miss the end

I am trying RxJS.
My use case is to parse a log file and group lines by topic ( i.e.: the beginning of the group is the filename and then after that I have some lines with user, date/time and so on)
I can analyse the lines using regExp. I can determine the beginning of the group.
I use ".scan" to group the lines together, when I've the beginning of new group of line, I create an observer on the lines I've accumulated ... fine.
The issue is the end of the file. I've started a new group, I am accumulating lines but I can not trigger the last sequence as I do not have the information that the end. I would have expect to have the information in the complete (but not)
Here is an example using number. Begin of group can multi of 3 or 5. (remark: I work in typescript)
import * as Rx from "rx";
let r = Rx.Observable
.range(0, 8)
.scan( function(acc: number[], value: number): number[]{
if (( value % 3 === 0) || ( value % 5 === 0)) {
acc.push(value);
let info = acc.join(".");
Rx.Observable
.fromArray(acc)
.subscribe( (value) => {
console.log(info, "=>", value);
});
acc = [];
} else {
acc.push(value);
}
return acc;
}, [])
.subscribe( function (x) {
// console.log(x);
});
This emit:
0 => 0
1.2.3 => 1
1.2.3 => 2
1.2.3 => 3
4.5 => 4
4.5 => 5
6 => 6
I am looking how to emit
0 => 0
1.2.3 => 1
1.2.3 => 2
1.2.3 => 3
4.5 => 4
4.5 => 5
6 => 6
7.8 => 7 last items are missing as I do not know how to detect end
7.8 => 8
Can you help me, grouping items?
Any good idea, even not using scan, is welcome.
Thank in advance
You can use the materialize operator. See the documentation here and the marbles here, and an example of use from SO.
In your case, I would try something like (untested but hopefully you can complete it yourself, note that I don't know a thing about typescript so there might be some syntax errors):
import * as Rx from "rx";
let r = Rx.Observable
.range(0, 8)
.materialize()
.scan( function(acc: number[], materializedNumber: Rx.Notification<number>): number[]{
let rangeValue: number = materializedNumber.value;
if (( rangeValue % 3 === 0) || ( rangeValue % 5 === 0)) {
acc.push(rangeValue);
generateNewObserverOnGroupOf(acc);
acc = [];
} else if ( materializedNumber.kind === "C") {
generateNewObserverOnGroupOf(acc);
acc = [];
} else {
acc.push(rangeValue);
}
return acc;
}, [])
// .dematerialize()
.subscribe( function (x) {
// console.log(x);
});
function generateNewObserverOnGroupOf(acc: number[]) {
let info = acc.join(".");
Rx.Observable
.fromArray(acc)
.subscribe( (value) => {
console.log(info, "=>", value);
});
The idea is that the materialize and dematerialize works with notifications, which encodes whether the message being passed by the stream is one of next, error, completed kinds (respectively 'N', 'E', 'C' values for the kind property). If you have a next notification, then the value passed is in the value field of the notification object. Note that you need to dematerialize to return to the normal behaviour of the stream so it can complete and free resources when finished.

Resources