I'm running a complex computation (a Markov chain model)
let memoize f =
let cache = new ConcurrentDictionary<'key,'value>()
cache, fun x -> cache.GetOrAdd(x, Func<'key, 'value>f)
to cache intermediate results of multiple functions. The overall structure is something like this
module Foo =
[...]
let _, foo' = memoize foo
module Bar =
[...]
let _, bar' = memoize bar
module Main =
open Foo
open Bar
[...]
let result =
foobar (foo' a) (bar' b)
Typically I run this once and then the program terminates, but it's obviously not nice to not clean up those cache dictionaries. Also, I sometimes need to call the model for many different inputs and then I quickly run into memory issues. What's the best way to clean up multiple caches at once?
Edit
A mentioned in the comments, it would of course be possible to collect all caches into a list. But I'd have to box the dictionaries and it doesn't seem nice to me. Is there a better (overall) strategy?
I would suggest using a more robust caching structure than ConcurrentDictionary so that you can specify an expiration policy. Here's one on FSSnip that wraps ConcurrentDictionary and allow for time-based expiration, but you could add expiration based on other criteria. This would allow you to just use memoizeWithExpiration without having to worry about clean-up on the calling side.
This would be my suggestion, simple and effective:
module Foo =
[...]
let fcache, foo' = memoize foo
module Bar =
[...]
let bcache, bar' = memoize bar
module Main =
open Foo
open Bar
let clearCaches = [
fcache.Clear
bcache.Clear
]
[...]
let result =
foobar (foo' a) (bar' b)
let clearAll() =
clearCaches |> Seq.iter (fun clear -> clear())
Update
If you wanted to collect the clear functions automatically the memoize function could do it, like this:
let clearCaches = Dictionary<_,_>()
let memoize (name:string) f =
let cache = new ConcurrentDictionary<'key,'value>()
clearCaches.Add(name, cache.Clear)
fun x -> cache.GetOrAdd(x, Func<'key, 'value>f)
module Foo =
[...]
let foo' = memoize "Foo.foo" foo
module Bar =
[...]
let bar' = memoize "Bar.bar" bar
module Main =
open Foo
open Bar
[...]
let result =
foobar (foo' a) (bar' b)
let clearAll() =
clearCaches |> Seq.iter (fun kvp -> kvp.Value())
Which would also allow you to clear them individually or using certain conditions, like by module, etc.
Related
I have a combo ReScript and TypeScript project. I want to use gentype to expose TypeScript friendly objects. Gentype works for string, bool, records, and other basic objects without any hassles. How can I work with list and char and other ReScript-specific types from TypeScript?
There is a shims feature but I don't know how to do that. I would assume there are built-in shims for the standard Belt library but can't find them.
I plan to do most of the calculations and heavy-lifting inside Rescript. So when the info is sent to TypeScript, I'll probably just be consuming the data, not modifying it. This might be iterating through a list to display the contents in a non-Rescript React project. I don't plan to manipulate the data.
One option is to convert lists to arrays inside ReScript before exporting the results. So any time I want to consume information from TypeScript, I'll create a function and/or type in ReScript that only has bool, number, string, array, and object. I could create an "interop" module with all the stuff I want to consume or use from TypeScript. This provides a clean separation between the ReScript and TypeScript world and is easy to get my head around but seems a little inefficient (like converting all lists to arrays) and extra work.
On the other hand, if using the Belt objects like list and map is cumbersome from TypeScript, even with the shims, then I'm probably better off creating my "interop" module anyway.
What is the recommended/simplest/best way to use list, map, char, and other ReScript specific objects from within TypeScript? Are there friendly Typescript definitions for the Belt standard library that I could use even if I wasn't using ReScript?
===
One additional note. I have experience trying to use F# (functional) from C#. It was painful. The best solution for me in that environment was to create an interface on the F# side that was easily consumable on the C# side that did not use the F# native objects.
As rescript compiles to JavaScript, and the output is very readable and (usually) straight-forward, you could just emulate what it generates.
For example, this rescript code:
let xs = list{1, 2, 3}
let _ = List.map(x => x + 1, xs)
is compiled into this (slightly simplified) JavaScript:
var List = require("./stdlib/list.js");
var xs = {
hd: 1,
tl: {
hd: 2,
tl: {
hd: 3,
tl: /* [] */0
}
}
};
List.map((x) => x + 1, xs);
There is a slight problem with the literal syntax of lists specifically, but that could be simplified a bit by using List.cons instead:
let xs = List.cons(1, List.cons(2, List.cons(3, list{})))
which becomes:
var xs = List.cons(1, List.cons(2, List.cons(3, /* [] */0)));
Pattern matching also isn't as convenient, obviously, but still pretty straight-forward for the simple things at least. For example:
let rec sum = xs => switch xs {
| list{} => 0
| list{x, ...rest} => x + sum(rest)
}
becomes:
function sum(xs) {
if (xs) {
return xs.hd + sum(xs.tl);
} else {
return 0;
}
}
Most other types don't have any special compiler support, and so becomes just plain function calls. Using Belt.Map for example, despite using some advanced language features, compiles to very straight-forward JavaScript:
module IntCmp = Belt.Id.MakeComparable({
type t = int
let cmp = (a, b) => a - b
})
let m = Belt.Map.make(~id=module(IntCmp))
let _ = Belt.Map.set(m, 0, "a")
becomes (more or less):
var Belt_Id = require("./stdlib/belt_Id.js");
var Belt_Map = require("./stdlib/belt_Map.js");
var IntCmp = Belt_Id.MakeComparable({
cmp: (a, b) => a - b
});
var m = Belt_Map.make(IntCmp);
Belt_Map.set(m, 0, "a");
I'm trying to memoize a member function of a class, but every time the member is called (by another member) it makes a whole new cache and 'memoized' function.
member x.internal_dec_rates =
let cache = new Dictionary< Basis*(DateTime option), float*float>()
fun (basis:Basis) (tl:DateTime option) ->
match cache.TryGetValue((basis,tl)) with
| true, (sgl_mux, sgl_lps) -> (sgl_mux, sgl_lps)
| _ ->
let (sgl_mux, sgl_lps) =
(* Bunch of stuff *)
cache.Add((basis,tl),(sgl_mux,sgl_lps))
sgl_mux,sgl_lps
I'm using Listing 10.5 in "Real World Functional Programming" as a model. I've tried using a memoization higher-order function and that doesn't help. The above listing has the memoization built in directly.
The problem is, when I call it e.g.
member x.px (basis:Basis) (tl: DateTime option) =
let (q,l) = (x.internal_dec_rates basis tl)
let (q2,l2) = (x.internal_dec_rates basis tl)
(exp -q)*(1.-l)
execution goes to the 'let cache=...' line, defeating the whole point. I put in the (q2,l2) line in order to make sure it wasn't a scope problem, but it doesn't seem to be.
In fact I did a test using Petricek's code as a member function and that seems to have the same issue:
// Not a member function
let memo1 f =
let cache = new Dictionary<_,_>()
(fun x ->
match cache.TryGetValue(x) with
| true, v -> v
| _ -> let v = f x
cache.Add(x,v)
v
)
member x.factorial = memo1(fun y->
if (y<=0) then 1 else y*x.factorial(y-1))
Even the internal recursion of x.factorial seems to set up a new 'cache' for each level.
What am I doing wrong, and how can I make this work?
In response to your comment on Jack's answer, this doesn't have to become tedious. Given a memoize function:
let memoize f =
let cache = Dictionary()
fun x ->
match cache.TryGetValue(x) with
| true, v -> v
| _ ->
let v = f x
cache.Add(x, v)
v
Define each of your functions as let-bound values and return them from your methods:
type T() as x =
let internalDecRates = memoize <| fun (basis: Basis, tl: DateTime option) ->
(* compute result *)
Unchecked.defaultof<float * float>
let px = memoize <| fun (basis, tl) ->
let (q,l) = x.InternalDecRates(basis, tl)
let (q2,l2) = x.InternalDecRates(basis, tl)
(exp -q)*(1.-l)
member x.InternalDecRates = internalDecRates
member x.Px = px
The only "boilerplate" is the let binding and call to memoize.
EDIT: As kvb noted, in F# 3.0 auto-properties allow a more concise solution:
type T() as x =
member val InternalDecRates = memoize <| fun (basis: Basis, tl: DateTime option) ->
(* compute result *)
Unchecked.defaultof<float * float>
member val Px = memoize <| fun (basis, tl) ->
let (q,l) = x.InternalDecRates(basis, tl)
let (q2,l2) = x.InternalDecRates(basis, tl)
(exp -q)*(1.-l)
I see a lot of long answers here; the short answer is that
member x.P = code()
defines a property P which has a getter that runs code() every time P is accessed. You need to move the cache creation into the class's constructor, so that it will only run once.
As others already said, this cannot be done just by defining a single member in F# 2.0. You either need a separate field (let bound value) for a cache or for a local function that is memoized.
As mentioned by kvb, in F# 3.0, you can do this using member val which is a property that is initialized when the object is created (and has an automatically generated backing field where the result is stored). Here is a complete sample that demonstrates this (it will work in Visual Studio 2012):
open System.Collections.Generic
type Test() =
/// Property that is initialized when the object is created
/// and stores a function value 'int -> int'
member val Foo =
// Initialize cache and return a function value
let cache = Dictionary<int, int>()
fun arg ->
match cache.TryGetValue(arg) with
| true, res -> res
| false, _ ->
let res = arg * arg
printfn "calculating %d" arg
cache.Add(arg, res)
res
// Part of the property declaration that instructs
// the compiler to generate getter for the property
with get
The with get part of the declaration can be omitted, but I include it here to make the sample clearer (you can also use with get, set to get a mutable property). Now you can call test.Foo as a function and it caches the value as required
let t = Test()
t.Foo(10)
t.Foo(10)
The only problem with this approach is that t.Foo is actually compiled as a property that returns a function (instead of being compiled as a method). This is not a big problem when you use the class from F#, but it would be a problem if you were calling it from C# (because C# would see the member as a property of type FSharpFunc<int, int>, which is hard to use).
John is correct -- you need to move the cache dictionary into a private, let-bound member of the type.
Type members are compiled a bit differently than let-bound values in modules, which is the reason for the difference in behavior. If you copy/paste the body of your x.internal_dec_rates method and assign it to a let-bound value in a module, it should work correctly then, because the F# compiler will compile it as a closure which gets created once and then assigned to a static readonly field of the module.
A couple of other tips, for good measure:
Type member methods can use optional parameters -- so you can slightly simplify the method signature if you like.
You can create the cache key just once and reuse it (this also helps avoid mistakes).
You can simplify the (sgl_mux, sgl_lps) pattern-matching code by just assigning the tuple a name (e.g., value), since you're just returning the whole tuple anyway.
Here's my take on your code:
type FooBar () =
let cache = new Dictionary< Basis*(DateTime option), float*float>()
member x.internal_dec_rates (basis : Basis, ?tl : DateTime) =
let key = basis, tl
match cache.TryGetValue key with
| true, value -> value
| _ ->
// sgl_mux, sgl_lps
let value =
(* Bunch of stuff *)
cache.Add (key, value)
value
You need to move the dictionary outside the function call - like
let cache = new Dictionary< Basis*(DateTime option), float*float>()
member x.internal_dec_rates =
fun (basis:Basis) (tl:DateTime option) ->
match cache.TryGetValue((basis,tl)) with
| true, (sgl_mux, sgl_lps) -> (sgl_mux, sgl_lps)
| _ ->
let (sgl_mux, sgl_lps) =
(* Bunch of stuff *)
cache.Add((basis,tl),(sgl_mux,sgl_lps))
sgl_mux,sgl_lps
This way the cache persists across the function calls. Your memo1 has the same problem. In the original version, you create a new cache every time you call the function, this way we just have a single cache, which persists across function calls.
In addition to the other answers, note that in F# 3.0 you can use automatically implemented properties, which will behave as you want:
member val internal_dec_rates = ...
Here, the right hand side is evaluated only once, but everything is self-contained.
When trying to print
pop
I get all this weird looking formatting in F# interactive, which basically turns the printing useless. Is there someway other to correctly format this?
The code is the following:
#light
open System
let rng = new Random()
type Individual = { x:int; y:int }
type ScoredIndividual = { individual:Individual; score:int }
let genGene() = rng.Next(-10, 10)
let genRandInd() = { x=genGene(); y=genGene() }
let genRandPop popSize = [ for _ in 1 .. popSize -> genRandInd() ]
let getScoredPop f pop = List.map (fun i -> { individual=i; score=(f i)}) pop
let fitnessFun ind = ind.x * ind.x + ind.y * ind.y
let pop = 30 |> genRandPop |> getScoredPop fitnessFun
You can override ToString or use StructuredFormatDisplayAttribute to customize the string representation. This article contains some useful information about customizing output in fsi.
you might want to do fsi.AddPrinter for your ScoredIndividual type to control what's written to the console
That's pretty rough, and I couldn't find any "easy" way to fix it. However, FsEye can make it nicer (while it does delete the newlines, those spaces are in there good):
I am new to Scala and am trying to get a list of random double values:
The thing is, when I try to run this, it takes way too long compared to its Java counterpart. Any ideas on why this is or a suggestion on a more efficient approach?
def random: Double = java.lang.Math.random()
var f = List(0.0)
for (i <- 1 to 200000)
( f = f ::: List(random*100))
f = f.tail
You can also achieve it like this:
List.fill(200000)(math.random)
the same goes for e.g. Array ...
Array.fill(200000)(math.random)
etc ...
You could construct an infinite stream of random doubles:
def randomList(): Stream[Double] = Stream.cons(math.random, randomList)
val f = randomList().take(200000)
This will leverage lazy evaluation so you won't calculate a value until you actually need it. Even evaluating all 200,000 will be fast though. As an added bonus, f no longer needs to be a var.
Another possibility is:
val it = Iterator.continually(math.random)
it.take(200000).toList
Stream also has a continually method if you prefer.
First of all, it is not taking longer than java because there is no java counterpart. Java does not have an immutable list. If it did, performance would be about the same.
Second, its taking a lot of time because appending lists have linear performance, so the whole thing has quadratic performance.
Instead of appending, prepend, which had constant performance.
if your using mutable state anyways you should use a mutable collection like buffer which you can add too with += (which then would be the real counterpart to java code).
but why dont u use list comprehension?
val f = for (_ <- 1 to 200000) yield (math.random * 100)
by the way: var f = List(0.0) ... f = f.tail can be replaced by var f: List[Double] = Nil in your example. (no more performance but more beauty ;)
Yet more options! Tail recursion:
def randlist(n: Int, part: List[Double] = Nil): List[Double] = {
if (n<=0) part
else randlist(n-1, 100*random :: part)
}
or mapped ranges:
(1 to 200000).map(_ => 100*random).toList
Looks like you want to use Vector instead of List. List has O(1) prepend, Vector has O(1) append. Since you are appending, but using concatenation, it'll be faster to use Vector:
def random: Double = java.lang.Math.random()
var f: Vector[Double] = Vector()
for (i <- 1 to 200000)
f = f :+ (random*100)
Got it?
Ok, so let's say I have a type defined like so:
type Foo =
| Bar of (SomeType * SomeType * SomeType * SomeType)
| ...(other defs)
so I have a Bar, that is basically a tuple of 4 SomeTypes. I want to access individual members of the tuple. I tried this:
let Bar (one, two, three, four) = someBar
But when I try to refer to one, or two later on in the function it says that "the value or constructor is not defined" So it is not treating the assignment as expected. What is the correct way to do this?
Also, if i try:
let one,two,three,four = someBar
It complains:
someBar was expected to have type 'a*'b*'c*'d but here has type Foo
thanks,
You just need to add another set of parentheses:
let (Bar(one,two,three,four)) = someBar
As Stephen points out, without the additional parens the compiler treats this line of code as the definition of a new function called Bar. He is also right that pattern matching would probably be more appropriate if there are other cases in the discriminated union.
Given
type Foo =
| Bar of (int * int * int * int)
| Bar2 of string
let x = Bar(1,2,3,4)
let Bar(y1,y2,y3,y4) = x
the last let binding is interpreted as a function, Bar : 'a * 'b * 'c * 'd -> Foo. The function name is throwing you off, since it is the same as your union case, but it's the same as if you had defined let some_func_takes_a_tuple_and_returns_x (y1,y2,y3,y4) = x.
I think you may have to be a little more verbose:
let y1,y2,y3,y4 =
match x with
| Bar(y1,y2,y3,y4) -> y1,y2,y3,y4
Which is fair enough, since unlike tuple decomposition let bindings, decomposing Bar here is dangerous because the match is incomplete (x could actually be some other Foo case, like Bar2).
Edit
#kvb knows the secret to making this work as you expect!