Thursday, September 24, 2015

Regular expressions: Rust vs F# vs Scala

Let's implement the following task: read first 10M lines from a text file of the following format:

then find all lines containing Microsoft namespace in them, and format the type names the usual way, like "Microsoft.Win32.IAssemblyEnum".

First, F#:

Now Rust:

After several launches the file was cached by the OS and both implementations became non IO-bound. F# one took 29 seconds and 31MB of RAM at peak; Rust - 11 seconds and 18MB.

The Rust code is as twice as long as F# one, but it's handling all possible errors explicitly - no surprises at runtime at all. The F# code may throw some exceptions (who knows what kind of them? Nobody). It's possible to wrap all calls to .NET framework with `Choice.attempt (fun _ -> ...)`, then define custom Error types for regex related code, for IO one and a top-level one, and the code'd be even longer then Rust's, hard to read and it would still give no guarantee that we catch all possible exceptions.

Update 4 Jan 2016: Scala added:

Ok, it turns out that regex performance may depend on whether it's case sensitive or not. What's worse, I tested F# with case insensitive pattern, but Rust - for case sensitive. Anyway, as I've upgraded my machine recently (i5-750 => i7-4790K), I've rerun F# and Rust versions in both the regex modes and added Scala to the mix. First, case sensitive mode:
  • F# (F# 4.0, .NET 4.6.1) - 4.8 secs
  • Scala (2.11.7, Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65) - 3.5 secs
  • Rust (1.7.0-nightly (bfb4212ee 2016-01-01) - 5.9 secs
Now, case insensitive:
  • F# (F# 4.0, .NET 4.6.1) - 15.5 secs
  • Scala (2.11.7, Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65) - 3.2 secs
  • Rust (1.7.0-nightly (bfb4212ee 2016-01-01) - 6.1 secs

Although case sensitive patterns performs roughly the same on all the platforms, it's quite surprising that Rust is not the winner.

Scala is faster in case insensitive mode (?), Rust is slightly slower and now the question: what's wrong with .NET implementation?.. It performs more than 3 times slower that case sensitive and the others.

Update 4 Jan 2016: D added.


  • regex - 10.6 s (DMD), 7.8 s (LDC)
  • ctRegex! - 6.9 s (DMD), 6.6 s (LDC)

Update 6 Jan 2016: Elixir added:

It takes 56 seconds to finish.

Update 6 Jan 2016: Haskell added:


I takes 20 seconds.

Update 7 Jan 2016: Nemerle added:


It takes 3.8 seconds (case sensitive) and 7.1 seconds (case insensitive).

Update 8 Jan 2016: Nemerle PEG added:


It takes 4.1 seconds.

All results so far:

  Case sensitive Case insensitive
F# 4,80 15,50
Scala 3,50 3,20
Rust 5,90 6,10
DMD 6,90  
LDC 6,60  
Elixir 56,00  
Hakell 20,00  
Nemerle 3,80 7,10
Nemerle PEG 4,10 4,20



Saturday, July 18, 2015

Elixir: first look

I don't have a clear impression about Elixir language yet. I don't like it has Ruby like syntax, but do like it has pipe operator and macros. So, Fibonacci:

It executes in about 13 seconds which is on pair (even faster for unknown reason) with Erlang, no surprises here.

  • D (GDC) - 0.990
  • C# - 1.26
  • D (DMD) - 1.3
  • C++ - 1.33
  • F# - 1.38
  • Nemerle - 1.45
  • Rust - 1.66
  • Go - 2.38
  • Haskell - 2.8
  • Clojure - 9
  • Elixir - 13
  • Erlang - 17
  • Ruby - 60
  • Python - 120

Monday, June 22, 2015

SHA1 compile time checked literals: F# vs Nemerle vs D

I've always been interested in metaprogramming. Sooner or later, I'm starting to feel constrained within a language without it. F# is a really nice language, but I'm afraid I'd have got bored with it if it'd not have Type Providers, for example. Why metaprogramming is so important? Because it allows changing a language without cracking the compiler. It allows making things which seemed to be impossible to implement.

I'm dealing with cryptography hashes a lot at work, nothing rocket since, just MD5, SHA-1 and so on. And I write tons of tests where such hashes are used in form of string literals, like this:


The problem with this code is that the compiler cannot guarantee that the hex string in the last line represents a valid SHA-1. If it does not, the test will fail at runtime for a reason it's not intended to.

OK, now we can formulate our task: provide a language construct to enforce a string literal being a valid SHA-1 hexadecimal, at compile time. We will explore how much work it's required to implement such a simple feature in F#, Nemerle and D. It's also interesting how well the development workflow is for each of this languages - IDE integration, error reporting and testing cycle.

F#


Using Type Providers is the only way to check (at compile time) that a string is a valid hex one and that it's length is exactly 40 characters (SHA-1 is a 20-bytes hash). Actually, I've written this type provider before. The interesting part looks like this:


It includes caching, and `HexParser` module is not shown, but those details are not important here. It's simple and it generates Value property which directly returns byte array, created in compile-time.

Error reporting:






Nemerle

Nemerle has full fledged macros, which strictly more powerful than F#'s Type Providers. Let's see if they allow solving the task in an elegant way:


Error reporting:



D


The code does not use any unusual stuff and does not manipulate AST. Just plane D code. Very elegant. Note that the template is defined in the same file as its usage. Contrast this with F# and Nemerle where you have to place your Type Provider / macros into a dedicated assembly.

Error reporting:


The error is located in the template itself, not at the instantiation point though.

Performance

I added 1000 usages of the TP, macro and template and measured compilation time.

  • F# - 5 seconds
  • Nemerle - 2 seconds
  • D - the compiler crashes with "Error: out of memory" after 1 minute work.

Saturday, June 20, 2015

Fib: C++, C# and GDC

As a reference implementation, I added C++ one:


It's execution time is 1.33 seconds, which surprisingly is not the best result so far.
A C# version:

Also, I compiled this D code with GDC compiler and it executed in 990 ms, which is the best result:

  • D (GDC) - 0.990
  • C# - 1.26
  • D (DMD) - 1.3
  • C++ - 1.33
  • F# - 1.38
  • Nemerle - 1.45
  • Rust - 1.66
  • Go - 2.38
  • Haskell - 2.8
  • Clojure - 9
  • Erlang - 17
  • Ruby - 60
  • Python - 120

Unfortunately, I have not managed to compile the D code with LDC compiler, it returns the following error:

Building: DFib (Release)
Performing main compilation...
Current dictionary: d:\git\DFib\DFib
D:\ldc2-0.15.2-beta1-win64-msvc\bin\ldc2.exe -O3 -release "main.d"   "-od=obj\Release" "-of=d:\git\DFib\DFib\bin\Release\DFib.exe"
LINK : fatal error LNK1181: cannot open input file 'kernel32.lib'
Error: C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\link.exe failed with status: 1181
Exit code 1181

Saturday, May 16, 2015

Composing custom error types in F#

I strongly believe that we should keep code as referential transparent as possible. Unfortunately, F# language does not encourage programmers to use Either monad to deal with errors. The common practice in the community is using common in the rest .NET (imperative) world exception based approach. From my experienced, almost all bugs found in production are caused by unhandled exceptions. 

The problem


In our project we've used the Either monad for error handling with great success for about two years. ExtCore is a great library making dealing with Either, Reader, State and other monads and their combinations really easy. Consider a typical error handling code, which make use Choice computation expression from ExtCore:


The code is a bit hairy because of explicit error mapping. We could introduce an operator as a synonym for Choice.mapError, like <!>, after which the code could become a bit cleaner:


(actually it's the approach we use at in our team).


Rust composable errors


I was completely happy until today, when I read Error Handling in Rust article and found out how elegantly errors are composed using From trait. By implementing it for an error type, you enable auto converting lower level errors to be convertable to it by try! macro, which eliminates error mapping completely. I encourage the reader to read that article because it explains good error handling in general, it's totally applicable to F#.

Porting to F#


Unfortunately, there's no static interface implementation neither in F# nor in .NET, so we cannot just introduce IError with a static member From: 'a -> 'this, like we can in Rust. But in F# we can use statically resolved type parameters to get the result we need. The idea is that each "higher level" error type defines a bunch of static methods, each of which converts some lower level error type to one of the error type cases: 

Now we can write a generic function which can create any higher level error type, which defines From methods:

Now we can rewrite our processFile function without explicit mapping to concrete error cases:

Great. But it's still not as clean. The remaining bit is to modify Choice computation expression builder so that it can do the same implicit conversion in its Bind method (its ChoiceBuilder from ExtCore as is, but without For and While methods):

The CE now requires all errors to be convertable to its main error type, including the error type itself, so we have to add one more From static method to Error type, and we finally can remove any noise from our processFile function:

Monday, May 4, 2015

Go: fib

Go code is relatively low-level since it does not have "foreach over range" syntax construct:

Results are not as impressive for a systems language: 2.38 seconds. And it lays below Rust but under Haskell:
  • C# - 1.26
  • D (DMD) - 1.3
  • F# - 1.38
  • Nemerle - 1.45
  • Rust - 1.66
  • Go - 2.38
  • Haskell - 2.8
  • Clojure - 9
  • Erlang - 17
  • Ruby - 60
  • Python - 120

Saturday, April 11, 2015

Computing cryptography hashes: Rust, F#, D and Scala

Let's compare how fast Rust, D and F# (.NET actually) at computing cryptography hashes, namely MD5, SHA1, SHA256 and SHA512. We're going to use rust-crypto cargo:

Results:
  • MD5 - 3.39s 
  • SHA1 - 2.89s 
  • SHA256 - 6.97s
  • SHA512 - 4.47s

Now the F# code:


Results (.NET 4.5, VS 2013, F# 3.1):
  • MD5CryptoServiceProvider - 2.32s (32% faster)
  • SHA1CryptoServiceProvider - 2.92s (1% slower)
  • SHA256Managed - 16.50s (236% slower)
  • SHA256CryptoServiceProvider - 11.50s (164% slower)
  • SHA256Cng - 11.71s (168% slower)
  • SHA512Managed - 61.04s (1365% slower)
  • SHA512CryptoServiceProvider - 21.88s (489% slower)
  • SHA512Cng - 22.19s (496% slower)
(.NET 4.6, VS 2015, F# 4.0):

  • MD5CryptoServiceProvider elapled 2.55
  • SHA1CryptoServiceProvider elapled 2.89
  • SHA256Managed elapled 17.01
  • SHA256CryptoServiceProvider elapled 8.74
  • SHA256Cng elapled 8.75
  • SHA512Managed elapled 23.42
  • SHA512CryptoServiceProvider 5.81
  • SHA512Cng elapled 5.79



D:



DMD
  • MD5 - 16.05s (470% slower)
  • SHA1 - 2.35s (19% faster)
  • SHA256 - 47.96s (690% slower (!))
  • SHA512 - 61.47s (1375% slower (!))
LDC2
  • MD5 - 2,18s (55% faster)
  • SHA1 - 2.88s (same)
  • SHA256 - 6,79s (3% faster)
  • SHA512 - 4,6s (3% slower)
GDC
  • MD5 - 2,43 (29% faster)
  • SHA1 - 2,84 (2% faster)
  • SHA256 - 12,62 (45% slower)
  • SHA512 - 8,56 (48% slower)


Scala:



  • MD5 - 4.2s (23% slower)
  • SHA1 - 6.09s (110% slower)
  • SHA256 - 9.96s (42% slower)
  • SHA512 - 7.32s (63% slower)
Interesting things:

  • Rust and D (LDC2) show very close results. D is significantly faster on MD5, so it's the winner!
  • D (DMD) has very bad performance on all algorithms, except SHA1, where it's won.
  • SHA512Managed .NET class is very slow. Do not use it.



Sunday, March 29, 2015

Rust: fib

Rust is an interesting language. It is not a primitive one, like Go where we don't have ADTs, pattern matching and generics (but we do have Nils). And it's advertising as a safe and performant system language. Today is the very first day I'm looking at it. Let's "smoke" test it with Fibonacci :)


Debug: 3.44 seconds, release: 1.66 seconds. This is not very impressive, but pretty fast indeed.
  • C# - 1.26
  • D (DMD) - 1.3
  • F# - 1.38
  • Nemerle - 1.45
  • Rust - 1.66
  • Haskell - 2.8
  • Clojure - 9
  • Erlang - 17
  • Ruby - 60
  • Python - 120
It's very interesting how it'll behave in concurrent Fibonacci test.

The compiler is quite slow: it takes 2-3 seconds to build this tiny program.

Saturday, January 10, 2015

Parallel reduce: Hopac, Asyncs, Tasks and Scala's Futures

Tuomas Hietanen posted a parallel reduce function that uses TPL Tasks. I found it interesting to compare performance of this function with analogues implemented using F# Asyncs, Hopac Jobs and Scala Futures.
The author uses noop long-running reduce function to show that it's really run in parallel. In this blog post we are benchmarking another aspect of the implementations: how much extra cost is introduced by a particular parallezation mechanism (library) itself.

We translate the original code almost as is to Tasks and Hopac:


And Scala's Futures:


The results (Core i5, 4 cores):

  • Sequential List.reduce: Real: 00:00:00.014, CPU: 00:00:00.015, GC gen0: 0, gen1: 0, gen2: 0 
  • Tasks: Real: 00:00:01.790, CPU: 00:00:05.678, GC gen0: 36, gen1: 10, gen2: 1 
  • Hopac: Real: 00:00:00.514, CPU: 00:00:01.482, GC gen0: 27, gen1: 2, gen2: 1 
  • Asyncs: Real: 00:00:37.872, CPU: 00:01:48.405, GC gen0: 90, gen1: 29, gen2: 4
  • Scala Futures: 4.8 seconds

(Hopac - 3.4 times faster, Asyncs - 21.1 times slower, Scala - 1.8 times slower)

Hopac is ~3.5 times faster than TPL. What's wrong with Asyncs? I don't know. Maybe they are not intended for highly concurrent scenarios. Or my code may not be the most efficient. Any ideas, guys?

Let's test the leaders on larger arrays:


(Hopac is 3.37 times faster, Scala is 1.5 times slower)


(Hopac is 5.25 times faster, Scala is 1.05 times slower)

Wednesday, January 7, 2015

Fibonacci: Hopac vs Async vs TPL Tasks on .NET and Mono

Hopac claims that its Jobs are much more lightweight that F# Asyncs. There are many benchmarks on Hopac github repository, but I wanted to make a simple and straightforward benchmark and what could be simpler that parallel Fibonacci algorithm? :) (actually there's a more comprehensive  benchmark in the Hopac repository itself, see Fibonacci.fs)

Sequential Fibonacci function is usually defined as

So write a parallel version in Hopac where each step is performed in a Job and all these Jobs are (potentially) run in Parallel by Hopac's scheduler

An equivalent parallel algorithm written using F# Asyncs

...and using TPL Tasks

All three functions create *a lot* of parallel jobs/asyncs/tasks. For example, for calculating fib (34) they create ~14 million of jobs (this is why Fibonacci was chose for this test). To make them work efficiently we will use the sequential version of fib for small N, then switch to parallel version

Now we can run both of the function with different "level"s in order to find on which value the functions starts to perform good (x-axis: level, y-axis: time (ms),  blue line: the sequential function, orange line: hopac/async/tasks function):

Hopac
Async
Tasks


Hopac reaches performance equivalent to the sequential implementation at level = 9, Async - at level = 17 and Tasks at level = 11.

If we modify the code so we can count how many jobs/asyncs are created during the calculation


We get the following results (n = 42): 

* Sequential, Real: 00:00:01.849, CPU: 00:00:01.840, GC gen0: 0, gen1: 0, gen2: 0
* Hopac (level = 9) jobs: 28761996, Real: 00:00:01.700, CPU: 00:00:05.600, GC gen0: 89, gen1: 1, gen2: 0
* Async (level = 17) asyncs: 605898, Real: 00:00:01.515, CPU: 00:00:04.804, GC gen0: 4, gen1: 2, gen2: 0
* Tasks (level = 11) tasks: 5675789, Real: 00:00:01.813, CPU: 00:00:06.302, GC gen0: 18, gen1: 0, gen2: 0

So, Hopac was able to create and processed ~47x more jobs than Async and ~5x more jobs than Tasks. Hopac is impressive and F# Asyncs are frustrating.  

PS: Rewriting the async version without async computation explicit expression, like this

does not improve performance at all. 

Running on Mono (Ubuntu 14.10 x64, mono 3.10)

* Sequential, Real: 00:00:02.637, CPU: 00:00:02.636, GC gen0: 0, gen1: 0
* Hopac (level = 17) jobs: 629133Real: 00:00:02.447, CPU: 00:00:06.106, GC gen0: 26, gen1: 1
* Async (level = 21) asyncs: 92375Real: 00:00:02.845, CPU: 00:00:05.590, GC gen0: 86, gen1: 3
* Tasks (level = 33) tasks: 143Real: 00:00:14.111, CPU: 00:00:03.782, GC gen0: 0, gen1: 0

Hopac can handle ~6.8x more jobs than F# Async. I'm not sure if F# asyncs performs very well on Mono or it's because everything works extremely slowly there. What about TPL, it's obviously broken on Mono (official Hopac Fibonacci benchmark does not even run TPL version on mono: Fibonacci.fs#L233).