tweag / distributed-closure Goto Github PK

Serializable closures for distributed programming.

License: BSD 3-Clause "New" or "Revised" License

Haskell 96.67% Nix 3.33%

distributed-closure's Introduction

distributed-closure

Leverage the -XStaticPointers extension from GHC 7.10 onwards for distributed programming using lightweight serializable closures. This package implements a serializable closure abstraction on top of static pointers. Serializable closures can be shipped around a computer cluster. This is useful for implementing intuitive and modular distributed applications. See this blog post for a hands on introduction and this paper for the original motivation.

Example

In GHC 8 and above, remoting a computation on another node using this library goes along the lines of

data NodeId

-- Assume we're given a spaw primitive.
spawn :: NodeId -> Closure (IO ()) -> IO ()

-- A computation to remote on another node.
hello :: String -> IO ()
hello name = putStrLn ("Hello, " ++ name)

main = do
  name <- getLine
  spawn "someAddress" (static hello `cap` name)

An example of sending static pointers and closures through a communication channel is provided under examples/ClientServer.hs in the source repository.

distributed-closure does not implement sending/receiving/spawning closures - that's left for higher-level frameworks. Only closure creation, composition and (de)serialization.

distributed-closure's People

Contributors

Stargazers

Watchers

Forkers

qnikst iu-parfunc lucasdicioccio julianzhang brandon-leapyear mpickering facundominguez

distributed-closure's Issues

Build failure with GHC 8

> /tmp/stackage-build8$ stack unpack distributed-closure-0.3.0.0
Unpacked distributed-closure-0.3.0.0 to /tmp/stackage-build8/distributed-closure-0.3.0.0/
> /tmp/stackage-build8/distributed-closure-0.3.0.0$ runghc -clear-package-db -global-package-db -package-db=/home/stackage/work/builds/nightly/pkgdb Setup configure --package-db=clear --package-db=global --package-db=/home/stackage/work/builds/nightly/pkgdb --libdir=/home/stackage/work/builds/nightly/lib --bindir=/home/stackage/work/builds/nightly/bin --datadir=/home/stackage/work/builds/nightly/share --libexecdir=/home/stackage/work/builds/nightly/libexec --sysconfdir=/home/stackage/work/builds/nightly/etc --docdir=/home/stackage/work/builds/nightly/doc/distributed-closure-0.3.0.0 --htmldir=/home/stackage/work/builds/nightly/doc/distributed-closure-0.3.0.0 --haddockdir=/home/stackage/work/builds/nightly/doc/distributed-closure-0.3.0.0 --flags=
Configuring distributed-closure-0.3.0.0...
> /tmp/stackage-build8/distributed-closure-0.3.0.0$ runghc -clear-package-db -global-package-db -package-db=/home/stackage/work/builds/nightly/pkgdb Setup build
Building distributed-closure-0.3.0.0...
Preprocessing library distributed-closure-0.3.0.0...
[1 of 3] Compiling Control.Distributed.Closure.Internal ( src/Control/Distributed/Closure/Internal.hs, dist/build/Control/Distributed/Closure/Internal.o )

src/Control/Distributed/Closure/Internal.hs:32:1: warning: [-Wunused-imports]
    The import of ‘GHC.Fingerprint’ is redundant
      except perhaps to import instances from ‘GHC.Fingerprint’
    To import instances alone, use: import GHC.Fingerprint()

src/Control/Distributed/Closure/Internal.hs:61:1: warning: [-Wredundant-constraints]
    • Redundant constraint: Typeable a
    • In the type signature for:
           fromDynClosure :: Typeable a => DynClosure -> Closure a

src/Control/Distributed/Closure/Internal.hs:134:1: warning: [-Wredundant-constraints]
    • Redundant constraint: Typeable a
    • In the type signature for:
           cap :: Typeable a => Closure (a -> b) -> Closure a -> Closure b
[2 of 3] Compiling Control.Distributed.Closure ( src/Control/Distributed/Closure.hs, dist/build/Control/Distributed/Closure.o )

src/Control/Distributed/Closure.hs:48:1: error:
    • Potential superclass cycle for ‘Static’
        one of whose superclass constraints is headed by a type variable:
          ‘c’
      Use UndecidableSuperClasses to accept this
    • In the class declaration for ‘Static’

Derive static versions of `StandaloneDeriving` instances with `withStatic`

Instances declared with StandaloneDeriving are not recognized by withStatic, and they don't get their static version defined. The following program does not compile (missing instance for (Static (Show D))).

{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE StaticPointers #-}
{-# LANGUAGE StandaloneDeriving #-}
{-# Language TemplateHaskell #-}

import Control.Distributed.Closure
import Control.Distributed.Closure.TH
import Data.Typeable

data D = D
  deriving (Typeable)

withStatic [d|
  deriving instance Show D
  |]

showC :: forall a. (Typeable a, Static (Show a)) => Closure a -> String
showC a = unclosure $ static (\Dict -> show)
  `cap` (closureDict :: Closure (Dict (Show a)))
  `cap` a

main :: IO ()
main = do
  putStrLn $ showC $ static D

Neither does withStatic work with derived instances

withStatic [d|
  data D = D
    deriving (Typeable, Show)
  |]
-- Static (Show D) doesn't get defined

As a result, there is no way to auto-generate static versions of instances with default implementations.

Tests fail to compile on GHC 8.6

When compiling tests on GHC 8.6, we get below error:

tests/test.hs:31:1: error:
    • Illegal nested constraint ‘Static (Show (F a))’
      (Use UndecidableInstances to permit this)
    • In the instance declaration for
        ‘Static (Show (T1 a_a8ni b_a8nj))’
   |
31 | withStatic [d|
   | ^^^^^^^^^^^^^^...

tests/test.hs:31:1: error:
    • Illegal nested constraint ‘Show (F a)’
      (Use UndecidableInstances to permit this)
    • In the instance declaration for ‘Show (T1 a_a8ni b_a8nj)’
   |
31 | withStatic [d|
   | ^^^^^^^^^^^^^^...

The offending line is this one:

withStatic [d|
  ...
  instance Show (F a) => Show (T1 a b) where show = undefined
  ...
  |]

where F is a type family. The change mentioned here is probably the cause: https://ghc.haskell.org/trac/ghc/wiki/Migration/8.6#UndecidableInstancesispickier

Removing the offending instance or setting -XUndecidableInstances solves the issue.

Add examples of sending closures over the network

Could you kindly add a few examples of sending a closure or a plain function over the network and a process listening to that, accepting the function and then applying that function to some data? It would be really beneficial to have a few examples to show the entire flow of

 Node 1                                                         Node 2
----------                                                   -----------

function   --------------------------->
                    send                                   accept the function and 
                                                            apply it to a dataset

Provide `Static` instance for `Static`

I find that I need this in my code when I have closures that create closures:

instance Static c => Static (Static c) where
  closureDict = closureDict

How to handle types that require non-trivial serialization?

I am implementing distributed reference counting. These references require a non-trivial calculation to when serialized and de-serialized – a serialized reference still counts as reference, and thus the reference count must be increased and decreased respectively. Copying a serialized reference must also increase the reference count.

I wonder: How does this work with distributed-closure? It seems that Encoded values are copied...

Provide `Static` instances for longer tuples

There is a Static instance for 2-tuples:

instance (Static c1, Static c2, Typeable c1, Typeable c2, (c1, c2)) => Static (c1, c2) where
  closureDict = static pairDict `cap` closureDict `cap` closureDict

Please add respective instances for longer tuples as well.

Add `Closure` to the definition of `staticPure`

StaticApplicative typeclass is currently defined as:

class StaticApply f => StaticApplicative f where
  staticPure :: Typeable a => a -> f a

My suggestion is to change the signature to

class StaticApply f => StaticApplicative f where
  staticPure :: Typeable a => Closure a -> f a

My reasoning is that, the biggest point of Static* typeclasses are to represent composition of the values that can be distributed over the network. However, since that a can not be serialised, it constrains the use of StaticApplicative typeclass to only work on local values.

I can understand that probably this class is built for a specific purpose and that constraint wasn't necessary in that case; however I'd argue that having Closure there is less restrictive than the current state, since one can always unclosure it in the instance if a concrete a is required.

My objective is to implement StaticApplicative for the type defined here. dConstAggr should be the staticPure in case of my suggestion.

Security model

If I understand correctly, in a Haskell program, different modules can use static on their own top-level variables to get StaticPtr a where a is the type of the top-level variable. Then a StaticKey can be obtained from the StaticPtr. Later, with a StaticKey, a lookup can be performed using unsafeLookupStaticPtr and then deRefStaticPtr to get back a value of type a.

When I write a program using static pointers, e.g. to implement something like RPC, I would typically be getting the StaticKeys from some external source, e.g. disk or network, and then trying to get a value using unsafeLookupStaticPtr. The value I get from unsafeLookupStaticPtr could be unsafe for 2 reasons: 1. The type assumed is wrong. 2. The type assumed is correct but the value is some function from some module (e.g. from some hackage package) that I didn't even know 'exposed' a StaticPtr in the static table. If I inadvertently run that function, bad, unexpected, things may happen. I think the key problem is that any module can make its own static pointers and GHC does not mandate that only the main program is allowed to do so.

Some other libraries (e.g. Control.Distributed.Static) avoid the the second problem by requiring the explicit use of a table during the 'lookup' (RemoteTable). It seems that distributed-closure is using the Haskell static table directly. Is it susceptible to the second type of safety problem? If so, what can I do to restrict the functions that I can get from unclosure?

New hackage release

Please make a new Hackage release. There has been quite a bit of changes since the last release.

`withStatic` instance fails when type families appear in constraints

The following program

{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE StaticPointers #-}
{-# LANGUAGE UndecidableInstances #-}

import Data.Constraint
import Data.Typeable
import Control.Distributed.Closure
import GHC.StaticPtr

data T a = T
  deriving Show

type family F a where
  F Int = Int
  F a = a

class C a where
  m :: Maybe a

instance C Int where m = Just 0

instance C (F a) => C (T a) where
  m = Just T

instance (Typeable a, Static (C (F a))) => Static (C (T a)) where
  closureDict = closure (static g) `cap` closureDict
--  closureDict = closure (static g :: StaticPtr (Dict (C (F a)) -> Dict (C (T a)))) `cap` closureDict

g :: Dict (C (F a)) -> Dict (C (T a))
g Dict = Dict

main :: IO ()
main = print (m :: Maybe (T Int))

fails with

tf.hs:30:17: error:
    • Could not deduce (Typeable c0) arising from a use of ‘cap’
      from the context: (Typeable a1, Static (C (F a1)))
        bound by the instance declaration at tf.hs:29:10-59
      The type variable ‘c0’ is ambiguous
    • In the expression: closure (static g) `cap` closureDict
      In an equation for ‘closureDict’:
          closureDict = closure (static g) `cap` closureDict
      In the instance declaration for ‘Static (C (T a))’

tf.hs:30:33: error:
    • Couldn't match type ‘a0’ with ‘a’
      Expected type: Dict c0 -> Dict (C (T a))
        Actual type: Dict (C (F a0)) -> Dict (C (T a0))
    • In the body of a static form: g
      In the first argument of ‘closure’, namely ‘(static g)’
      In the first argument of ‘cap’, namely ‘closure (static g)’
    • Relevant bindings include
        closureDict :: Closure (Dict (C (T a))) (bound at tf.hs:30:3)

tf.hs:30:42: error:
    • Could not deduce (Static c0) arising from a use of ‘closureDict’
      from the context: (Typeable a1, Static (C (F a1)))
        bound by the instance declaration at tf.hs:29:10-59
      The type variable ‘c0’ is ambiguous
      These potential instance exist:
        instance (Typeable a, Static (C (F a))) => Static (C (T a))
          -- Defined at tf.hs:29:10
    • In the second argument of ‘cap’, namely ‘closureDict’
      In the expression: closure (static g) `cap` closureDict
      In an equation for ‘closureDict’:
          closureDict = closure (static g) `cap` closureDict

However it succeeds if the type signature of static g is added as in the commented line. Unfortunately withStatic produces the version with no type signature.

Both variants look correct to me. I've managed to reduce the example to this:

{-# LANGUAGE ConstraintKinds #-}
{-# LANGUAGE GADTs #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE StaticPointers #-}
{-# LANGUAGE UndecidableInstances #-}
{-# LANGUAGE UndecidableSuperClasses #-}

import Data.Constraint
import Data.Typeable
import GHC.StaticPtr

class c => Static c where
  closureDict :: Closure (Dict c)

data Closure a where
  CStaticPtr :: StaticPtr a -> Closure a
  CAp :: Closure (a -> b) -> Closure a -> Closure b

data T a = T
  deriving Show

type family F a where
  F Int = Int
  F a = a

class C a where
  m :: Maybe a

instance C Int where m = Just 0

instance C (F a) => C (T a) where
  m = Just T

instance (Typeable a, Static (C (F a))) => Static (C (T a)) where
  closureDict = CStaticPtr (static g) `CAp` closureDict

g :: Dict (C (F a)) -> Dict (C (T a))
g Dict = Dict

main :: IO ()
main = print (m :: Maybe (T Int))

If StaticPtr is removed from the CStaticPtr constructor and static g is replaced with g, then the program builds fine.

Client server example doesn't build with ghc-8.4.3

The build fails with the following error:

examples/ClientServer.hs:83:1: error:
    • Illegal instance for a type synonym
      A class instance must be for a class
    • In the instance declaration for ‘Serializable SerializableInt’
   |
83 | withStatic [d|
   | ^^^^^^^^^^^^^^...

In order to fix it I have to define some instances manually which seemed to work.

mpickering@79433e8