transient-haskell / transient Goto Github PK

• "Now this is another story: this time choose' is a Transient primitive. It stops the current thread and launches a new thread for each of the elements in the list. Since the main thread dies, the computation could finish before the launched threads complete. This is the reason for the name "transient": because it manages transient threads. This is why it is called keep. It prevents the program from exiting when the main thread dies. exit escapes the keep block."
  • Exit is just an identifier, right? So what actually escapes? And why can the computation only finish because the main thread dies?
• "In this applicative expression, both async operations run in parallel, within different threads. When the first finishes, it inspect the result of the other. If it has no result yet, the inspecting thread stores its result and dies. When the other thread finishes, it sees the result of the previous one. Then, it completes the applicative expression and prints the result. The computation brought to async runs in the IO monad:"
  • Too many words are used to identify threads here, including "other", "inspecting thread" and "previous one". This paragraph would benefit from a sequence diagram.
• "It produces the same result, but this time, the original thread is the one that runs the IO computation of the second term, instead of dying."
  • What is meant by "original"? Why not just say "main"? Also, why call it "second term"? It could be given a name and it would be easier to refer to.
• "Actually, a Cloud computation is a transient one, with a very thin layer that assures that the type system makes the program aware of the need to log and recover intermediate results, in order to re-create the closures at the remote node."
  • The last sentence is very complicated, and how can the type system tell any program anything? Does the Cloud cloud monad force logging? If so, this sentence could be made simpler.
• "Unlike Cloud Haskell and other distributed frameworks that use static closures, Transient distributed programs don't need to be identical in all nodes, neither do the nodes have to share the same architecture. It is enough to share the same snippet of code that is called."
  • What is meant by architecture? CPU architecture?

We're working on porting libp2p to haskell under (github.com/MatrixAI). But I'm very intrigued by your project here. While libp2p is just a sort of communication system (although a very sophisticated one at that), it lacks any higher level of composability. I'm wondering how compatible is transient with different underlying communication primitives? I haven't grokked transient yet, but libp2p sort of exposes bytes-streams that are the transparently mapped to other nodes in the network.

There are notions of addressing services that supply bytestreams or receive bytestreams in libp2p however.

See http://matrix.hackage.haskell.org/package/transient and I can't figure out which bounds need to be added to transient.cabal other than probably base >= 4.6.

Here's the compile error I'm seeing with GHC 7.6 through 7.10:

[1 of 1] Compiling Main             ( move.hs, dist/dist-sandbox-a1f04431/build/move/move-tmp/Main.o )

move.hs:77:34:
    Couldn't match expected type ‘(t, t1)’ with actual type ‘Node’
    Relevant bindings include
      remoteHost :: t (bound at move.hs:77:9)
      remotePort :: t1 (bound at move.hs:77:20)
    In the expression: head $ tail nodes
    In a pattern binding: (remoteHost, remotePort) = head $ tail nodes
    In the expression:
      do { nodes <- logged getNodes;
           logged $ liftIO $ print $ "NODES=" ++ show nodes;
           let (remoteHost, remotePort) = head $ tail nodes;
           examples' remoteHost remotePort }
xcabal: Error: some packages failed to install:
transient-0.1.0.1 failed during the building phase. The exception was:
ExitFailure 1

Hi, does transient heavily rely on network-info? Would it be hard or would it be possible to drop this dependency?

The reason for it is that there is a 1 year old bug in network-info that doesn't allow cross-compiling. I'd love to give transient a try in the environment that requires cross-compilation and network-info seems to be the only outstanding issue preventing me from compiling transient.

Cheers.

There are two versions of state management primitives, ending with Data and ending with State, it creates unnecessary confusion. It is better to not have redundancy at all, therefore we can deprecate and later remove one set of these. I guess the ones with State suffix should be removed.

https://gitter.im/Transient-Transient-Universe-HPlay/Lobby?at=5e57aeadfcb6354d55de5f90

I am running a distributed pi example on 3 physical nodes where I connect nodes one-by-one to the same "main" node.

What I see is that the 3rd node prints the option message twice. Does it mean that every message is received twice?

Here are the connect process logs:

Node 1 (main, 192.168.1.6)

connecting to: ("192.168.1.6",PortNumber 1111)
Connected to modes: [("192.168.1.6",PortNumber 1111)]
Enter  "start"  to: Start the calculation
Connected to modes: [("192.168.1.153",PortNumber 1111),("192.168.1.6",PortNumber 1111)]
Connected to modes: [("192.168.1.153",PortNumber 1111),("192.168.1.6",PortNumber 1111)]
Connected to modes: [("192.168.1.151",PortNumber 1111),("192.168.1.153",PortNumber 1111),("192.168.1.6",PortNumber 1111)]

Node 2 (192.168.1.153)

Press end to exit
connecting to: ("192.168.1.6",PortNumber 1111)
Enter  "start"  to: Start the calculation

Node 3 (192.168.1.151)

Press end to exit
connecting to: ("192.168.1.6",PortNumber 1111)
Enter  "start"  to: Start the calculation
Enter  "start"  to: Start the calculation

Hello Alberto,

shouldn't there be user level functions in the Logged module to allow fetching and setting the log?

I wan't to persist the log to disk. Currently I would have to use Log from the internal package together with getData and setData.

Do I need to to persist all components of Log?

I am also not sure how to set all the variables of Log correctly when setting the restored log I have retrieved from the disk before starting the computation.

Rene.

Could you please update Hackage to the version where clustered/mclustered issue is fixed?

Using keep causes problems when working interactively in ghci. Very frustrating.

I should be emphasized near the beginning of the tutorial that keep' should be used instead of keep in interactive sessions.

Learn more about this new warning:

src/Transient/Logged.hs:136:11: warning: [-Wsimplifiable-class-constraints]
    The constraint ‘Loggable a’ matches an instance declaration
    instance (Show a, Read a, Typeable a) => Loggable a
      -- Defined at src/Transient/Logged.hs:31:10
    This makes type inference for inner bindings fragile;
      either use MonoLocalBinds, or simplify it using the instance

full log

Same warning appears in Transient Universe builds multiple times.

This program creates three tasks under the original thread (since threads 0 precludes the creation of new threads for choose). It uses the backtracking promitives:

main= keep' $ do
   x <- return 0 `onBack` \s ->  do
            forward ""
            case s of
                "a" -> threads 0 $ choose [1..3 :: Int]
                _   -> empty
   liftIO $ print x
   if x== 0 then back "a" else back "b"
   return ()

It produces:
0 -- generated by return 0
1 -- due to onBack and choose
2 -- due to onBack and choose
3 -- due to onBack and choose

Now, the same/equivalent program using exceptions only produces two results:

main= keep' $ do
   x <- return 0 `onException'` \(ErrorCall s) ->  do
            continue
            case s of
                "a" -> threads 0 $ choose [1..3 :: Int]
                _   -> empty
   liftIO $ print x
   if x== 0 then error "a"  else error "b"
   return ()

0
1

It seems that the first time that empty is executed, after choose 1 has been executed, which triggers "b" exception, stop the whole computation. It shouldn't, since exceptions are managed by the same mechanism.

I cannot get this code to run to completion and end gracefully. It always ends with a thread blocked indefinitely in an MVar operation error.

import Transient.Base
import Transient.Indeterminism
import Control.Monad.IO.Class
import Control.Concurrent (threadDelay)
import System.Random (randomRIO)

f _ = do
  threadDelay =<< randomRIO (0, 10000000)
  pure ["1", "2", "3"]

main :: IO ()
main = keep' $ do
  uris <- liftIO $ f ()
  url <- choose uris
  liftIO $ (mapM_ putStrLn =<< f url)

The program should print 1, 2, and 3, 3 times each. I do not care what order they are printed in, only that each number is printed 3 times.

I am using Stackage LTS 7.14.

Build failure with GHC 8.2.1 as seen on the Stackage build server.

[1 of 6] Compiling Transient.Internals ( src/Transient/Internals.hs, dist/build/Transient/Internals.o )

src/Transient/Internals.hs:96:26: error:
    Not in scope: type constructor or class ‘TypeRep’
   |
96 |   , mfData      :: M.Map TypeRep SData
   |                          ^^^^^^^

We catch the IO exceptions in the MonadIO instance and handle them using the backtracking mechanism. However exceptions that might be thrown in the transient monad itself are not caught and handled in the same way, they just escape outside. We must have a consistent exception handling.

Implement some benchmarks and make all Travis builds to run them.

When running this code

import Control.Applicative ((<$>), (<*>), (<|>))
import Control.Monad.IO.Class (liftIO)
import Transient.Base (keep, threads, async)

main = keep $ threads 1 $ do
    x <- (,) <$> (event 1 <|> event 2) <*> (event 3 <|> event 4)
    liftIO $ putStrLn $ show x
    where
        event n = async (return n :: IO Int)

Four results are expected:

(1,3)
(1,4)
(2,3)
(2,4)

But it actually produces only the last two.

@agocorona should we update URLs in our packages, e.g.

…/agocorona/transient ➞ …/transient-haskell/transient

in all packages?

Collect and group behave differently for the same failure case. When we are trying to collect more results than the number of tasks we have started collect and group show different behavior. Collect is successful though it returns less results than requested after a long while (When BlockedIndefinitelyOnMVar is caught). On the other hand group fails and stops the computation. Try the following to reproduce:

import Control.Monad.IO.Class (liftIO)
import Data.List (sum)
import Transient.Internals (keep)
import Transient.Indeterminism (choose, collect, group)

main = keep $ do
    --collect 101 squares >>= liftIO . putStrLn . show . sum
    group 101 squares >>= liftIO . putStrLn . show . sum
    where
        squares = do
            x <- choose [1..100]
            return (x * x)

Hi, I have been reading through the transient tutorial and I happened upon a strange hang induced by the following code:

main = keep (((,) <$> async (print 1) <*> liftIO (print 2)) *> liftIO (print 3))

For some reason, this prints 1 and 2, but never 3. However, using do-notation works fine:

main = keep $ do
  _ <- (,) <$> async (print 1) <*> liftIO (print 2)
  liftIO (print 3)

Any idea what might be going on? Thank you!

I would suggest the following:

1. Move keep, option, input to a separate module, say Transient.ConsoleIO. We can possibly call the new module Transient.Input if there are more input methods to be covered. We can rename keep to something more appropriate like withConsole or withAsyncInput or interactive.

2. keep' and exit can stay in Base. But we can rename keep' to something more appropriate likewait or stayput.

Amazing work !

few typos I found while reading:

However, while spark is a framework, transient is a pure library where the primitives are first class in the language, and map-reduce is a particular operation int transient, but it is not hard-coded in a rigid framework.

=> in

import Transient.EVars

main= keep $ do
  var <- newEVar
  comp1 <|>  comp3
  liftIO $ putStrLn "world"

comp1= do
   r <- readEVar var
   liftIO $ putStrLn r

comp2= writeEVar var "hello"

comp1 <|> comp3 => comp1 <|> comp2

Like in the case of spark, reduce is the operation that executes everithing, while map is declarative. Let's see an example:

-> everything

In the last example on input elements, this line occurs:

r <- getRadio [fromStr v ++> setRadioActive v n | v <- ["red","green","blue"]]

But n is not defined anywhere...

Hi,

I'm trying to get transient working but unfortunately I get this error:

Configuring library for transient-0.6.0.1..
Preprocessing library for transient-0.6.0.1..
Building library for transient-0.6.0.1..
[1 of 7] Compiling Transient.Internals ( src/Transient/Internals.hs, dist/build/Transient/Internals.o )

src/Transient/Internals.hs:368:10: error:
    • Could not deduce (Semigroup (TransIO a))
        arising from the superclasses of an instance declaration
      from the context: Monoid a
        bound by the instance declaration
        at src/Transient/Internals.hs:368:10-39
    • In the instance declaration for ‘Monoid (TransIO a)’
    |
368 | instance Monoid a => Monoid (TransIO a) where
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cabal: Failed to build transient-0.6.0.1 (which is required by
exe:effects-playground from effects-playground-0.1.0.0). See the build log
above for details.

Any ideas on how to solve it?

I have a few other dependencies such as polysemy and fused-effects working.

I just noticed the compile failure below w/ the latest release

(see also http://104.239.175.197:8080/package/transient for an overall status)

Configuring component lib from transient-0.5.3...
Preprocessing library transient-0.5.3...
[1 of 6] Compiling Transient.Internals ( src/Transient/Internals.hs, /tmp/matrix-worker/1490802822/dist-newstyle/build/x86_64-linux/ghc-8.0.2/transient-0.5.3/build/Transient/Internals.o )
[2 of 6] Compiling Transient.Indeterminism ( src/Transient/Indeterminism.hs, /tmp/matrix-worker/1490802822/dist-newstyle/build/x86_64-linux/ghc-8.0.2/transient-0.5.3/build/Transient/Indeterminism.o )
[3 of 6] Compiling Transient.Logged ( src/Transient/Logged.hs, /tmp/matrix-worker/1490802822/dist-newstyle/build/x86_64-linux/ghc-8.0.2/transient-0.5.3/build/Transient/Logged.o )

src/Transient/Logged.hs:74:54: error:
    • Couldn't match expected type ‘Data.ByteString.Lazy.Internal.ByteString’
                  with actual type ‘BS.ByteString’
      NB: ‘BS.ByteString’ is defined in ‘Data.ByteString.Internal’
          ‘Data.ByteString.Lazy.Internal.ByteString’
            is defined in ‘Data.ByteString.Lazy.Internal’
    • In the first argument of ‘deserialize’, namely ‘logstr’
      In the second argument of ‘seq’, namely ‘deserialize logstr’
      In the expression: BS.length logstr `seq` deserialize logstr

src/Transient/Logged.hs:195:33: error:
    • Couldn't match expected type ‘Data.ByteString.Lazy.Internal.ByteString’
                  with actual type ‘Data.ByteString.Builder.Internal.Builder’
    • In the second argument of ‘(==)’, namely ‘show1 n’
      In the expression: s == show1 n
      In the expression:
        if s == show1 n then
            do { setData $ Log recover t full;
                 return $ Just () }
        else
            return Nothing

With the current design of the logging primitives, it is possible to easily run into inconsistent behavior. For example:

main = keep $ restore $ do
     r <- choose [1..5 :: Int]
     logged $ liftIO $ print ("A",r)
     suspend ()
     liftIO $ print ("B",r)

Here, the programmer forgot to add logged to the choose primitive. This results in an incorrect output. Another example, if we use suspend without logging, the program always suspends:

main = keep $ do
     r <- choose [1..5 :: Int]
     liftIO $ print ("A",r)
     suspend ()
     liftIO $ print ("B",r)

Instead we should either throw an error or ignore the suspend since we are not logging.

A potential solution to these problems is to design the API such that we use the logged primitive one level above the suspend. If the parent is logged then logging is inherited by all the child computations and we can suspend a child. Since we know the logged/not logged state we can change the behavior of suspend based on that. Also, the logged primitive itself can do the job of restore as well. The code will look like:

main = keep $ logged $ do
     r <- choose [1..5 :: Int]
     liftIO $ print ("A",r)
     suspend ()
     liftIO $ print ("B",r)

I have not thought much about it, so I am not sure if this solution has any inherent problems.

Since we create threads quite often it may be helpful to use raw fork instead of forkIO. Something like this:

-- A version of forkIO that does not include the outer exception
-- handler: saves a bit of time when we will be installing our own
-- exception handler.
{-# INLINE rawForkIO #-}
rawForkIO :: IO () -> IO ThreadId
rawForkIO action = IO $ \ s ->
   case (fork# action s) of (# s1, tid #) -> (# s1, ThreadId tid #)

LICENSE and cabal file say MIT but source files say GPL3. Would be great if licensing could be made consistent throughout.

After removing the n (#20) I start the server using stack runghc app/Main.hs. I open my browser at localhost:8081, and it says it can't find Index.html (I wrote a comment on a commit for that). How do I launch the example from the bottom of the tutorial?

This program won't compile:

main = keep $ error "Raised ErrorCall exception"

But this compiles fine:

main = keep $ error "Raised ErrorCall exception" >> return ()

That's because keep has a Typeable constraint. Is there a way around this? If not we should at least document this behavior.

groupByTime asynchronous primitive returns the first set of results after the specified time interval but returns the next set of results immediately.

In the current implementation if there is no transient level exception handler installed we just silently ignore the IO exception. Consider this simple program:

main = keep $ liftIO $ error "Raised ErrorCall exception" >> return ()

Instead we should re-raise the exception if there is no handler installed, so that it does not go unnoticed. See the MonadIO instance and the back function for relevant code.

Consider this program:

main = keep $ restore $ do
     r <- logged $ threads 3 $ choose [1..10 :: Int]
     logged $ liftIO $ print ("A",r)
     suspend ()
     logged $ liftIO $ print ("B",r)

This does not perform full 10 iterations.

I am by far the least qualified person to review your work, but I find it relentlessly fascinating. I want your work to gain more traction, but I think that English is not your first language. I would like to offer my help to do some editing on your FP-complete posts to make them sound more natural in English. I can't offer much, but as I read through them I could offer some edits/questions that might help your posts gain more traction.

The finish primitives are almost the same as the exception primitives. If we rename the finish primitive to throwException or equivalent we can just get rid or all other finish primitives and we get the ability to throw custom/user defined exceptions as well. With this, finish can just become a special exception type.

      r <-  (runAt node1 (return "hello "))
                <>  (runAt node2 (return "world" ))

      runAt nodex whatever

fails at whatever with connection error. It only happens with <*> or <> in the line before

• Test simple continuation works
• Test asynchronous execution
• Test events

@agocorona what else?

If that happens, do:
apt-get install netbase

snoyberg/http-client#292

I'm not sure if this is the right place to submit this issue, but I notice that the profits example on fpcomplete require a final price event in the eventList in order to print out the total.

For example, if I changed the eventList to

eventList =
    [ Event "quantity" 10
    , Event "price"     2
    , Event "price"     3
    , Event "quantity" 30
    ]

The the output is

new event: quantity
quantity=10

new event: price
price=2

total=20

new event: price
price=3

total=30

new event: quantity
quantity=30

END

That is, there is no final total=90 emitted.

Is this the expected behaviour?

An example that shows how to use the distributed features on "real" nodes would be appreciated: how to create remote nodes, should listen be called only for the local node or for all nodes, how to wait until all the nodes connect, etc.

This code does not backtrack to the first handler:

main= keep $ do
  onException \(e :: ErrorCall) -> liftIO $ print e
  onException \(e :: SomeConcreteException) -> liftIO $ print e
  error "err"
  return()

I met a juicy 404 instead

Currently, before you can use callService/callService' to call services on other nodes, you must use connection, listen or initNode established some connection state. This restrict should be removed because callService/callService' know how to connect to the node where the services it wants to call located.