@@ -5,13 +5,13 @@ title: Documentation
55
66### Cloud Haskell Platform
77
8- This is the [ * Cloud Haskell Platform* ] [ 1 ] . Cloud Haskell is a set of libraries
8+ This is the [ * Cloud Haskell Platform* ] [ cloud-haskell ] . Cloud Haskell is a set of libraries
99that bring Erlang-style concurrency and distribution to Haskell programs. This
1010project is an implementation of that distributed computing interface, where
1111processes communicate with one another through explicit message passing rather
1212than shared memory.
1313
14- Originally described by the joint [ Towards Haskell in the Cloud] [ 12 ] paper,
14+ Originally described by the joint [ Towards Haskell in the Cloud] [ haskell11-ch ] paper,
1515Cloud Haskell has be re-written from the ground up and supports a rich and
1616growing number of features for
1717
@@ -22,40 +22,37 @@ growing number of features for
2222* working with several network transport implementations (and more in the pipeline)
2323* supporting * static* values (required for remote communication)
2424
25- There is a recent
26- [ presentation] ( http://sneezy.cs.nott.ac.uk/fun/2012-02/ coutts-2012-02-28.pdf )
25+ There is a
26+ [ presentation] [ fun201202- coutts]
2727on Cloud Haskell and this reimplementation, which is worth reading in conjunction
2828with the documentation and wiki pages on this website..
2929
3030Cloud Haskell comprises the following components, some of which are complete,
3131others experimental.
3232
33- * [ distributed-process] [ 2 ] : Base concurrency and distribution support
34- * [ distributed-process-platform] [ 3 ] : The Cloud Haskell Platform - APIs
35- * [ distributed-static] [ 4 ] : Support for static values
36- * [ rank1dynamic] [ 5 ] : Like ` Data.Dynamic ` and ` Data.Typeable ` but supporting polymorphic values
37- * [ network-transport] [ 6 ] : Generic ` Network.Transport ` API
38- * [ network-transport-tcp] [ 7 ] : TCP realisation of ` Network.Transport `
39- * [ network-transport-inmemory] [ 8 ] : In-memory realisation of ` Network.Transport ` (incomplete)
40- * [ network-transport-composed] [ 9 ] : Compose two transports (very preliminary)
41- * [ distributed-process-simplelocalnet] [ 10 ] : Simple backend for local networks
42- * [ distributed-process-azure] [ 11 ] : Azure backend for Cloud Haskell (proof of concept)
33+ * [ distributed-process] [ distributed-process ] : Base concurrency and distribution support
34+ * [ distributed-process-platform] [ distributed-process-platform ] : The Cloud Haskell Platform - APIs
35+ * [ distributed-static] [ distributed-static ] : Support for static values
36+ * [ rank1dynamic] [ rank1dynamic ] : Like ` Data.Dynamic ` and ` Data.Typeable ` but supporting polymorphic values
37+ * [ network-transport] [ network-transport ] : Generic ` Network.Transport ` API
38+ * [ network-transport-tcp] [ network-transport-tcp ] : TCP realisation of ` Network.Transport `
39+ * [ network-transport-inmemory] [ network-transport-inmemory ] : In-memory realisation of ` Network.Transport ` (incomplete)
40+ * [ network-transport-composed] [ network-transport-composed ] : Compose two transports (very preliminary)
41+ * [ distributed-process-simplelocalnet] [ distributed-process-simplelocalnet ] : Simple backend for local networks
42+ * [ distributed-process-azure] [ distributed-process-azure ] : Azure backend for Cloud Haskell (proof of concept)
4343
4444One of Cloud Haskell's goals is to separate the transport layer from the
45- * process layer* , so that the transport backend is entirely independent:
46- it is envisaged that this interface might later be used by models
47- other than the Cloud Haskell paradigm, and that applications built
48- using Cloud Haskell might be easily configured to work with different
49- backend transports.
45+ * process layer* , so that the transport backend is entirely independent. In fact
46+ other projects can and do reuse the transport layer, even if they don't use or
47+ have their own process layer (see e.g. [ HdpH] [ hdph ] ).
5048
5149Abstracting over the transport layer allows different protocols for
5250message passing, including TCP/IP, UDP,
53- [ MPI] ( http://en.wikipedia.org/wiki/Message_Passing_Interface ) ,
51+ [ MPI] ( http://en.wikipedia.org/wiki/Message_Passing_Interface ) ,
5452[ CCI] ( http://www.olcf.ornl.gov/center-projects/common-communication-interface/ ) ,
55- ZeroMQ, SSH, MVars, Unix pipes, and more. Each of these transports would provide
56- its own implementation of the ` Network.Transport ` and provide a means of creating
57- new connections for use within ` Control.Distributed.Process ` . This separation means
58- that transports might be used for other purposes than Cloud Haskell.
53+ [ ZeroMQ] ( http://zeromq.org ) , [ SSH] ( http://openssh.com ) , MVars, Unix pipes, and more. Each of these transports provides
54+ its own implementation of the ` Network.Transport ` API and provide a means of creating
55+ new connections for use within ` Control.Distributed.Process ` .
5956
6057The following diagram shows dependencies between the various subsystems,
6158in an application using Cloud Haskell, where arrows represent explicit
@@ -94,8 +91,8 @@ In this diagram, the various nodes roughly correspond to specific modules:
9491 Transport Implementation : Network.Transport.*
9592
9693An application is built using the primitives provided by the Cloud
97- Haskell layer, provided by ` Control.Distributed.Process ` module, which
98- provides abstractions such as nodes and processes.
94+ Haskell layer, provided by the ` Control.Distributed.Process ` module, which
95+ defines abstractions such as nodes and processes.
9996
10097The application also depends on a Cloud Haskell Backend, which
10198provides functions to allow the initialisation of the transport layer
@@ -105,21 +102,21 @@ It is, of course, possible to create new Cloud Haskell nodes by
105102using a Network Transport Backend such as ` Network.Transport.TCP `
106103directly.
107104
108- The Cloud Haskell interface and backend, make use of the Transport
105+ The Cloud Haskell interface and backend make use of the Transport
109106interface provided by the ` Network.Transport ` module.
110107This also serves as an interface for the ` Network.Transport.* `
111108module, which provides a specific implementation for this transport,
112- and may, for example, be based on some external library written in
109+ and may, for example, be based on some external library written in
113110Haskell or C.
114111
115112### Network Transport Abstraction Layer
116113
117- Cloud Haskell's generic [ network-transport] [ 6 ] API is entirely independent of
114+ Cloud Haskell's generic [ network-transport] [ network-transport ] API is entirely independent of
118115the concurrency and messaging passing capabilities of the * process layer* .
119116Cloud Haskell applications are built using the primitives provided by the
120- * process layer* (i.e., [ distributed-process] [ 2 ] ), which provides abstractions
117+ * process layer* (i.e., [ distributed-process] [ distributed-process ] ), which provides abstractions
121118such as nodes and processes. Applications must also depend on a Cloud Haskell
122- Backend , which provides functions to allow the initialisation of the transport
119+ backend , which provides functions to allow the initialisation of the transport
123120layer using whatever topology might be appropriate to the application.
124121
125122` Network.Transport ` is a network abstraction layer geared towards specific
@@ -128,7 +125,7 @@ classes of applications, offering the following high level concepts:
128125* Nodes in the network are represented by ` EndPoint ` s. These are heavyweight stateful objects.
129126* Each ` EndPoint ` has an ` EndPointAddress ` .
130127* Connections can be established from one ` EndPoint ` to another using the ` EndPointAddress ` of the remote end.
131- * The ` EndPointAddress ` can be serialised and sent over the network, where as ` EndPoint ` s and connections cannot.
128+ * The ` EndPointAddress ` can be serialised and sent over the network, whereas ` EndPoint ` s and connections cannot.
132129* Connections between ` EndPoint ` s are unidirectional and lightweight.
133130* Outgoing messages are sent via a ` Connection ` object that represents the sending end of the connection.
134131* Incoming messages for ** all** of the incoming connections on an ` EndPoint ` are collected via a shared receive queue.
@@ -160,16 +157,16 @@ of other `Network.Transport` APIs if required, but for the most part this
160157is irrelevant and the application will interact with Cloud Haskell through
161158the * Process Layer* and * Platform* .
162159
163- For more details about ` Network.Transport ` please see the [ wiki page] [ 20 ] .
160+ For more details about ` Network.Transport ` please see the [ wiki page] ( /wiki/networktransport.html ) .
164161
165162### Concurrency and Distribution
166163
167164The * Process Layer* is where Cloud Haskell's support for concurrency and
168165distributed programming are exposed to application developers. This layer
169- deals explicitly with
166+ deals explicitly with
170167
171168The core of Cloud Haskell's concurrency and distribution support resides in the
172- [ distributed-process] [ 2 ] library. As well as the APIs necessary for starting
169+ [ distributed-process] [ distributed-process ] library. As well as the APIs necessary for starting
173170nodes and forking processes on them, we find all the basic primitives required
174171to
175172
@@ -215,7 +212,7 @@ runProcess :: LocalNode -> Process () -> IO ()
215212{% endhighlight %}
216213
217214Once we've spawned some processes, they can communicate with one another
218- using the messaging primitives provided by [ distributed-processes] [ 2 ] ,
215+ using the messaging primitives provided by [ distributed-processes] [ distributed-processes ] ,
219216which are well documented in the haddocks.
220217
221218### What is Serializable
@@ -254,10 +251,10 @@ We create channels with a call to `newChan`, and send/receive on them using the
254251channelsDemo :: Process ()
255252channelsDemo = do
256253 (sp, rp) <- newChan :: Process (SendPort String, ReceivePort String)
257-
254+
258255 -- send on a channel
259256 spawnLocal $ sendChan sp "hello!"
260-
257+
261258 -- receive on a channel
262259 m <- receiveChan rp
263260 say $ show m
@@ -272,7 +269,7 @@ need to spawn a process and send a bunch a messages to it, then wait for
272269replies however; we can’t send a ` ReceivePort ` since it is not ` Serializable ` .
273270
274271` ReceivePort ` s can be merged, so we can listen on several simultaneously. In the
275- latest version of [ distributed-process] [ 2 ] , we can listen for * regular* messages
272+ latest version of [ distributed-process] [ distributed-process ] , we can listen for * regular* messages
276273and multiple channels at the same time, using ` matchChan ` in the list of
277274allowed matches passed ` receiveWait ` and ` receiveTimeout ` .
278275
@@ -313,7 +310,7 @@ and decide whether to oblige or not.
313310
314311### Rethinking the Task Layer
315312
316- [ Towards Haskell in the Cloud] [ 12 ] describes a multi-layered architecture, in
313+ [ Towards Haskell in the Cloud] [ haskell11-ch ] describes a multi-layered architecture, in
317314which manipulation of concurrent processes and message passing between them
318315is managed in the * process layer* , whilst a higher level API described as the
319316* task layer* provides additional features such as
@@ -322,19 +319,19 @@ is managed in the *process layer*, whilst a higher level API described as the
322319* data centric processing model
323320* a promise (or * future* ) abstraction, representing the result of a calculation that may or may not have yet completed
324321
325- The [ distributed-process-platform] [ 18 ] library implements parts of the
322+ The [ distributed-process-platform] [ distributed-process-platform ] library implements parts of the
326323* task layer* , but takes a very different approach to that described
327- in the original paper and implemented by the [ remote] [ 14 ] package. In particular,
324+ in the original paper and implemented by the [ remote] [ remote ] package. In particular,
328325we diverge from the original design and defer to many of the principles
329- defined by Erlang's [ Open Telecom Platform] [ 13 ] , taking in some well established
326+ defined by Erlang's [ Open Telecom Platform] [ OTP ] , taking in some well established
330327Haskell concurrency design patterns along the way.
331328
332- In fact, [ distributed-process-platform] [ 18 ] does not really consider the
329+ In fact, [ distributed-process-platform] [ distributed-process-platform ] does not really consider the
333330* task layer* in great detail. We provide an API comparable to remote's
334- ` Promise ` in Control.Distributed.Process.Platform.Async. This API however,
335- is derived from Simon Marlow's [ Control.Concurrent.Async] [ 19 ] package, and is not
331+ ` Promise ` in ` Control.Distributed.Process.Platform.Async ` . This API however,
332+ is derived from Simon Marlow's [ Control.Concurrent.Async] [ async ] package, and is not
336333limited to blocking queries on ` Async ` handles in the same way. Instead our
337- [ API] [ 17 ] handles both blocking and non-blocking queries, polling
334+ [ API] [ d-p-platform-async ] handles both blocking and non-blocking queries, polling
338335and working with lists of ` Async ` handles. We also eschew throwing exceptions
339336to indicate asynchronous task failures, instead handling * task* and connectivity
340337failures using monitors. Users of the API need only concern themselves with the
@@ -356,13 +353,13 @@ demoAsync = do
356353
357354 -- we can cancel the task if we want to
358355 -- cancel hAsync
359-
356+
360357 -- or cancel it and wait until it has exited
361358 -- cancelWait hAsync
362-
359+
363360 -- we can wait on the task and timeout if it's still busy
364361 Nothing <- waitTimeout (within 3 Seconds) hAsync
365-
362+
366363 -- or finally, we can block until the task is finished!
367364 asyncResult <- wait hAsync
368365 case asyncResult of
@@ -379,7 +376,7 @@ around `Async` that disallows side effects is relatively simple, and we
379376do not consider the presence of side effects a barrier to fault tolerance
380377and automated process restarts. Erlang does not forbid * IO* in its processes,
381378and yet that doesn't render supervision trees ineffective. They key is to
382- provide a rich enough API that statefull processes can recognise whether or
379+ provide a rich enough API that stateful processes can recognise whether or
383380not they need to provide idempotent initialisation routines.
384381
385382The utility of preventing side effects using the type system is, however, not
@@ -391,7 +388,7 @@ Work is also underway to provide abstractions for managing asynchronous tasks
391388at a higher level, focussing on workload distribution and load regulation.
392389
393390The kinds of task that can be performed by the async implementations in
394- [ distributed-process-platform] [ 3 ] are limited only by their return type:
391+ [ distributed-process-platform] [ distributed-process-platform ] are limited only by their return type:
395392it ** must** be ` Serializable ` - that much should've been obvious by now.
396393The type of asynchronous task definitions comes in two flavours, one for
397394local nodes which require no remote-table or static serialisation dictionary,
@@ -400,11 +397,11 @@ and another for tasks you wish to execute on remote nodes.
400397{% highlight haskell %}
401398-- | A task to be performed asynchronously.
402399data AsyncTask a =
403- AsyncTask
400+ AsyncTask
404401 {
405402 asyncTask :: Process a -- ^ the task to be performed
406403 }
407- | AsyncRemoteTask
404+ | AsyncRemoteTask
408405 {
409406 asyncTaskDict :: Static (SerializableDict a)
410407 -- ^ the serializable dict required to spawn a remote process
@@ -430,14 +427,14 @@ domain was more *haskell-ish* than working with bare send and receive primitives
430427The ` Async ` sub-package also provides a type safe interface for receiving data,
431428although it is limited to running a computation and waiting for its result.
432429
433- The [ Control.Distributed.Processes.Platform.ManagedProcess] [ 21 ] API provides a
430+ The [ Control.Distributed.Processes.Platform.ManagedProcess] [ d-p-platform-ManagedProcess ] API provides a
434431number of different abstractions that can be used to achieve similar benefits
435432in your code. It works by introducing a standard protocol between your process
436433and the * world outside* , which governs how to handle request/reply processing,
437434exit signals, timeouts, sleeping/hibernation with ` threadDelay ` and even provides
438435hooks that terminating processes can use to clean up residual state.
439436
440- The [ API documentation] [ 21 ] is quite extensive, so here we will simply point
437+ The [ API documentation] [ d-p-platform-ManagedProcess ] is quite extensive, so here we will simply point
441438out the obvious differences. A process implemented with ` ManagedProcess `
442439can present a type safe API to its callers (and the server side code too!),
443440although that's not its primary benefit. For a very simplified example:
@@ -489,8 +486,8 @@ API, which looks a lot like `Async` but manages exit signals in a single thread
489486configurable task pools and task supervision strategy part of its API.
490487
491488More complex examples of the ` ManagedProcess ` API can be seen in the
492- [ Managed Processes tutorial] [ 22 ] . API documentation for HEAD is available
493- [ here] [ 21 ] .
489+ [ Managed Processes tutorial] ( tutorials/tutorial3.html ) . API documentation for HEAD is available
490+ [ here] [ d-p-platform-ManagedProcess ] .
494491
495492### Supervision Trees
496493
@@ -500,25 +497,22 @@ TBC
500497
501498TBC
502499
503- [ 1 ] : http://www.haskell.org/haskellwiki/Cloud_Haskell
504- [ 2 ] : https://github.com/haskell-distributed/distributed-process
505- [ 3 ] : https://github.com/haskell-distributed/distributed-process-platform
506- [ 4 ] : http://hackage.haskell.org/package/distributed-static
507- [ 5 ] : http://hackage.haskell.org/package/rank1dynamic
508- [ 6 ] : http://hackage.haskell.org/package/network-transport
509- [ 7 ] : http://hackage.haskell.org/package/network-transport-tcp
510- [ 8 ] : https://github.com/haskell-distributed/network-transport-inmemory
511- [ 9 ] : https://github.com/haskell-distributed/network-transport-composed
512- [ 10 ] : http://hackage.haskell.org/package/distributed-process-simplelocalnet
513- [ 11 ] : http://hackage.haskell.org/package/distributed-process-azure
514- [ 12 ] : http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/remote.pdf
515- [ 13 ] : http://en.wikipedia.org/wiki/Open_Telecom_Platform
516- [ 14 ] : http://hackage.haskell.org/package/remote
517- [ 15 ] : http://www.erlang.org/doc/design_principles/sup_princ.html
518- [ 16 ] : http://www.erlang.org/doc/man/supervisor.html
519- [ 17 ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-Async.html
520- [ 18 ] : https://github.com/haskell-distributed/distributed-process-platform
521- [ 19 ] : http://hackage.haskell.org/package/async
522- [ 20 ] : /wiki/networktransport.html
523- [ 21 ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-ManagedProcess.html
524- [ 22 ] : /tutorials/tutorial3.html
500+ [ cloud-haskell ] : http://haskell-distributed.github.io/documentation.html
501+ [ fun201202-coutts ] : http://sneezy.cs.nott.ac.uk/fun/2012-02/coutts-2012-02-28.pdf
502+ [ distributed-process ] : https://github.com/haskell-distributed/distributed-process
503+ [ distributed-process-platform ] : https://github.com/haskell-distributed/distributed-process-platform
504+ [ distributed-static ] : http://hackage.haskell.org/package/distributed-static
505+ [ rank1dynamic ] : http://hackage.haskell.org/package/rank1dynamic
506+ [ network-transport ] : http://hackage.haskell.org/package/network-transport
507+ [ network-transport-tcp ] : http://hackage.haskell.org/package/network-transport-tcp
508+ [ network-transport-inmemory ] : https://github.com/haskell-distributed/network-transport-inmemory
509+ [ network-transport-composed ] : https://github.com/haskell-distributed/network-transport-composed
510+ [ distributed-process-simplelocalnet ] : http://hackage.haskell.org/package/distributed-process-simplelocalnet
511+ [ distributed-process-azure ] : http://hackage.haskell.org/package/distributed-process-azure
512+ [ hdph ] : http://hackage.haskell.org/package/hdph
513+ [ haskell11-ch ] : http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/remote.pdf
514+ [ OTP ] : http://en.wikipedia.org/wiki/Open_Telecom_Platform
515+ [ remote ] : http://hackage.haskell.org/package/remote
516+ [ d-p-platform-async ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-Async.html
517+ [ async ] : http://hackage.haskell.org/package/async
518+ [ d-p-platform-ManagedProcess ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-ManagedProcess.html
0 commit comments