8

I am working on a big project that involves a lot of web based and AI work. I am extremely comfortable with Python, though my only concern is with concurrent programming and scaling this project to make it work on clusters. Thus, Clojure for AI and support for Java function calls and bring about concurrent programming.

Is this a good idea to do all the web-based api work with Python and let Clojure take care of most of the concurrent AI work?

Edit: Let me explain the interaction in detail. Python would be doing most of the dirty work (scraping, image processing, improving the database and all that.) Clojure, if possible, would either deal with the data base or get the data from Python. I except something CPython sort of linking with Python and Clojure.

Edit2: Might be a foolish question to ask, but this being a rather long term project which will evolve quite a bit and go under several iterations, is Clojure a language here to stay? Is it portable enough?

Improve this question
12
  • 2
    What kind of interface are you thinking of between those layers? Jython? Shared database backend? Message queue? ...? As it stands, the question is just too broad to answer. Commented May 11, 2012 at 20:18
  • Have updated the question a bit. I am confused myself as to assign which task to whom and how to make the interaction work. Any suggestions will be more than welcome. Commented May 11, 2012 at 20:22
  • 1
    Don't cross the streams! Commented May 11, 2012 at 20:40
  • 1
    @Puck if you want to directly do calls between the languages, by far the easier approach is to use the JVM -- so Jython, not CPython. Doing "the dirty work" in Python seems suboptimal -- the JVM is quite a lot faster than the CPython runtime even before threading is taken into account. Commented May 11, 2012 at 20:40
  • 2
    I think it's pretty safe to say Clojure is here to stay. (And these days, even boutique languages tend to survive. After all, production-ready languages are free now. No more $600 just to buy a compiler.) As for portability, Clojure should be as portable as Java, so yeah, pretty portable. I really can't see any reason not to use Clojure. (Though there are lots of other great options too!) Commented May 11, 2012 at 20:41

4 Answers 4

Reset to default
14

I built an embarrassingly parallel number-crunching application with a backend in Clojure (on an arbitrary number of machines) and a frontend in Ruby on Rails. I don't particularly like RoR, but this was a zero-budget project at the time and we had a Rails programmer at hand who was willing to work for free.

The Clojure part consisted of (roughly) a controller, number crunching nodes, and a server implementing a JSON-over-HTTP API which was the interface to the Rails web app. The Clojure nodes used RabbitMQ to talk to each other. Because we defined clear APIs between different parts of the application, it was easy to later rewrite the frontend in Clojure (because that better suited our needs).

If you're working on a distributed project with a long life span and continuous development effort, it could make sense to design the application as a number of separate modules that communicate through well defined APIs (json, bson, ... over AMQP, HTTP, ... or a database). That means you can get started quickly using a language you're comfortable with, and rewrite parts in another language at a later stage if necessary.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

This seems almost exactly like OP's scenario, except swapping out Python and inserting RoR.
6

I can't see a big problem with using Python for the web apps and Clojure for the concurrent data crunching / back end code. I assume you would use something like JSON over http for the communications between the two, which should work fine.

I'd personally use Clojure for both (using e.g. the excellent Noir as a web framework and Korma for the database stuff.), but if as you say your experience is mostly in Python then it probably makes sense to stick with Python from a productivity perspective (in the short term at least).

To answer the questions regarding the future of Clojure:

  • It's definitely here to stay. It has a very active community and is probably one of the "hottest" JVM languages right now (alongside Scala and Groovy). It seems to be doing particularly well in the big data / analytics space
  • Clojure has a particular advantage in terms of library support, since it can easily make use of any Java libraries. This is a huge advantage for a new langauge from a practical perspective, since it immediately solves what is usually one of the biggest issues in getting a new language ecosystem off the ground.
  • Clojure is a new language that is still undergoing quite a lot of development. If you choose to use Clojure, you should be aware that you will need to put in some effort to stay current and keep your code up to date with the latest Clojure versions. I've personally not found this to be an issue, but it may come as a surprise to people used to more "stable" languages like Java.
  • Clojure is very portable - it will basically run anywhere that you can get a reasonably modern JVM, which is pretty much everywhere nowadays.
Improve this answer

Comments

4

If you can build both sides to use Data and Pure(ish) Functions to communicate then this should work very well. wrapping your clojure functions in web services that take and retrun JSON (or more preferably clojure forms) should make them accessible to your Python based front end will no extra fuss.

Of course it's more fun to write it in Clojure all the way through. ;)

If this is a long term project than building clean Functional (as in takes and returns values) interfaces that exchange Data becomes even more important because it will give you the ability to evolve the components independently.

Improve this answer

1 Comment

Thank you for the answer. Have edited the question for the second time. Do take a look.
0

In such scenarios I personally like to start in the below sequence.

  • Divide the system into subsystems with "very clear" definition of what each system does and that definition should follow the principle of "do one thing and keep it simple". At this stage don't think about language etc.
  • Choose the platform (not languages) on which these subsystems will run. Ex: JVM, Python VM, NodeJs, CLR(Mono), other VMs. Try to select few platforms or if possible just one as that does make life easier down the road in terms of complexity.
  • Choose the language to program those platforms. This is very subjective but for JVM you can go with Clojure or Jython (in case you like Dynamic languages as I do).

As far as Clojure future is concerned, this is a language developed by "community of amazing programmers" and not by some corporation. I hope that clears your doubt about the "long term" concern of Clojure. By the way Clojure is LISP, so you can modify the language the way you want it and fix things yourself even if someone don't do that for you.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.