1

I have the following problem to solve.

I need to write a java program that:

  1. reads JSON object j1,j2,...,jn from a web service.
  2. does some number crunching on each object to come up with j1',j2',...,jn'
  3. Sends objects j1',j2',...,jn' to a web service.

The computational, space requirements for steps 1,2, and 3 can vary at any given time.

For example:

  1. The time it takes to process a JSON object at step 2 can vary depending on the contents of the JSON Object.
  2. The rate of objects being produced by the webservice in step 1 can go up or down with time.
  3. The consuming web service in step 3 can get backlogged.

To address the above design concerns want to implement the following architecture:

enter image description here

  1. Read JSON objects from the external webservice and place them on a Q
  2. An automatically size adjusting worker thread pool that consumes JSON objects from the Q and processes them. After processing them, places the resulting objects on the second Q
  3. An automatically size adjusting worker thread pool that consumes JSON objects from the second Q to send them to the consuming webservice.

Question:

I am curious if there is framework which I can use to solve this problem?

Notes:

  1. I can solve this using a range of components like custom Queues, Threadpools using the concurrency package -- however I'm looking for a solution that allows the writing of such a solution.
  2. This is not going to live inside a container. This will be a Java process where the entry point is public static void main(String args[])
  3. However if there is a container suited to this paradigm I would like to learn about it.
  4. I could split this into multiple processes, however I'd like to keep it very simple and in a single process.

Thanks.

Thanks.

Improve this question
1
  • Sorry about that ... must be my tired brain. Commented Oct 23, 2012 at 18:53

2 Answers 2

Reset to default
1

try Apache camel or Spring Integration to wire things up. These are kind of integration frameworks, will ease your interaction with webservices. What you need to do is define a route from webservice 1 -> number cruncher -> web service 2. routing and conversion required in between can be handled by the framework itself

you'd implement your cruncher as a camel processor. parallelizing your cruncher may be achieved via SEDA; Camel has a component for this pattern. Another alternate would be AsyncProcessor

I'd say you first take a look at the principles behind frameworks like camel. The abstractions they create are very relevant for the problem in hand.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Mmm, @mzzzzb, how do you think Twitter Storm would fit the bill?
@user1172468, im afraid i dont know anything about Storm. just looked into it and seems interesting, not in a position to advise. one more thing to learn :)
1

I'm not exactly sure what the end question is for your post, but you have a reasonable design concept. One question I have for you is what environment are you in? Are you in a JavaEE container or just a simple standalone application?

If you are in a container, it would make more sense to have Message Driven Beans processing off of the JMS queues than having a pool of worker threads.

If in your own container, it would make more sense for you to manage the thread pool yourself. With that said, I would also consider having separate applications running that pull the work off of the queues which would lead to a better scaling architecture for you. If the need ever came up, you could add more machines with more workers pointing at the one queue.

Improve this answer

2 Comments

Many thanks @Jeff I'll edit the question and refine it based on your input but basically: 1. No Container 2. Want to keep is simple: one single process
To my knowledge, there isn't any one stop shop frameworks but you can link a few together. You can use a framework such as Apache Axis for consuming and pushing to web services and the Apache Commons Pooling for the pooling for threads and automatically growing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.