Java library for Pattern mining on time based data

Question

We have a number representing volume of posts per minute on numerous subjects. We want to be able to find patterns so that we can predict what the volume of posts will be in the future.

We want the pattern detection process to be automatic (no human interaction required) and we have been wondering if there is any way to automate it. We have been reading about "pattern mining", but we haven't been able to find any java libraries we could start working with.

Are there any Java libraries for Pattern mining on time based data that we could use to automate this pattern mining process? Thanks in advance.

Why don't you use SimpleDateFormat if it is about date parsing? Or, even better, Joda Time? Can you give a sample? — fge
– fge, Commented Jan 5, 2012 at 14:50
By "automate" I would write code to do it. Data mining tends to be very specific to what you are trying to mine and how the data is represented. Computers don't automagically find patterns like a human would, you need to write code to determine how closely the data fits a model. — Peter Lawrey
– Peter Lawrey, Commented Jan 5, 2012 at 14:51

Phil · Accepted Answer · 2017-08-17 15:01:57Z

2

You could check my open source Data Mining framework: http://www.philippe-fournier-viger.com/spmf/ (SPMF)

It provides more than 130 algorithms. Several of them are designed to perform time-related data mining tasks such as discovering sequential patterns and sequential rules in a set of sequences.

I don't know if some of them would fit your needs. But you can check it out.

edited Aug 17, 2017 at 15:01

answered Mar 1, 2012 at 0:28

Phil

3,5303 gold badges36 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Chris J · Accepted Answer · 2012-01-05 15:05:15Z

0

Well, there are some pretty solid data mining libraries out there for Java. I have used WEKA for classification and association mining (http://www.cs.waikato.ac.nz/ml/weka/). I have also used Mahout for clustering (http://mahout.apache.org/). You do need to know what you are doing beforehand with regards handling your data. 95% of your time will be spent on cleaning out bad data and preprocessing what remains into a format that can be used by those frameworks.

You are a little light on details of your problem. Sounds like you want some sort of alerting system based on the frequency of some set of events in your data. If it is purely based on frequencies (i.e. - something that defies your assumed distribution of events over time) then you probably want to look at a technique call n-gramming. Trying to avoid being esoteric here but you want to use n-grams where n is not a fixed length. It is a technique common used in NLP and if you are familiar with the longest common substring problem, then you should have some idea of how to got about doing it.

answered Jan 5, 2012 at 15:05

Chris J

9,2827 gold badges42 silver badges40 bronze badges

2 Comments

rreyes1979 Over a year ago

I have given a bit more detail on the problem. I am not sure n-gramming will be able to help us here. Nevertheless, are there any other ideas you could suggest us?

Chris J Over a year ago

Okay, what you want to do is build a regression model. I'm not really a regression guy but they are easy enough do if you are using a framework like WEKA. In fact, here is an article about it: ibm.com/developerworks/opensource/library/os-weka1/index.html. Basically from your standpoint, you want to format your data, feed it into WEKA, persist the regression model that you train, then use that model to do predictions.

nflacco · Accepted Answer · 2012-01-08 04:43:30Z

You are looking to do two very different things: 1) text classification (topics) 2) predicting future topics/volume

1) For text classification, any of the standard NLP libraries is ok- GATE, OpenNLP, LingPipe etc. Personally I would use NLTK or just write my own topic classifier as I think the big java libraries are unwieldily and user-unfriendly. Easier to have something lightweight.

You probably need to train your classifier with labeled data- if you have labeled data, good, if not, time to start labeling. The most important thing as always is data quality- how representative is your training data of the data you expect to see? How good are your features (n-grams, word n-grams, etc)?

2) For prediction, there are any number of statistical models you could use. Personally I'd go for a stochastic model, but that's just cause I spent too much time studying those.

Radi · Accepted Answer · 2012-01-05 16:10:02Z

-1

you can use GATE (General Architecture for Text Engineering) a open source infrastructure for text processing , you can use its IDE (Gate Developer) to build text processing components using a comprehensive set of other plugins or if you need to use it on your code you need to embed its jar file into your project . this tool has a language called JAPE (Java Annotation Patterns Engine) it allows you to recognise regular expressions in annotations on documents (you can annotate the document using gate analyzers or you can develop your own analyzers).

answered Jan 5, 2012 at 16:10

Radi

6,66218 gold badges65 silver badges92 bronze badges

Collectives™ on Stack Overflow

Java library for Pattern mining on time based data

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related