Python machine learning, feature selection

Question

I am working on a classification task related to written text and I wonder how important it is to perform some kind of "feature selection" procedure in order to improve the classification results.

I am using a number of features (around 40) related to the subject, but I am not sure if all the features are really relevant or not and in which combinations. I am experementing with SVM (scikits) and LDAC (mlpy).

If a have a mix of relevant and irrelevant features, I assume I will get poor classification results. Should I perform a "feature selection procedure" before classification?

Scikits has an RFE procedure that is tree-based that is able to rank the features. Is it meaningful to rank the features with a tree-based RFE to choose the most important features and to perform the actual classification with SVM (non linear) or LDAC? Or should I implement some kind of wrapper method using the same classifier to rank the features (trying to classify with different groups of features would be very time consuming)?

Look into using evolutionary algorithms to improve your results by selection. — Joel Cornett
– Joel Cornett, Commented Mar 11, 2012 at 20:10
Well, you can certainly try doing feature selection; this is its intended use case. Without more details of your problem, this is very hard to answer, though. You could also experiment with different feature sets manually. — Fred Foo
– Fred Foo, Commented Mar 11, 2012 at 20:55

ogrisel · Accepted Answer · 2012-06-24 06:58:09Z

1

Just try an see if it improves the classification score as measured with cross validation. Also before trying RFE, I would try less CPU intensive schemes such as univariate chi2 feature selection.

edited Jun 24, 2012 at 6:58

answered Mar 12, 2012 at 8:16

ogrisel

40.3k14 gold badges120 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Dan Stowell · Accepted Answer · 2012-10-31 16:02:52Z

0

Having 40 features is not too bad. Some machine-learning is impeded by irrelevant features, but many things are quite robust to them (eg naive Bayes, SVM, decision trees). You probably don't need to do feature selection unless you decide to add many more features in.

It's not a bad idea to throw away useless features, but don't waste your own mental time on trying that out unless you have a particular motivation to.

answered Oct 31, 2012 at 16:02

Dan Stowell

4,7782 gold badges22 silver badges31 bronze badges

Collectives™ on Stack Overflow

Python machine learning, feature selection

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related