Tokenizer for indexing source code

Question

Lets say you want to create a search engine for a site like github or stackoverflow, where the majority of the textual content is actually source code. What would be a good Lucene tokenizer for dealing with documents of this type?

Generally, source code uses whitespace and punctuation to separate things, so I would think simply using StandardAnalyzer would probably work pretty well, or possibly a custom analyzer like StandardAnalyzer but without the LowercaseFilter, depending on your needs. Is there some particular feature you are looking for, as far as how you would like code to be tokenized? — femtoRgon
– femtoRgon, Commented Nov 23, 2013 at 18:51

Arun · Accepted Answer · 2013-11-24 00:29:14Z

1

This is what you are looking for : http://www.opensourceconnections.com/2013/02/18/indexing-stackoverflow-in-solr/ . This covers all the steps and more.

answered Nov 24, 2013 at 0:29

Arun

1,80711 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Tokenizer for indexing source code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related