Java How to Convert Token to String?

Question

Cant seem to figure out how to convert a Token (StringTokenizer) into a String.

I have a String[] of keywords, and I read in multiple lines of text from a text file. StringTokenizer is used to chop the sentence up,,, at which point I need to pass each Token (the String representation of the Token) to a function for word analysis.

Is there a simple way to extract the string value from the token?

Any help appreciated guys....

Great help guys thanks again!!

I don't understand, your tokens are already String arent' they? — reef
– reef, Commented Mar 7, 2011 at 16:36

Shawn · Accepted Answer · 2011-03-07 16:39:48Z

2

Use the method nextToken() to return the token as a string

StringTokenizer st = new StringTokenizer("this is a test");
         while (st.hasMoreTokens()) {
             wordAnalysis(st.nextToken());//sends string to your function
         }

edited Mar 7, 2011 at 16:39

answered Mar 7, 2011 at 16:34

Shawn

7,3636 gold badges37 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Donal.Lynch.Msc Over a year ago

in the while loop above, I call a function which expects a String. How do I convert the Token to a String do you know?

reef Over a year ago

The nextToken method returns a String: see download.oracle.com/javase/1.5.0/docs/api/java/util/….

Edwin Buck · Accepted Answer · 2011-03-07 16:37:29Z

1

nextToken() called on a StringTokenizer, already returns a String. You don't need to convert that String to a String.

A Token is just a small (string) portion of a larger String. Tokens are used in various forms of processing, and a word (the word token) is needed to differentiate the category of items from the specific items.

For example, every word, period, and comma in this string is a token.

Would lend itself to the following tokens (all of which are strings):

For 
example
, 
every 
word
,
period
,
and 
comma 
in 
this 
string 
is 
a 
token
.

answered Mar 7, 2011 at 16:37

Edwin Buck

71.2k7 gold badges103 silver badges145 bronze badges

Comments

dmcnelis · Accepted Answer · 2011-03-07 16:38:39Z

1

for(int i = 0; i<stringArray.length; i++) {
    StringTokenizer st = new StringTokenizer(stringArray[i]);
    while(st.hasMoreTokens()) {
       callMyProcessingMethod(st.nextToken());
    }
}

answered Mar 7, 2011 at 16:38

dmcnelis

2,9231 gold badge21 silver badges28 bronze badges

Comments

Jordan Owens · Accepted Answer · 2011-03-07 16:39:28Z

1

StringTokenizer is now a legacy class in Java. As of Java 1.4 it is recommended to use the split() method on String which returns a String[].

http://download.oracle.com/javase/1.4.2/docs/api/java/lang/String.html#split(java.lang.String)

answered Mar 7, 2011 at 16:39

Jordan Owens

7424 silver badges10 bronze badges

2 Comments

Edwin Buck Over a year ago

Odd, there's nothing in the API to indicate that's it's deprecated (or legacy). Also split doesn't do the same thing. Token boundaries are easily identified by "next char" sequences, while REGEX's require a full pattern to match.

Jordan Owens Over a year ago

Sorry, it's actually the API doc for StringTokenizer that mentions that it is deprecated. download.oracle.com/javase/6/docs/api/java/util/…

Collectives™ on Stack Overflow

Java How to Convert Token to String?

4 Answers 4

2 Comments

Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related