Flex lexer output modification

Question

How can I use flex lexer in C++ and modify a token's yytext value? Lets say, I have a rule like this:

"/*"    {
        char c;
        while(true)
            {
            c = yyinput();
            if(c == '\n')
                ++mylineno;

            if (c==EOF){
                yyerror( "EOF occured while processing comment" );
                break;
            }
            else if(c == '*')
                {
                if((c = yyinput()) == '/'){
                    return(tokens::COMMENT);}
                else
                    unput(c);
                }
            }
        }

And I want to get token tokens::COMMENT with value of comment between /* and */. (The bove solution gives "/*" as the value.

Additional, very important is tracking the line number, so I'm looking for solution supporting it.

EDIT Of course I can modify the yytext and yyleng values (like yytext+=1; yyleng-=1, but still I cannot solve the above problem)

Do you have a parser taking tokens from here. or it just lexer only? You can solve this easily in the parser. — user995502
– user995502, Commented Apr 5, 2013 at 14:30
I really would like to solve it in the lexer - is it possible? — Wojciech Danilo
– Wojciech Danilo, Commented Apr 5, 2013 at 14:41
Check out this existing answer: stackoverflow.com/a/2130124/1003855 — Josh
– Josh, Commented Apr 5, 2013 at 14:45
@danilo2 ok. Then how do you handle strings? do you have some pool to store them in? or show me how you recognize a string literal. — user995502
– user995502, Commented Apr 5, 2013 at 14:50

Community · Accepted Answer · 2017-05-23 12:12:53Z

1

I still think start conditions are the right answer.

%x C_COMMENT
char *str = NULL;
void addToString(char *data)
{
    if(!str)
    { 
        str = strdup(data);
    }
    else
    {
        /* handle string concatenation */
    }
}

"/*"                       { BEGIN(C_COMMENT); }
<C_COMMENT>([^*\n\r]|(\*+([^*/\n\r])))*    { addToString(yytext); }
<C_COMMENT>[\n\r]          { /* handle tracking, add to string if desired */ }
<C_COMMENT>"*/"            { BEGIN(INITIAL); }

I used the following as references:
http://ostermiller.org/findcomment.html
https://stackoverflow.com/a/2130124/1003855

You should be able to use a similar regular expression to handle strings.

edited May 23, 2017 at 12:12

CommunityBot

11 silver badge

answered Apr 5, 2013 at 14:58

Josh

1,8241 gold badge17 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Wojciech Danilo Over a year ago

Your solutin do not track the line number. So if you want to get place information, you cannot do it.

Josh Over a year ago

If that's a requirement, then I will modify my code to support it.

Wojciech Danilo Over a year ago

Yes it is (I updated the question), so if you know the answer, I'll be thankful for it.

Collectives™ on Stack Overflow

Flex lexer output modification

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related