What is a token string in c++?

Question

The #define creates a macro, which is the association of an identifier or parameterized identifier with a token string. After the macro is defined, the compiler can substitute the token string for each occurrence of the identifier in the source file.

https://learn.microsoft.com/en-ca/cpp/preprocessor/hash-define-directive-c-cpp?view=vs-2017

Surprisingly, the question has not been directly asked, rather asking about tokenization, tokenizer, tokening etc. Even searching on DuckDuckGo, the closest question was on quora asking,

What is a string token in c++?

And it is not obvious to me whether string token and token string would be synonymous. So just to be clear:

What is a token string in c++?

The compiler deoes no such thing. That's the job of the preprocessor. If you want to find out what the compiler gets to see after the preprocessro run gcc -E foo.c — Swordfish
– Swordfish, Commented Dec 4, 2018 at 19:04
The answer to the Quora link was written by someone who appears to have wanted to answer a different question. — user4581301
– user4581301, Commented Dec 4, 2018 at 19:06
And keep an eye out for the Tolkien strings. They're the short ones with hairy feet. — user4581301
– user4581301, Commented Dec 4, 2018 at 19:11
One (st)ring to rule them all, one (st)ring to find them, One (st)ring to bring them all and in the darkness bind them. — Swordfish
– Swordfish, Commented Dec 4, 2018 at 19:13

NathanOliver · Accepted Answer · 2018-12-04 19:20:35Z

4

In this case the token string is the macro body. In

#defined MAKE_MY_FUNC(x) void x(int bar)

The void x(int foo) part would be considered the token string and when you use MAKE_MY_FUNC like

MAKE_MY_FUNC(foo){ std::cout << bar; }

then the token string would be subsituted in and the code would be transformed into

void foo(int foo){ std::cout << bar; }

Your article gives you what they call the token-string in the second paragraph

The token-string argument consists of a series of tokens, such as keywords, constants, or complete statements. One or more white-space characters must separate token-string from identifier. This white space is not considered part of the substituted text, nor is any white space that follows the last token of the text.

edited Dec 4, 2018 at 19:20

answered Dec 4, 2018 at 19:06

NathanOliver

183k29 gold badges323 silver badges439 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

What is a token string in c++?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related