How to train Word Embeddings in Tensorflow

Question

So I have changed the model described here to perform multi-class text classification instead of a binary class classification. http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/

My model is overfitting even after applying L2 regularization so I am wanting to use a pre-trained word2vec model. But I am extremely new to Tensorflow & deep learning & am not sure where to start.

Code: https://github.com/dennybritz/cnn-text-classification-tf/blob/master/text_cnn.py#L27

Here is the relevant code that I want to change to use the Google pre-trained word2vec model:

        # Embedding layer
    with tf.device('/cpu:0'), tf.name_scope("embedding"):
        W = tf.Variable(
            tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
        self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

It will be very helpful if someone can point me to how can i incorporate this in the code. I looked at the embedding_lookup doc but that doesn't seem to have the information i am looking for.

Community · Accepted Answer · 2017-05-23 12:32:02Z

1

Refere to these links:

edited May 23, 2017 at 12:32

CommunityBot

11 silver badge

answered Oct 8, 2016 at 20:43

Jules Gagnon-Marchand

3,8011 gold badge24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rookie Over a year ago

I have already seen the 1st post and made the following change with no luck:

model = word2vec.Word2Vec.load_word2vec_format('./word2vec/pre-trained/GoogleNews-vectors-negative300.bin', binary=True)                  embedding = model.syn0               vocab_size    = embedding.shape[0]             embedding_dim = embedding.shape[1]

. But the above gives me the following message:

UserWarning: Converting sparse IndexedSlices to a dense Tensor with 900000000 elements. This may consume a large amount of memory

and the training seems to be hung at this step

Jingyang Li · Accepted Answer · 2018-07-13 05:55:13Z

0

  W = tf.Variable(
        tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
        name="W")

This line created a word dic randomly to represent the word in your own vocab, So you just need load the pre-trained word2vec dic to replace your 'W', it will works

answered Jul 13, 2018 at 5:55

Jingyang Li

133 bronze badges

Collectives™ on Stack Overflow

How to train Word Embeddings in Tensorflow

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related