Tensorflow implementation of word2vec

I tried out your proposed way to generate batches - having a loop and using the whole skip-window. The results are:

1. Faster generation of batches

For a batch size of 128 and a skip window of 5

• generating batches by looping over the data one by one takes 0.73s per 10,000 batches
• generating batches with the tutorial code and num_skips=2 takes 3.59s per 10,000 batches

2. Higher danger of overfitting

Keeping the rest of the tutorial code as it is, I trained the model with both ways and logged the average loss every 2000 steps:

This pattern occurred repeatedly. It shows that using 10 samples per word instead of 2 can cause overfitting.

Here is the code that I used for generating the batches. It replaces the tutorial's generate_batch function.

data_index = 0

def generate_batch(batch_size, skip_window):
    global data_index
    batch = np.ndarray(shape=(batch_size), dtype=np.int32)  # Row
    labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32)  # Column

    # For each word in the data, add the context to the batch and the word to the labels
    batch_index = 0
    while batch_index < batch_size:
        context = data[get_context_indices(data_index, skip_window)]

        # Add the context to the remaining batch space
        remaining_space = min(batch_size - batch_index, len(context))
        batch[batch_index:batch_index + remaining_space] = context[0:remaining_space]
        labels[batch_index:batch_index + remaining_space] = data[data_index]

        # Update the data_index and the batch_index
        batch_index += remaining_space
        data_index = (data_index + 1) % len(data)

    return batch, labels

Edit: The get_context_indices is a simple function, which returns the index slice in the skip_window around data_index. See the slice() documentation for more info.

There is a parameter named num_skips which denotes the number of (input, output) pairs generated from the single window: [skip_window target skip_window]. So num_skips restrict the number of context words we would use as output words. And that is why the generate_batch function assert num_skips <= 2*skip_window. The code just randomly pick up num_skip context words to construct training pairs with target. But I don't know how num_skips affects the performance.