Why does the loss of Word2Vec model trained by gensim at first increase for a few epochs and then decrease? - gensim

Question

Actually this is a sequel to:
post
I am training a Word2Vec model using gensim, with parameters hs=1, sg=0 and negative=0. Less training time is required after the code is modified, but something seems to go wrong with the loss, it would at first increase then decrease, I don't know what happened.
The code is as follows:
from gensim.models.keyedvectors import KeyedVectors
from gensim.models import word2vec
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',
level=logging.INFO)
sentences = word2vec.Text8Corpus("text8") # loading the corpus
from gensim.models.callbacks import CallbackAny2Vec
loss_list = []
class Callback(CallbackAny2Vec):
def __init__(self):
self.epoch = 0
def on_epoch_end(self, model):
loss = model.get_latest_training_loss()
loss_list.append(loss)
print('Loss after epoch {}:{}'.format(self.epoch, loss))
model.running_training_loss = 0.0
self.epoch = self.epoch + 1
from gensim.models import KeyedVectors,word2vec,Word2Vec
import time
start_time = time.time()
model = word2vec.Word2Vec(sentences, hs=1, sg=0, negative=0, compute_loss=True, epochs=30, callbacks=[Callback()])
end_time = time.time()
print('Running time: %s seconds' % (end_time - start_time))
The codes are actually written in jupyter, as can be seen in the screenshot:
and the output is like this:
More details about the output:
Loss after epoch 0:39370848.0
Loss after epoch 1:43579636.0
Loss after epoch 2:45213772.0
Loss after epoch 3:46132356.0
Loss after epoch 4:46788412.0
Loss after epoch 5:47218508.0
Loss after epoch 6:47553520.0
Loss after epoch 7:47793332.0
Loss after epoch 8:47995616.0
Loss after epoch 9:48134664.0
Loss after epoch 10:48224960.0
Loss after epoch 11:48326640.0
Loss after epoch 12:48371072.0
Loss after epoch 13:48405980.0
Loss after epoch 14:48437804.0
Loss after epoch 15:48417612.0
Loss after epoch 16:48415112.0
Loss after epoch 17:48396260.0
Loss after epoch 18:48349064.0
Loss after epoch 19:48301088.0
Loss after epoch 20:48247328.0
Loss after epoch 21:48167340.0
Loss after epoch 22:48053500.0
Loss after epoch 23:47937300.0
Loss after epoch 24:47810964.0
Loss after epoch 25:47669088.0
Loss after epoch 26:47500524.0
Loss after epoch 27:47300488.0
Loss after epoch 28:47044920.0
Loss after epoch 29:46747080.0
Running time: 259.9046218395233 seconds

Answer 1

I wouldn't expect that pattern of rising-then-falling loss; I would think that usual SGD optimization would usually ahieve falling full-epoch loss from the very beginning.
However, if the end result vectors are still performing well, I wouldn't worry too much about surprises in secondary progress indicators, like that loss tally, for a number of reasons:
as noted in my previous answer (& further discussed in the Gensim project open issue #2617), Gensim's external loss-reporting has a number of known bugs & inconsistencies. Any oddness in observed loss-reporting may be a side-effect of those issues, without necessarily indicating any problem with the actual training updates.
it appears you're finishing 30 training epochs in 260 seconds – each full training pass in under 9 seconds. That suggests your training data is pretty tiny – perhaps too small to be a good example of word2vec capabilities, or too small for the default 100-dimensional vectors. That smallness, or other peculiarities in the training data, might contribute to atypical loss trends, or exercising some of the other weaknesses of the current loss-tallying code. If the same hard-to-explain pattern occurs with a more typical training corpus – such as one 100x larger – then it'd be more interesting to do a deep-dive investigation to understand what's happening. But unexpected results on tiny/unusual/atypical training runs might just be because such runs are far from where the usual intuitions apply, and getting to the bottom of their causes less productive than acquiring enough data to run the algorithm in a more typical/reliable way.