No classification improvements with using set_session() for Keras - session

Question

I want to use the allow_growth configuration with Keras and tensorflow backend.
As suggested f. e. in use tensorflow.GPUOptions within Keras when using tensorflow backend I implemented my code as follows:
import keras.backend.tensorflow_backend as K
import tensorflow as tf
...
gpu_options = tf.GPUOptions(allow_growth=True)
config = tf.ConfigProto(gpu_options=gpu_options)
sess = tf.Session(config = config)
K.set_session(sess)
model = Sequential()
model.add(...)
...
model.compile(...)
model.fit(...)
With nvidia-smi I can see, that dynamic allocation works. But with this adaption applied, I'm not able to achieve any improvements in the classification accuracy, which means that the validation error remains the same over multiple epochs.
Hope you can help me.
EDIT:
Output of the first 10 epochs:
Epoch 1/30
307/307 [==============================] - 45s 147ms/step - loss: 3.7766 - acc: 0.0558 - val_loss: 3.7457 - val_acc: 0.0420
Epoch 2/30
307/307 [==============================] - 43s 140ms/step - loss: 3.7372 - acc: 0.0566 - val_loss: 3.7309 - val_acc: 0.0420
Epoch 3/30
307/307 [==============================] - 44s 143ms/step - loss: 3.7222 - acc: 0.0566 - val_loss: 3.7170 - val_acc: 0.0420
Epoch 4/30
307/307 [==============================] - 45s 146ms/step - loss: 3.7079 - acc: 0.0566 - val_loss: 3.7037 - val_acc: 0.0420
Epoch 5/30
307/307 [==============================] - 46s 150ms/step - loss: 3.6944 - acc: 0.0566 - val_loss: 3.6911 - val_acc: 0.0420
Epoch 6/30
307/307 [==============================] - 45s 147ms/step - loss: 3.6815 - acc: 0.0565 - val_loss: 3.6793 - val_acc: 0.0420
Epoch 7/30
307/307 [==============================] - 44s 144ms/step - loss: 3.6693 - acc: 0.0566 - val_loss: 3.6680 - val_acc: 0.0420
Epoch 8/30
307/307 [==============================] - 44s 143ms/step - loss: 3.6577 - acc: 0.0566 - val_loss: 3.6573 - val_acc: 0.0420
Epoch 9/30
307/307 [==============================] - 45s 147ms/step - loss: 3.6467 - acc: 0.0566 - val_loss: 3.6473 - val_acc: 0.0420
Epoch 10/30
307/307 [==============================] - 44s 143ms/step - loss: 3.6363 - acc: 0.0565 - val_loss: 3.6378 - val_acc: 0.0420