Training did not improve the model performance on validation data - validation

Question

I trying to train my Resnet-50 network on a database which collects 5968 images for training and 1492 for validation (746 classes with 8 images/class for training and 2 images/class for validation). I used ImageDataGenerator flow_from_directory method to get labels from folders
My problem is that during the training, the accuracy of the training was increasing and the loss was decreasing which is good. In fact, the validation accuracy was very low (around 0.003) and there is no improvement. Also the validation loss is very high and still oscillating into very high values!!
Here is my code
import numpy as np
from keras_preprocessing.image import ImageDataGenerator
from keras.utils.vis_utils import plot_model
import resnet
import json
from keras.callbacks import ModelCheckpoint, EarlyStopping
import keras
import pydot as pyd
keras.utils.vis_utils.pydot = pyd
data_path_l =".\\TRAIN\\left_750\\"
test_data_path_l =".\\TEST\\left_750\\"
num_classes=746
train_images=5968
val_images=1492
batch_size=32
epochs=500
img_channels=3
img_rows=224
img_cols=224
input_imgen = ImageDataGenerator(shear_range = 0.2,
zoom_range = 0.2,
rotation_range=5.,
horizontal_flip = True)
valid_imgen = ImageDataGenerator()
train_it = input_imgen.flow_from_directory(directory=data_path_l,target_size=(img_rows,img_cols),
color_mode="rgb",
batch_size=batch_size,
class_mode="categorical",
shuffle=False,
)
valid_it = valid_imgen.flow_from_directory(directory=test_data_path_l,target_size=(img_rows,img_cols),
color_mode="rgb",
batch_size=batch_size,
class_mode="categorical",
shuffle=False,
)
model = resnet.ResnetBuilder.build_resnet_50((img_channels, img_rows, img_cols), num_classes)
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
filepath=".\\conv2D_models\\model-{epoch:02d}-{loss:.4f}.hdf5"
mc = ModelCheckpoint(filepath, save_weights_only=False, verbose=1,
monitor='loss', mode='min')
history=model.fit_generator(train_it,
steps_per_epoch= train_images // batch_size,
validation_data = valid_it,
validation_steps = val_images // batch_size,
epochs = epochs,callbacks=[mc],
shuffle=False)
model.save('resnet2D_1sample.h5')
and here is a part of training epochs:
Epoch 00059: saving model to .\conv2D_models\model-59-3.6342.hdf5
Epoch 60/500
186/186 [==============================] - 262s 1s/step - loss: 3.6074 - acc: 0.4078 - val_loss: 12.1131 - val_acc: 0.0034
Epoch 00060: saving model to .\conv2D_models\model-60-3.6084.hdf5
Epoch 61/500
186/186 [==============================] - 276s 1s/step - loss: 3.5681 - acc: 0.4236 - val_loss: 12.0455 - val_acc: 0.0034
Epoch 00061: saving model to .\conv2D_models\model-61-3.5683.hdf5
Epoch 62/500
186/186 [==============================] - 100s 536ms/step - loss: 3.4684 - acc: 0.4415 - val_loss: 10.2444 - val_acc: 0.0068
Epoch 00062: saving model to .\conv2D_models\model-62-3.4674.hdf5
Epoch 63/500
186/186 [==============================] - 96s 516ms/step - loss: 3.4523 - acc: 0.4414 - val_loss: 11.6459 - val_acc: 0.0062
Epoch 00063: saving model to .\conv2D_models\model-63-3.4530.hdf5
Epoch 64/500
186/186 [==============================] - 96s 516ms/step - loss: 3.3837 - acc: 0.4782 - val_loss: 12.3293 - val_acc: 0.0062
Epoch 00064: saving model to .\conv2D_models\model-64-3.3847.hdf5
Epoch 65/500
186/186 [==============================] - 96s 515ms/step - loss: 3.2915 - acc: 0.5045 - val_loss: 12.8812 - val_acc: 0.0034
Epoch 00065: saving model to .\conv2D_models\model-65-3.2928.hdf5
Epoch 66/500
186/186 [==============================] - 96s 517ms/step - loss: 3.2506 - acc: 0.5129 - val_loss: 13.2886 - val_acc: 0.0034
Epoch 00066: saving model to .\conv2D_models\model-66-3.2527.hdf5
Epoch 67/500
186/186 [==============================] - 96s 515ms/step - loss: 3.2511 - acc: 0.5123 - val_loss: 14.4090 - val_acc: 0.0034
Epoch 00067: saving model to .\conv2D_models\model-67-3.2530.hdf5
Epoch 68/500
186/186 [==============================] - 97s 519ms/step - loss: 3.2632 - acc: 0.5163 - val_loss: 16.2364 - val_acc: 0.0027
Epoch 00068: saving model to .\conv2D_models\model-68-3.2650.hdf5
Epoch 69/500
186/186 [==============================] - 96s 517ms/step - loss: 3.1477 - acc: 0.5585 - val_loss: 16.2729 - val_acc: 0.0021
Epoch 00069: saving model to .\conv2D_models\tmodel-69-3.1487.hdf5
Epoch 70/500
186/186 [==============================] - 96s 516ms/step - loss: 2.9347 - acc: 0.6099 - val_loss: 16.7732 - val_acc: 0.0014
Epoch 00070: saving model to .\conv2D_models\model-70-2.9369.hdf5
Epoch 71/500
186/186 [==============================] - 96s 515ms/step - loss: 2.7118 - acc: 0.6715 - val_loss: 15.4640 - val_acc: 0.0075
Epoch 00071: saving model to .\conv2D_models\model-71-2.7134.hdf5
Epoch 72/500
186/186 [==============================] - 96s 517ms/step - loss: 2.6145 - acc: 0.6835 - val_loss: 16.2367 - val_acc: 0.0055
Epoch 00072: saving model to .\conv2D_models\model-72-2.6159.hdf5
Epoch 73/500
186/186 [==============================] - 96s 517ms/step - loss: 2.5492 - acc: 0.6816 - val_loss: 16.8155 - val_acc: 0.0000e+00
Epoch 00073: saving model to .\conv2D_models\model-73-2.5503.hdf5
Epoch 74/500
186/186 [==============================] - 96s 516ms/step - loss: 2.5743 - acc: 0.6786 - val_loss: 14.1867 - val_acc: 0.0021
Epoch 00074: saving model to .\conv2D_models\model-74-2.5759.hdf5
Epoch 75/500
186/186 [==============================] - 96s 516ms/step - loss: 2.5295 - acc: 0.6962 - val_loss: 12.3790 - val_acc: 0.0055
could anyone suggest to me some potential raisons that leads to this strange training behavior because it's been blocking me for a week..