Your input ran out of data; interrupting training

Question

Your input ran out of data; interrupting training

A data shortage error pops up. About 4-5 days ago, this error did not occur on the same code. Code for creating the model:

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(27, 48, 1)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('sigmoid'))
model.compile(loss="categorical_crossentropy", optimizer="adam",
              metrics=["accuracy"])

Training code

datagen = ImageDataGenerator()
dirTrain = "/content/GeneratedI/train"
train_data = datagen.flow_from_directory(dirTrain, target_size=(27, 48), batch_size=100,
                                         class_mode="categorical", color_mode="grayscale")
dirVal = "/content/GeneratedI/val"
validation_data = datagen.flow_from_directory(dirVal, target_size=(27, 48), batch_size=100,
                                              class_mode="categorical", color_mode="grayscale")
print("Training the network...")
t_start = time.time()
history = model.fit_generator(train_data,
                              steps_per_epoch=60000 // 10,
                              epochs=1,
                              validation_data=validation_data,
                              validation_steps=10000 // 10)
print(time.time() - t_start)

The actual images are exactly there, 60k for training and 10k for validation. Even the conclusion confirms this:

Found 60000 images belonging to 10 classes.
Found 10000 images belonging to 10 classes.

Using Google Colaboratory

0

python tensorflow keras

Author: Даниил Кустов, 2020-05-19

Source

1 answers

score 1 · Accepted Answer

history = model.fit_generator(train_data,
                              steps_per_epoch=60000 // 10,  # <-- 1
                              epochs=1,
                              validation_data=validation_data,
                              validation_steps=10000 // 10)  # <-- 2

In the first and second cases, you need to divide by the size of the butch. You have it 100, judging by the generator:

train_data = datagen.flow_from_directory(dirTrain, 
                                         target_size=(27, 48), 
                                         batch_size=100,  # <-- 3
                                         class_mode="categorical", 
                                         color_mode="grayscale")