For early stopping we mean stop training the model as soon as the performance on the validation set decreases, meaning when the error goes up again.