Custom callback function about ultralytics HOT 7 OPEN

@YEONCHEOL-HA hello! Great questions regarding the use of callbacks for implementing early stopping in YOLOv8. Let's address each one:

Q1: Yes, using a custom callback is the appropriate approach for implementing early stopping based on validation loss criteria in YOLOv8, as there isn't a built-in argument for this specific functionality.

Q2: The code snippet you provided for adding a callback is correct. You can use it to append your custom callback function to the desired event.

Q3: For your requirement of checking the validation loss after each epoch, you should use the Validator callbacks. Specifically, the on_val_epoch_end callback would be suitable as it triggers after each validation epoch completes.

Q4: Your early stopping code is almost there, but it needs a slight modification to maintain the best_loss, wait, and patience variables outside the callback function to preserve their values across different epochs. Here’s a revised version:

model = YOLO("yolov8n-cls.pt")

best_loss = float('inf')
wait = 0
patience = 10

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val_loss')
    if val_loss is None:
        return
    improvement = (best_loss - val_loss) / best_loss * 100
    if improvement < 2:
        wait += 1
    else:
        best_loss = val_loss
        wait = 0
    if wait >= patience:
        print("No improvement, stopping early.")
        model.stop_training = True
    print(f"Epoch {epoch + 1}: Improvement {improvement:.2f}%, Best Loss {best_loss:.4f}")

model.add_callback('on_val_epoch_end', early_stopping_callback)

This setup should correctly implement early stopping based on your criteria. If you have any more questions or need further assistance, feel free to ask. Happy coding! 🚀

from ultralytics.

thanks for your answer.

and then in following code, should i need to input patience?

Early stopping option isn't performed

Are there any other arguments I need to declare in my code for custom early stopping options?

and then what is the correct name of validation loss(val_loss or val/loss)?

model.train( data='/content/drive/MyDrive/cls/', epochs=300, imgsz=640, batch=-1, workers=4, rect=True, multi_scale=True, verbose=True, plots=True, )

from ultralytics.

Hello @YEONCHEOL-HA,

Thank you for your follow-up questions!

1. Patience Parameter: Yes, you should define the patience variable outside the callback function to maintain its state across epochs. This variable is crucial for determining how many epochs without improvement should trigger early stopping.

2. Additional Arguments: The code you've provided for training looks good for general training purposes. For integrating the early stopping, ensure that your callback is correctly set up as discussed previously. No additional arguments in the model.train() function are necessary for early stopping.

3. Validation Loss Name: In the context of callbacks, the key for validation loss typically depends on how it's logged within the YOLOv8 framework. It's often labeled as val_loss, but this can vary based on the specific implementation or modifications in the logging system. You might need to print out the logs dictionary in your callback to confirm the exact key used for validation loss.

Here's a small tweak to ensure you're using the right key:

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val_loss')  # Adjust this key based on your logs output
    if val_loss is None:
        print("Validation loss not found in logs.")
        return
    # Early stopping logic follows

If you have any more questions or need further clarification, feel free to ask. Happy coding! 🚀

from ultralytics.

Q1. what is the metric for early stopping option in yolov8 classfication model?

Q2. what is the metric for early stopping option in yolov8 Detection model?

from ultralytics.

Hello @YEONCHEOL-HA,

Thank you for your questions! Let's address them one by one:

Q1. What is the metric for early stopping option in YOLOv8 classification model?

For the YOLOv8 classification model, early stopping is typically based on the validation loss (val_loss). This metric helps in determining whether the model's performance on the validation set is improving or not. If the validation loss does not decrease by a specified threshold over a certain number of epochs (patience), early stopping can be triggered to prevent overfitting and save computational resources.

Q2. What is the metric for early stopping option in YOLOv8 Detection model?

Similarly, for the YOLOv8 detection model, early stopping is usually based on the validation loss as well. In the context of object detection, this could be the loss associated with bounding box regression, classification, or a combination of both. Monitoring the validation loss ensures that the model is not just memorizing the training data but is also generalizing well to unseen data.

To implement custom early stopping in YOLOv8, you can use a callback function. Here’s a brief example of how you might set up an early stopping callback for a classification model:

model = YOLO("yolov8n-cls.pt")

best_loss = float('inf')
wait = 0
patience = 10

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val_loss')  # Ensure this key matches your logs
    if val_loss is None:
        print("Validation loss not found in logs.")
        return
    improvement = (best_loss - val_loss) / best_loss * 100
    if improvement < 2:
        wait += 1
    else:
        best_loss = val_loss
        wait = 0
    if wait >= patience:
        print("No improvement, stopping early.")
        model.stop_training = True
    print(f"Epoch {epoch + 1}: Improvement {improvement:.2f}%, Best Loss {best_loss:.4f}")

model.add_callback('on_val_epoch_end', early_stopping_callback)

Make sure to adjust the key for validation loss (val_loss) based on your specific logging output.

If you encounter any issues or need further assistance, please ensure you are using the latest versions of torch and ultralytics. If the problem persists, providing a minimum reproducible code example would be very helpful for us to investigate further. You can find more details on creating a reproducible example here.

Feel free to reach out with any more questions. Happy coding! 😊

from ultralytics.

thanks for your answer!

`import torch
from ultralytics import YOLO
from ultralytics.engine.trainer import BaseTrainer
from ultralytics.utils import LOGGER

model = YOLO("yolov8n-cls.pt")

best_loss = float('inf')
wait = 0
patience = 5

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val/loss')  # Ensure this key matches your logs
    if val_loss is None:
        print("Validation loss not found in logs.")
        return
    improvement = (best_loss - val_loss) / best_loss * 100
    if improvement < 2:
        wait += 1
    else:
        best_loss = val_loss
        wait = 0
    if wait >= patience:
        print("No improvement, stopping early.")
        model.stop_training = True
    print(f"Epoch {epoch + 1}: Improvement {improvement:.2f}%, Best Loss {best_loss:.4f}")

model.add_callback('on_val_epoch_end', early_stopping_callback)

model.train(data='/content/drive/MyDrive/cls/', epochs=100, patience=5)

following image is result.csv file.
According to the earlystopping code, the early stopping option must be performed in epoch 1. however early stopping option is perfomed in epoch 11.

Q2. When training resumes(resume = True), warmup eapoch and learning late is applied in training? think that the decrease in loss values ​​immediately after resuming training(during 2 ~ 3 epoch) is related to warm-up.
`

from ultralytics.

Hello @YEONCHEOL-HA,

Thank you for sharing your code and the detailed explanation! Let's address your questions and concerns.

Early Stopping Issue

It looks like you've implemented the early stopping callback correctly. However, the discrepancy in the early stopping behavior might be due to how the best_loss and wait variables are being managed across epochs. Let's ensure that these variables are correctly initialized and updated.

Here's a refined version of your code:

import torch
from ultralytics import YOLO
from ultralytics.engine.trainer import BaseTrainer
from ultralytics.utils import LOGGER

model = YOLO("yolov8n-cls.pt")

best_loss = float('inf')
wait = 0
patience = 5

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val/loss')  # Ensure this key matches your logs
    if val_loss is None:
        print("Validation loss not found in logs.")
        return
    improvement = (best_loss - val_loss) / best_loss * 100
    if improvement < 2:
        wait += 1
    else:
        best_loss = val_loss
        wait = 0
    if wait >= patience:
        print("No improvement, stopping early.")
        model.stop_training = True
    print(f"Epoch {epoch + 1}: Improvement {improvement:.2f}%, Best Loss {best_loss:.4f}")

model.add_callback('on_val_epoch_end', early_stopping_callback)

model.train(data='/content/drive/MyDrive/cls/', epochs=100)

Debugging Early Stopping

To debug why early stopping is not being triggered as expected, you can add some print statements to monitor the values of best_loss, val_loss, and wait:

def early_stopping_callback(epoch, logs):
    global best_loss, wait
    val_loss = logs.get('val/loss')  # Ensure this key matches your logs
    if val_loss is None:
        print("Validation loss not found in logs.")
        return
    improvement = (best_loss - val_loss) / best_loss * 100
    print(f"Epoch {epoch + 1}: val_loss={val_loss}, best_loss={best_loss}, improvement={improvement:.2f}%, wait={wait}")
    if improvement < 2:
        wait += 1
    else:
        best_loss = val_loss
        wait = 0
    if wait >= patience:
        print("No improvement, stopping early.")
        model.stop_training = True
    print(f"Epoch {epoch + 1}: Improvement {improvement:.2f}%, Best Loss {best_loss:.4f}")

Warmup and Learning Rate on Resume

When resuming training (resume=True), the warmup epochs and learning rate schedule are typically applied as they were during the initial training. This can indeed cause a temporary decrease in loss values immediately after resuming training.

To verify this, you can inspect the learning rate schedule and warmup settings in your training configuration. If you want to modify the warmup behavior upon resuming, you might need to adjust the training configuration accordingly.

Minimum Reproducible Example

If the issue persists, could you please provide a minimum reproducible example? This will help us investigate the problem more effectively. You can find more details on creating a reproducible example here.

Feel free to reach out with any more questions or updates. We're here to help! 😊

from ultralytics.