data lable wrong when train yoloworld about ultralytics HOT 19 OPEN

👋 Hello @ccl-private, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

from ultralytics.

the label in the train batch：

the label in text:

from ultralytics.

@ccl-private hello,

Thank you for bringing this to our attention. It looks like there might be an issue with the labeling consistency in your training data.

To help us investigate further, could you please ensure the following:

1. Reproducible Example: You've provided a good start, but please confirm that this issue persists with the latest versions of torch and ultralytics. If not, please upgrade your packages and try again.
2. Data Consistency: Ensure that your dataset annotations are consistent across all sources. Mixed datasets can sometimes lead to label mismatches.

Here's a quick checklist to verify your setup:

• Ensure that all datasets (Objects365, Flickr30k, GQA, and LVIS) have consistent label formats.
• Double-check the json_file paths and contents to ensure they align correctly with the image paths.

If the issue persists, please provide a minimal reproducible example that includes a small subset of your data, so we can replicate the problem on our end. You can find more details on creating a minimal reproducible example here.

Looking forward to your response!

from ultralytics.

task: detect
mode: train
model: yolov8s-worldv2.pt
data:
train:
yolo_data:
- VisDrone.yaml
val:
yolo_data:
- lvis.yaml
epochs: 100
time: null
patience: 100
batch: 128
imgsz: 640
save: true
save_period: -1
cache: false
device:

• 0
• 1
  workers: 8
  project: null
  name: train11
  exist_ok: false
  pretrained: true
  optimizer: Adam
  verbose: true
  seed: 0
  deterministic: true
  single_cls: false
  rect: false
  cos_lr: false
  close_mosaic: 10
  resume: false
  amp: true
  fraction: 1.0
  profile: false
  freeze: 19
  multi_scale: false
  overlap_mask: true
  mask_ratio: 4
  dropout: 0.0
  val: true
  split: val
  save_json: false
  save_hybrid: false
  conf: null
  iou: 0.7
  max_det: 300
  half: false
  dnn: false
  plots: true
  source: null
  vid_stride: 1
  stream_buffer: false
  visualize: false
  augment: false
  agnostic_nms: false
  classes: null
  retina_masks: false
  embed: null
  show: false
  save_frames: false
  save_txt: false
  save_conf: false
  save_crop: false
  show_labels: true
  show_conf: true
  show_boxes: true
  line_width: null
  format: torchscript
  keras: false
  optimize: false
  int8: false
  dynamic: false
  simplify: false
  opset: null
  workspace: 4
  nms: false
  lr0: 0.0001
  lrf: 0.1
  momentum: 0.937
  weight_decay: 0.0005
  warmup_epochs: 3.0
  warmup_momentum: 0.8
  warmup_bias_lr: 0.1
  box: 7.5
  cls: 0.5
  dfl: 1.5
  pose: 12.0
  kobj: 1.0
  label_smoothing: 0.0
  nbs: 64
  hsv_h: 0.015
  hsv_s: 0.7
  hsv_v: 0.4
  degrees: 0.0
  translate: 0.1
  scale: 0.5
  shear: 0.0
  perspective: 0.0
  flipud: 0.0
  fliplr: 0.5
  bgr: 0.0
  mosaic: 1.0
  mixup: 0.0
  copy_paste: 0.0
  auto_augment: randaugment
  erasing: 0.4
  crop_fraction: 1.0
  cfg: null
  tracker: botsort.yaml
  save_dir: runs/detect/train11

from ultralytics.

I only use lvis.yaml and VisDrone.yaml to test the train batch label.


car with different label occurs.

from ultralytics.

the dataset label of VisDrone.yaml should be right, because I have trained a yolov8n on it , and get a map of 0.4.

from ultralytics.

Environment
pip install ultralytics

Minimal Reproducible Example

from ultralytics import YOLOWorld
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

data = dict(
    train=dict(
        yolo_data=["VisDrone.yaml"],
    ),
    val=dict(yolo_data=["lvis.yaml"]),
)

model = YOLOWorld("yolov8s-worldv2.pt")
model.train(data=data, batch=128, epochs=100, trainer=WorldTrainerFromScratch, device=[0, 1],
            optimizer="Adam", lr0=0.0001, lrf=0.1, freeze=19)  # cache='disk'

from ultralytics.

Hello @ccl-private,

Thank you for providing the environment details and a minimal reproducible example. It looks like you're encountering an issue with inconsistent labels for the same target during training.

To help us investigate further, please ensure the following:

1. Latest Versions: Verify that you are using the latest versions of torch and ultralytics. If not, please upgrade your packages and try again:

   pip install --upgrade torch ultralytics

2. Data Consistency: Ensure that the labels in your VisDrone.yaml and lvis.yaml datasets are consistent. Mixed datasets can sometimes lead to label mismatches. Double-check the label mappings in both datasets to ensure they align correctly.

3. Minimal Reproducible Example: Your provided example is helpful. However, if the issue persists, please provide a small subset of your data that replicates the problem. This will help us reproduce the issue on our end. You can find more details on creating a minimal reproducible example here.

Here is a quick checklist to verify your setup:

• Ensure that all datasets have consistent label formats.
• Double-check the yaml files and their contents to ensure they align correctly with the image paths and labels.

If the issue continues, please share any additional details or observations that might help us diagnose the problem.

Thank you for your cooperation, and we look forward to resolving this issue for you! 😊

from ultralytics.

Thank you for your prompt reply.

1. Latest Versions: torch is 1.11.0, ultralytics is 8.2.36. I upgraded torch to 2.3.1, ultralytics to 8.2.38, but the problem still occurs.
2. Data Consistency: I have not modified any code in "VisDrone.yaml" and "lvis.yaml", there are from https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/VisDrone.yaml and https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/lvis.yaml. Images and labels are downloaded automatically. the labels in VisDrone.yaml and lvis.yaml may be not same, I will check it.
3. Minimal Reproducible Example: datasets are downloaded automatically, If necessary, I can delete most of the data and test again.

from ultralytics.

Hello @ccl-private,

Thank you for the detailed follow-up. Let's address the points you've mentioned:

1. Latest Versions: Great to hear that you've upgraded to the latest versions of torch and ultralytics. This helps ensure we are working with the most recent updates and fixes.

2. Data Consistency: It's crucial that the labels in VisDrone.yaml and lvis.yaml are consistent. Since these datasets are automatically downloaded, there shouldn't be any discrepancies, but it's always good to double-check. Ensure that the class indices and names align correctly across both datasets.

3. Minimal Reproducible Example: If you can create a smaller subset of your data that replicates the issue, it would be immensely helpful. This allows us to reproduce the problem on our end and investigate a solution more efficiently. You can follow the guidelines for creating a minimal reproducible example here.

Additionally, here are a few steps you can take to troubleshoot further:

• Label Verification: Manually inspect a few images and their corresponding labels from both datasets to ensure they are correctly annotated and consistent.
• Training Script: Ensure that your training script is correctly referencing the datasets and that there are no mix-ups in the paths or configurations.

If the issue persists, please share the minimal reproducible example, and we will dive deeper into the problem.

Thank you for your cooperation and patience. We're here to help you get this resolved! 😊

from ultralytics.

This time, I commented out all the other datasets and tested only with coco8.yaml. The issue persists.
coco8 dataset is downloaded automatically.

from ultralytics import YOLOWorld
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
data = dict(
    train=dict(
        yolo_data=["coco8.yaml"],
        # yolo_data=["/fast_disk/ccl/codes/test/ultralytics_ccl/cfg/datasets/Objects365.yaml"],
        # grounding_data=[
        #     dict(
        #         img_path="./datasets/flickr30k/images",
        #         json_file="./datasets/flickr30k/final_flickr_separateGT_train.json",
        #     ),
        #     dict(
        #         img_path="./datasets/GQA/images",
        #         json_file="./datasets/GQA/final_mixed_train_no_coco.json",
        #     ),
        # ],
    ),
    # val=dict(yolo_data=["lvis.yaml"]),
    val=dict(yolo_data=["coco8.yaml"]),
)
model = YOLOWorld("yolov8s-worldv2.pt")
model.train(data=data, batch=128, epochs=100, trainer=WorldTrainerFromScratch, device=[0, 1],
            optimizer="Adam", lr0=0.0001, lrf=0.1, freeze=19)  # cache='disk'

from ultralytics.

here is the train batch lable.

from ultralytics.

Hello @ccl-private,

Thank you for providing the additional details and the minimal reproducible example. It's very helpful for us to diagnose the issue.

Given that the problem persists even with the coco8.yaml dataset, it suggests there might be an underlying issue with the training configuration or the model itself.

Here are a few steps to help troubleshoot further:

1. Verify Dataset Labels: Ensure that the labels in the coco8.yaml dataset are consistent and correctly mapped. You can manually inspect a few images and their corresponding labels to confirm this.

2. Check Training Configuration: Double-check your training configuration to ensure there are no discrepancies. Specifically, ensure that the data dictionary and the train and val keys are correctly set up.

3. Simplify the Setup: Try simplifying the training setup to isolate the issue. For example, reduce the batch size and the number of epochs to see if the problem persists.

Here is a simplified version of your training script for further testing:

from ultralytics import YOLOWorld
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

data = dict(
    train=dict(
        yolo_data=["coco8.yaml"],
    ),
    val=dict(yolo_data=["coco8.yaml"]),
)

model = YOLOWorld("yolov8s-worldv2.pt")
model.train(data=data, batch=32, epochs=10, trainer=WorldTrainerFromScratch, device=[0, 1],
            optimizer="Adam", lr0=0.0001, lrf=0.1, freeze=19)

If the issue still occurs, please let us know, and we can investigate further. Your cooperation and patience are greatly appreciated as we work to resolve this.

Thank you! 😊

from ultralytics.

I noticed that the original version of yoloworld can pass the parameters "num_classes=1203, num_training_classes = 80". Your version does not specifically pass these parameters. Is this the reason?

from ultralytics.

Hello @ccl-private,

Thank you for your observation and for providing the link to the original YOLO-World configuration. The parameters num_classes and num_training_classes are indeed crucial for ensuring that the model correctly interprets the number of classes in your dataset.

In our version, these parameters might not be explicitly set, which could lead to inconsistencies in label assignments. To address this, you can manually specify these parameters in your training script to match your dataset's requirements.

Here's how you can modify your training script to include these parameters:

from ultralytics import YOLOWorld
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

data = dict(
    train=dict(
        yolo_data=["coco8.yaml"],
    ),
    val=dict(yolo_data=["coco8.yaml"]),
)

model = YOLOWorld("yolov8s-worldv2.pt")
model.train(data=data, batch=32, epochs=10, trainer=WorldTrainerFromScratch, device=[0, 1],
            optimizer="Adam", lr0=0.0001, lrf=0.1, freeze=19, num_classes=1203, num_training_classes=80)

Please try this adjustment and let us know if it resolves the issue. If the problem persists, providing a minimal reproducible example would be very helpful for further investigation. You can find more details on creating one here.

Thank you for your cooperation and patience. We're here to help! 😊

from ultralytics.

@ccl-private Hi! no worries at all, the labels are updated during training when you try to train a yolo-world model from scratch so basically the plotted image would be drawn on different objects with the same index. This is also the difference between yolo-world and other models when training.
However, the model is updated by the text features generated by CLIP, so the labels that look like incorrect would not affect the training process. It's just the plotted images showing different indices since it's not easy to directly access the textual labels for plotting.

from ultralytics.

I actually aim to combine GQA data with my own labeled data for full fine-tuning. As discussed here: AILab-CVC/YOLO-World#299. However, ultralytics' map does not rise but falls. During the investigation, I found and suspected a problem with train batch lable.

from ultralytics.

@Laughing-q It seems that it may be a parameter problem. Could you please help test and provide a fine-tuning parameter so as not to make the map value fall?

from ultralytics import YOLOWorld
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch

data = dict(
    train=dict(
        yolo_data=["Objects365.yaml"],
        grounding_data=[
            dict(
                img_path="../datasets/flickr30k/images",
                json_file="../datasets/flickr30k/final_flickr_separateGT_train.json",
            ),
            dict(
                img_path="../datasets/GQA/images",
                json_file="../datasets/GQA/final_mixed_train_no_coco.json",
            ),
        ],
    ),
    val=dict(yolo_data=["lvis.yaml"]),
)
model = YOLOWorld("yolov8s-worldv2.pt")
model.train(data=data, batch=128, epochs=100, trainer=WorldTrainerFromScratch)

from ultralytics.

@ccl-private Ideally you should prepare your dataset in yolo format and also the data.yaml, then extending the yolo_data with your custom data.yaml should be ok:

        yolo_data=["Objects365.yaml", "custom_data.yaml"],

from ultralytics.