Giter VIP home page Giter VIP logo

prepare_detection_dataset's Introduction

背景

万事开头难。之前写图像识别的博客教程,也是为了方便那些学了很多理论知识,却对实际项目无从下手的小伙伴,后来转到目标检测来了,师从烨兄、亚光兄,从他们那学了不少检测的知识和操作,今天也终于闲下了,准备写个检测系列的总结。一方面分享知识希望可以一起学习,另一方面让一部分人少走弯路,快速上路(入坑)。

此部分代码:Github 博客地址: 目标检测系列一:如何制作数据集?

更新

  • (28/03/2019)
    • 新增 csv2labelme

1. 内容介绍

系列一主要介绍如何在常见的几种数据格式之间进行转换,以及万能中介csv格式的使用,这里列出以下几个:

  • csv to coco
  • csv to voc
  • labelme to coco
  • labelme to voc
  • csv to json

2. 标准格式

在使用转换脚本之前,必须要明确的几种格式

2.1 csv

不要一看是csv文件就直接拿来运行,如果不是,可以自行修改代码,或者修改标注文件。

转换脚本支持的csv格式应为以下形式:

  • csv/
    • labels.csv
    • images/
      • image1.jpg
      • image2.jpg
      • ...

labels.csv 的形式:

/path/to/image,xmin,ymin,xmax,ymax,label

例如:

/mfs/dataset/face/0d4c5e4f-fc3c-4d5a-906c-105.jpg,450,154,754,341,face
/mfs/dataset/face/0ddfc5aea-fcdac-421-92dad-144.jpg,143,154,344,341,face
...

注:图片路径请使用绝对路径

2.2 voc

标准的voc数据格式如下:

  • VOC2007/
    • Annotations/
      • 0d4c5e4f-fc3c-4d5a-906c-105.xml
      • 0ddfc5aea-fcdac-421-92dad-144/xml
      • ...
    • ImageSets/
      • Main/
        • train.txt
        • test.txt
        • val.txt
        • trainval.txt
    • JPEGImages/
      • 0d4c5e4f-fc3c-4d5a-906c-105.jpg
      • 0ddfc5aea-fcdac-421-92dad-144.jpg
      • ...
2.3 coco

此处未使用测试集

  • coco/
    • annotations/
      • instances_train2017.json
      • instances_val2017.json
    • images/
      • train2017/
        • 0d4c5e4f-fc3c-4d5a-906c-105.jpg
        • ...
      • val2017
        • 0ddfc5aea-fcdac-421-92dad-144.jpg
        • ...
2.4 labelme
  • labelme/
    • 0d4c5e4f-fc3c-4d5a-906c-105.json
    • 0d4c5e4f-fc3c-4d5a-906c-105.jpg
    • 0ddfc5aea-fcdac-421-92dad-144.json
    • 0ddfc5aea-fcdac-421-92dad-144.jpg

Json file 格式: (imageData那一块太长了,不展示了)

{
  "version": "3.6.16",
  "flags": {},
  "shapes": [
    {
      "label": "helmet",
      "line_color": null,
      "fill_color": null,
      "points": [
        [
          131,
          269
        ],
        [
          388,
          457
        ]
      ],
      "shape_type": "rectangle"
    }
  ],
  "lineColor": [
    0,
    255,
    0,
    128
  ],
  "fillColor": [
    255,
    0,
    0,
    128
  ],
  "imagePath": "004ffe6f-c3e2-3602-84a1-ecd5f437b113.jpg",
  "imageData": ""   # too long ,so not show here
  "imageHeight": 1080,
  "imageWidth": 1920
}

3. 如何使用转换脚本

3.1 csv2coco

首先更改csv2coco.py中以下几个配置

classname_to_id = {"person": 1}  # for your dataset classes
csv_file = "labels.csv"  # annatations file path
image_dir = "images/"    # original image path
saved_coco_path = "./"   # path to save converted coco dataset

然后运行 python csv2coco.py

会自动创建文件夹并复制图片到相应位置,运行结束后得到如下:

  • coco/
    • annotations/
      • instances_train2017.json
      • instances_val2017.json
    • images/
      • train2017/
        • 0d4c5e4f-fc3c-4d5a-906c-105.jpg
        • ...
      • val2017
        • 0ddfc5aea-fcdac-421-92dad-144.jpg
        • ...
3.2 csv2voc

首先更改csv2voc.py中以下几个配置

csv_file = "labels.csv"
saved_path = ".VOC2007/" # path to save converted voc dataset     
image_save_path = "./JPEGImages/"   # converted voc images path
image_raw_parh = "images/"          # original image path

然后运行 python csv2voc.py

同样会自动创建文件夹并复制图片到相应位置,运行结束后得到如下:

  • VOC2007/
    • Annotations/
      • 0d4c5e4f-fc3c-4d5a-906c-105.xml
      • 0ddfc5aea-fcdac-421-92dad-144/xml
      • ...
    • ImageSets/
      • Main/
        • train.txt
        • test.txt
        • val.txt
        • trainval.txt
    • JPEGImages/
      • 0d4c5e4f-fc3c-4d5a-906c-105.jpg
      • 0ddfc5aea-fcdac-421-92dad-144.jpg
      • ...
3.3 labelme2coco

首先更改labelme2coco.py中以下几个配置

classname_to_id = {"person": 1}  # for your dataset classes
labelme_path = "labelme/"  # path for labelme dataset
saved_coco_path = "./"     # path for saved coco dataset

然后运行 python labelme2coco.py,生成文件形式同csv2coco

3.4 labelme2voc

首先更改labelme2voc.py中以下几个配置

labelme_path = "labelme/"  # path for labelme dataset
saved_coco_path = "./"     # path for saved coco dataset

然后运行 python labelme2voc.py,生成文件形式同csv2voc

3.5 csv2labelme

首先更改csv2labelme.py中以下几个配置

image_path = "./images/"  # path for images
csv_file = "./"     # path for csv annotations

然后运行 python csv2labelme.py,生成的json文件会保存在image_path下,切换路径过去,直接labelme便 可以查看标签.

4. 万能中介csv

从上面的转换格式中可以看出,并没有给出如何转到csv的,一是因为太过于简单,而是主流检测框架很少支持这种格式的数据输入。以下给出如何将标注信息写入csv

info = [[filename0,"xmin ymin xmax ymax label0"],
          filename1,"xmin ymin xmax ymax label1"]
csv_labels = open("csv_labels.csv","w")
for filename,bboxes in info:
    bbox = bboxes.split(" ")
    label = bbox[-1]
    csv_labels.write(filename+","+bbox[0]+","+bbox[1]+","+bbox[2]+","+bbox[3]+","+label+"\n")
csv_labels.close()

是不是非常简单。。。如果你不知道如何从原始的标签文件中读取得到标注信息,那没办法了,学学编程吧,23333

TODO

    1. Multiprocessing

致谢

感谢这么久以来对本项目支持的各位大佬!

Stargazers repo roster for @spytensor/prepare_detection_dataset

prepare_detection_dataset's People

Contributors

spytensor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

prepare_detection_dataset's Issues

一幅图像中有多个目标,CSV文件格式为?

一幅图像中有多个目标,labels.csv 形式是/path/to/image,xmin,ymin,xmax,ymax,label, xmin2, ymin2, xmax2, ymax2, label2吗?还是需要分开两条记录,
/path/to/image,xmin,ymin,xmax,ymax,label
/path/to/image,xmin2,ymin2,xmax2,ymax2,label2

有1个bug想请教一下

Traceback (most recent call last):
File "labelme2coco.py", line 115, in
train_instance = l2c_train.to_coco(train_path)
File "labelme2coco.py", line 30, in to_coco
self.images.append(self._image(obj, json_path))
File "labelme2coco.py", line 58, in _image
img_x = utils.img_b64_to_arr(obj['imageData'])
File "/home/hustget/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/labelme/utils/image.py", line 24, in img_b64_to_arr
img_data = base64.b64decode(img_b64)
File "/home/hustget/miniconda2/envs/open-mmlab/lib/python3.7/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (73941) cannot be 1 more than a multiple of 4
这个bug需要怎么改

使用csv2coco转出来的json存在图片重复的情况

{ "height": 2400, "width": 4000, "id": 2505, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2506, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2507, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2508, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2509, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2510, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2511, "file_name": "06266.jpg" }, { "height": 2400, "width": 4000, "id": 2512, "file_name": "06266.jpg" },

使用csv2coco转出来的json存在图片重复的情况,即一个box就生成了一张图,应该是多个box对应一张图才对

csv参数

你好,请问下如果一张图片由多个目标,那就有多个bbox的坐标值,那csv的格式应该是下面(1)还是(2)呢:
(1) image_path1, x1min, y1min, x1max, y1max, cat1, x2min, y2min, x2max, y2max, cat2.....
(2) image path1, x1min, y1min, x1max, y1max, cat1
image_path1, x2min, y2min, x2max, y2max, cat2
.....

help

我用您的代码转coco image字段的ID 不自增 可以交流以下吗 我VX wei4194162

csv格式问题

如果一个图片中有多个框(这种情况很多见),那么csv格式是一行对应一个框还是一行对应一张图片呢?

对于多类别的是否可用?

你好,对于多个类别的情况,classname_to_id 中不止一个类别时,感觉是否存在一定问题?
因为这里,划分数据集是这样做的:
train_path, val_path = train_test_split(json_list_path, test_size=0.12)
期待大佬的回复,谢谢啦

ValueError train_test_split函数没有train_target参数

您好,我运行了您的代码,可是出现了

ValueError: With n_samples=0, test_size=0.15 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.

错误,请问train_target该设成什么呢

一个图片中多个框的csv格式

你好,请问下如果一张图片中有多个目标,那csv的一行对应的是一个目标框信息还是多个目标信息, 麻烦您有时间解答下,谢谢!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.