Pick and Place Robot using Bin Packing

This project aims to use and simulate an efficient bin packing algorithm to pick and place boxes into bins(containers) in CoppeliaSim.

About the Project
- Tech Stack
- File Structure
Getting Started
Theory and Approach
Results and Demo
Future Work
Troubleshooting
Contributors
Acknowledgements and Resources
License

About The Project

Bin packing is one of the most interesting problems in combinatorics. It has multiple applications, such as packaging boxes in containers, loading trucks with weight capacity constraints, creating file backups in media and technology mapping in FPGA semiconductor chip design. This project aims to implement a bin packing algorithm that packs variable sized boxes into containers and simulate the same in a CoppeliaSim Scene.

Tech Stack

File Structure

 ┣Algorithms                           # ALgorithms written and tested
 ┃ ┣1D Algorithms                      # 1 Dimensional Bin packing algorithms
 ┃ ┃ ┣BestFitAlgorithm.cpp             
 ┃ ┃ ┣FirstFitAlgorithm.cpp            
 ┃ ┃ ┗NextFitAlgorithm.cpp             
 ┃ ┣2D Algorithms                      # 2 Dimensional Bin packing algorithms
 ┃ ┃ ┗ShelfNextFit.cpp
 ┃ ┗3D Algorithms                      # 3 Dimensional Bin packing algorithms
 ┃ ┃ ┣3DBinPacking.py
 ┃ ┃ ┣3dalgo-dragos.py
 ┃ ┃ ┗3dalgo-pseudo-initial.py         # Pseudocode used to give the general outline of the algorithm
 ┃ ┗Resources                          # Resources referred for Algorithms
 ┃ ┣RectangleBinPack.pdf
 ┃ ┣3D_Bin_Packing_by_erick_dube.pdf
 ┃ ┗master_thesis on_Bin_Packing.pdf
 ┃
 ┣Assets                          # Contains simulation result videos
 ┃
 ┣Script                          # Contains the scripts used to run the scene
 ┃ ┣zmqRemoteApi                  # Contains the ZMQ API client files
 ┃ ┣Scene_Script.py
 ┃ ┣bp3d_greedy.py
 ┃ ┗bp3d.py
 ┃ 
 ┣Simulation_Scene                                # CoppeliaSim Simulation Scenes 
 ┃ ┣EKLAVYA_PICK_PLACE.ttt
 ┃ ┣Simulation_Multiple Boxes.ttt
 ┃ ┣Simulation_four_boxes.ttt
 ┃ ┗scene_ortho_view.ttt
 ┃
 ┣include                              # Essential add-on files needed for ZMQ-API 
 ┃ ┣cbor.lua
 ┃ ┣simAddOnZmq remote API.lua
 ┃ ┗simZMQ.lua
 ┗README.md

Getting Started

Prerequisites

Used CoppeliaSim version 4.2.0
Tested on Windows 10 and MacOS versions 10.14.6 and 10.15.7
Used Anaconda for Spyder IDE.

Installations

Clone the repo

git clone https://github.com/sagarchotalia/Pick-and-Place-Robot-Eklavya.git

For CoppeliaSim v4.2, Clone ZMQ API repository in the CoppeliaSim/programming folder.
Open the include folder of this repository. Add the simAddOnZMQ remote API.lua file to your main CoppeliaSim directory (for the add-on).
Then navigate to the CoppeliaSim/lua folder. Paste the scripts cbor.lua and simZMQ.lua in them.
If there are any files with the same name as the above, simply replace them with the files provided in this repository, otherwise the API will show an error.
Install OpenCV in Spyder Terminal.

Execution

Open Spyder IDE using either the directly downloaded application or through Anaconda Navigator.
Open the scene in Coppeliasim. Don't click on the Play icon, the script contains a function that will run the scene for you.
Run the Scene_Script.py script

Theory and Approach

The main idea of the project is that an item comes packaged inside a box on the conveyor belt, and then gets detected by two specific cameras used for vision sensing (one for the length and breadth and the other for the height and length). Then, using this image data captured by the cameras, the three spatial dimensions of the box are determined by using image processing. After this happens, the conveyor stops and the box is picked up by the gantry components and placed at the desired position coordinates as determined by the bin packing algorithm.

In the script, first we connect with the ZeroMQ API at port 23000. Then, the object handles are accessed (this is required in order to manipulate the specific parts of the robot). After that, the simulation needs to be started and then, inside object_dimensions function OpenCV functions need to be used to detect the box dimensions as soon as they come into the frame of the camera. Once the dimensions are determined the bin packing algorithm is called and the coordinates returned are used so that the robot places the box at the desired position by using the pick_place function.