Done by Pietro Cipriani, Alessio Zeni, Fabio Giovanazzi for the fall semester of 2023 as an assignment for the Fundamentals of Robotics course.

Click on the image to download the video demo, or pick one of operational_space.mp4 or joint_space.mp4. The videos contain the terminals with the various ROS programs being run on top left, the block detection on bottom left, the block position recognition with the point cloud at the center, and the Gazebo simulation on the right.

You can find the project documentation for each part of the project here:

• Planner
• Position detection
• Block detector
• Controller
• Model training

Moreover, we collected all documentation in doxygen format for easy access. After downloading this repo, you can open ./html/index.html in your browser to view the docs of all methods and classes in the project.

• / the root folder of this repo is a catkin workspace, see Catkin workspace
• src/planner/ contains a catkin package for kinematic calculations
• src/position_detection/ contains a catkin package that extracts blocks positions from block detections and from the point cloud
• src/block_detector/ contains the catkin package responsible for detecting blocks in images
• src/controller/ is the catkin package that connects all components
• vision_training/ contains the code that was used to train the YoLoS model that recognizes blocks

The root folder of this repository is a catkin workspace created using catkin_make. In order for the workspace to be active, you need to run catkin_make in the root of this repository, and then source devel/setup.bash. The latter command can be put in .bashrc so that it's executed before starting any terminal session.

This project runs inside https://github.com/mfocchi/locosim. Execute all of the following commands as a normal user, not as root.

cd ~
git clone https://github.com/pietrocipriani/progetto-robotica
cd progetto-robotica
python3 -m pip install -r requirements.txt
catkin_make

Add source $HOME/progetto-robotica/devel/setup.bash to .bashrc and reopen any terminal to update the sourcing configuration.

Edit the file ~/ros_ws/src/locosim/robot_descriptions/gripper_description/gripper_description/urdf/soft_finger.xacro on line 25 and increase the upper limit of the gripper to 0.9 (otherwise big blocks can't be picked up):

- <limit effort="50" velocity="10.0" lower="-0.5" upper="0.3" />
+ <limit effort="50" velocity="10.0" lower="-0.5" upper="0.9" />

You will need 4 or 5 terminals open to run the project. Each command below is meant to be run in a separate terminal, but the order of commands should be preserved and before executing each command, the previous one should have finished setting up.

# runs the main simulation
cd ~/ros_ws/src/locosim/robot_control/base_controllers && python3 ur5_generic.py

# runs the block detection service
rosrun block_detector inspect_detected_blocks.py

# runs the block position detection service,
# and the corresponding visualization
rosrun position_detection precise_placement.py

# runs the controller node
rosrun controller controller_node

# (optional) opens a visualization for detected blocks
rosrun block_detector inspect_detected_blocks.py 6

The following commands were useful to work with ROS and with the mfocchi/locosim container.

These aliases allow avoiding to use the docker management script in mfocchi/locosim which has unwanted defaults and causes other issues. With the following commands the UIs run fine under wayland (thanks to xhost), ROS doesn't crash under Arch (thanks to --ulimit nofile=1024:524288) and is more straightforward. Replace MY_LAB_HOME and MY_USER appropriately, before adding this to your host computer .bashrc.

alias lab-robotica-run='sudo docker run --name lab_robotica --gpus all --user="$(id -u):$(id -g)" --workdir="/home/$USER" --volume="/etc/group:/etc/group:ro" --volume="/etc/shadow:/etc/shadow:ro" --volume="/etc/passwd:/etc/passwd:ro" --device=/dev/dri:/dev/dri -e "QT_X11_NO_MITSHM=1" --network=host --hostname=lab_rob_hn --add-host="lab_rob_hn:127.0.1.1" --volume "/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume="MY_LAB_HOME:$HOME" --env=HOME --env=USER --privileged --ipc=host -e SHELL --env="DISPLAY=$DISPLAY" --shm-size=2g --memory=16g --ulimit nofile=1024:524288 -itd mfocchi/trento_lab_framework:introrob && sudo docker exec --user root lab_robotica /usr/bin/sed -i s/%sudo/MY_USER/g /etc/sudoers'
alias lab-robotica-start-x='xhost +local:docker && sudo docker start lab_robotica'
alias lab-robotica-start-w='xhost +local:docker && xhost "+SI:localuser:$(id -un)" && sudo docker start lab_robotica'
alias lab-robotica-user="sudo docker exec -it --user root lab_robotica sh -c 'su MY_USER'"
alias lab-robotica-root='sudo docker exec -it --user root lab_robotica /bin/bash'

This alias is useful to kill all jobs in the background, so if a ROS command doesn't terminate, you can Ctrl+Z it and then run killbg:

alias killbg="kill -9 \`ps | grep -v -E 'bash|grep|ps|sed|tee|cat|CMD' | sed 's@^[^0-9]*\([0-9]\+\).*@\1@' | tee >(cat >&2)\`"