Hardware + Parts

• Turtlebot

• RealSense 3D Camera

  

  • the camera was secured to the front of TurtleBot, facing forward (I don’t have a better picture)

    

• Boxes of Varying Sizes (acting as chairs and tables)

Solution

Camera to Occupancy Grid

Using the ROS realsense2_camera camera node, the D435 camera publishes camera data to a variety of topics, as described here. We are specifically interested in the /camera/depth/color/points topic, which publishes a PointCloud2 message containing the 3D points and their corresponding RGB values being seen by the camera.

We then want to process the pointcloud and convert it into our 3D occupancy grid. This involves a few steps:

1. Transform the pointcloud from camera coordinates to Turtlebot coordinates. Because the camera was connected directly to the lab computer rather than the Turtlebot, this prevented us from directly computing the transform between the two coordinate frames. Since the camera is fixed in place, we manually calculated and applied this transformation.
2. Transform the pointcloud from Turtlebot coordinates to global coordinates This was done by obtaining the coordinate transform between the Turtlebot base_footprint (Turtlebot frame) and odom (global fixed frame) frames, and applying it to the pointcloud. Fortunately, both the camera and the Turtlebot used units of meters, so there was no need to scale the transformations.
3. Update 3D Occupancy Grid: draw rays from the Turtlebot location to each 3D point, marking all voxels along the ray as unoccupied and the voxel at the end of the ray as occupied This process was slow, as it was computationally intensive. To speed up the process, instead of drawing a ray from the turtlebot location to the 3D point, we instead drew rays from the voxels corresponding to the two locations. This greatly sped up computation speed, at a (unnoticeable) cost in precision. We additionally limited our z-axis to 10 centimeters, as any information in our occupancy grid beyond that height was redundant. We also labeled the occupied voxels with the corresponding color of their points, which we will use in segmentation.

Occupancy Grid to Object Detection

Next, we want to process our occupancy grid to identify and locate our objects.

In order to group our voxels into objects, we first ran agglomerative clustering on occupied voxels, which worked effectively in grouping the objects. We found that single linkage with a distance_threshold of 3 voxels worked the most consistently.

Because our boxes were painted green, purple, or unpainted brown, we then analyzed the color of each voxel grouping to identify the object. Attempts to threshold the raw RGB values were unsuccessful due to variations in brightness (i.e. if the boxes were underneath the lab station in the shadow vs in the light), but we found success in threshold the ratios between two channels.

This allowed us to identify objects, and calculate the location of each object by finding the mean location of the corresponding voxels.