3D Shape Segmentation With Projective Convolutional Networks

This is an interesting summary of an approach for shape segmentation. I think it's pretty cool how often VGG-16 gets used for transfer learning with good results. It's amazing that these models can represent enough knowledge to generate 3-D surfaces from single images. (I also like how many folks use airplanes as examples : - )



There's a website for the ShapeNet data set that they used as a benchmark in the video, and this paper describes the initial methods folks developed during the challenge right after the data set was released. That's a pretty neat approach. It reminds me a bit of the AIAA drag prediction workshops.


Here is the summary from the paper:

As a summary of all approaches and results, we have the following major observations:

1. Approaches from all teams on both tasks are deep learning based, which shows the unparallel popularity of deep learning for 3D understanding from big data
   
2. Various 3D representations, including volumetric and point cloud formats, have been tried. In particular, point cloud representation is quite popular and a few novel ideas to exploit point cloud format have been proposed
   
3. The evaluation metric for 3D reconstruction is a topic worth further investigation. Under two standard evaluation metrics (Chamfer distance and IoU), we observe that two different approaches have won the first place. In particular, the coarse-to-fine supervised learning method wins by the IoU metric, while the GAN based method wins by the Chamfer distance metric.
   

I like the hierarchical approach because it seems like it would be efficient. They use an octree data structure to allow them to only refine where they have a boundary label in a voxel. This reminds me a lot of Cartesian mesh refinement that some folks use in CFD for adaptive meshing.

Hierarchical Surface Generation from Single Image, Hierarchical Surface Prediction, by C. Hane, S.Tulsiani, J. Malik