Deep Learning-based Semantic Analysis of Sparse Light Field Ray Sets

Abstract

With the emergence of various light field (LF) acquisition systems and of novel techniques for processing and visualizing LFs, end-to-end LF systems start to head for the consumer market. Towards this, the semantic analysis of LFs can play a crucial role in LF processing (e.g. compression, storage and transmission), and in the standardization of LF representation schemes across various use cases. In this regard, we earlier have introduced fristograms as a tool to integrate semantics into LF processing. Fristograms collect sets of rays within a volume of a number of pixels in all 3 directions (horizontal, vertical and disparity) and thus enable semantic analysis based on the ray sets, and consequently semantic processing of LFs. Consequently, fristograms enable the application of filtering techniques considering the underlying characteristic of the scene (e.g. differentiate between Lambertian and non-Lambertian, occluded and dis-occluded regions in the scene). Motivated by the earlier results through statistical analysis of froxels enabling a significant reduction in number of rays while maintaining quality, in this paper, we explore learning-based analysis of froxels. Specifically, we propose to use a deep learning network to classify material properties (such as Lambertian, non-Lambertian, and outliers). Once the classification is done, the LF is filtered semantically. Preliminary results show that compared to the statistical ray analysis of froxels, a learning-based approach can reduce the number of rays even further, yet maintain the visual quality of the LF as measured by well-known quality metrics.