We extend a recently-proposed framework for the rate-distortion optimized transmission of packetized media. The original framework assumed that each packet has a single arrival deadline and that a packet is useless if it arrives after its deadline. In practice, however, packets may be associated with multiple deadlines. Examples include the case of compressed video that uses bi-directional prediction and the case of decoders that can recover from late packet arrivals through the accelerated retroactive decoding of the dependency chain. We extend the original framework to consider multiple deadlines. In our experimental results for the case of the accelerated retroactive decoding of late packets, the multiple-deadline formulation yields up to a 1.5 dB improvement in rate-distortion performance compared to the original, singledeadline formulation. The results indicate, furthermore, that accelerated retroactive decoding offers significant benefit only when coupled with a scheduler that considers multiple deadlines.

High resolution digital imaging sensors are becoming more widespread. The challenges in delivering this high resolution content to the client are posed by limited resolution of display panels and/or limited bit-rate for communications. We propose a video coding scheme which enables virtual pan/tilt/zoom functionality during the streaming session. This way the server can adapt and stream only those regions of the video content that are desired at that time at the client’s end. Apart from generating a multi-resolution representation, our coding scheme uses P slices from H.264/AVC for random access to arbitrary regions within every spatial resolution. We study the trade-off in the choice of slice size. A larger slice size enables higher coding efficiency for representing the entire scene but increases the pixel overhead. The pixel overhead is due to superfluous pixels that are transmitted but not displayed at the client’s end. The optimal slice size achieves the best trade-off and minimizes the expected number of bits transmitted to the client per frame. Our analysis helps to predict the optimal slice size, which depends on the signal as well as the display resolution. Experimental results confirm the optimality of the predicted slice size for various test cases. 1.

A theoretical framework to analyze the rate-distortion performance of a light field coding and streaming system is proposed. This framework takes into account the statistical properties of the light field images, the accuracy of the geometry information used in disparity compensation, and the prediction dependency structure or transform used to exploit correlation among views. Using this framework, the effect that various parameters have on compression efficiency is studied. The framework reveals that the efficiency gains from more accurate geometry, increase as correlation between images increases. The coding gains due to prediction suggested by the framework match those observed from experimental results. This framework is also used to study the performance of light field streaming by deriving a view-trajectorydependent rate-distortion function. Simulation results show that the streaming results depend both the prediction structure and the viewing trajectory. For instance, independent coding of images gives the best streaming performance for certain view trajectories. These and other trends described by the simulation results agree qualitatively with actual experimental streaming results.

Abstract—High-quality, photorealistic image-based rendering datasets are typically too large to download entirely before viewing, even when compressed. It is more suitable to instead stream the required image data to a remote user who can start interacting with the dataset immediately. This paper presents an interactive light field streaming system and proposes packet scheduling for transmitting the encoded image data in a rate-distortion optimized manner. An interactive light field streaming system must have low user latency. The system presented in this paper predicts the future user viewing trajectory to mitigate the effects of the low-latency constraints. Experimental results show that view prediction can improve performance, and that this improvement is limited by the prediction accuracy. The proposed packet scheduling algorithm considers network conditions and rate-distortion cost, knowledge of sent and received images, and the distortion for a set of images, to optimize the rendered image quality for the remote user. Rate-distortion optimized scheduling can be implemented either at the receiver or the sender. It is shown that this rate-distortion optimized packet scheduling can significantly improve performance over a heuristic scheduling approach. Experimental results also show that the encoding prediction dependency structure affects streaming performance both through the compression efficiency of the encoding and also through any decoding dependencies that may be introduced. Index Terms—Image-based rendering, image coding, image communication, interactive streaming, light field coding, light field streaming, light fields, rate-distortion optimized streaming. I.

A receiver-based rate-distortion optimized framework to interactively stream scalable bitstreams of light fields is proposed. Based on the desired viewpoints, the predicted future buffer state, the network characteristics, and the target transmission rate, the receiver customizes the data request in order to minimize the distortion in the frames to be rendered. Experimental results show that the proposed framework improves the rendering quality by 1.0 2.6 dB over a heuristic scheme at the same rate. Correspondingly, to achieve the same rendering quality the proposed framework saves 30% 40% of the rate over the heuristic scheme.

In this paper, we apply generalized sampling to imagebased rendering (IBR) data, more specifically, the lightfield. We show that in theory the lowest sampling rate of lightfield when we use generalized sampling can be as low as half of that when we use rectangular sampling. However, in practice rectangular sampling has several advantages over generalized sampling. We analyze the pros and cons for each sampling approach, and explain why in practice rectangular sampling is still more preferable.

this paper we describe a system to show some limited effects on a static toy-car model and present techniques that can be used in similar setups. Our focus is on creating apparent motion for animation

Abstract—High-spatial-resolution videos offer the possibility of viewing an arbitrary region-of-interest (RoI) interactively. Zoom functionality enables watching high-resolution content even on displays of lower spatial resolution. If arbitrary regions corresponding to arbitrary zoom factors can be served to the user, the transmission and/or decoding of the entire high-spatial-resolution video can be avoided. Moreover, if the video content can be encoded such that arbitrary RoIs corresponding to different zoom factors can be simply extracted from the compressed bitstream, we can avoid dedicated video encoding for each user. We propose such a video coding scheme that is vital in allowing the system to scale to large numbers of remote users as well as to encode and store the content for subsequent repeated playback. Apart from generating a multi-resolution representation, our coding scheme uses P slices from H.264/AVC. We study the tradeoff in the choice of slice size. A larger slice size enables higher coding efficiency for representing the entire scene but increases the number of pixels that have to be transmitted. The optimal slice size achieves the best tradeoff and minimizes the expected transmission bitrate. Experimental results confirm the optimality of our predicted slice size for various test cases. Furthermore, we propose an improvement based on background extraction and long-term memory motion-compensated prediction. Experiments indicate up to 85 % bitrate reduction while retaining efficient random access capability. Index Terms—region-of-interest, pan/tilt/zoom, interactive video streaming.

Image-based rendering (IBR) has attracted a lot of research interest recently. In this paper, we survey the various techniques developed for IBR, including representation, sampling and compression. The goal is to provide an overview of research for IBR in a complete and systematic manner. We observe that essentially all the IBR representations are derived from the plenoptic function, which is seven dimensional and difficult to handle. We classify various IBR representations into two categories based on how the plenoptic function is simplified, namely restraining the viewing space and introducing source descriptions. In the former category, we summarize six common assumptions that were often made in various approaches and discuss how the dimension of the plenoptic function can be reduced based on these assumptions. In the latter category, we further categorize the methods based on what kind of source description was introduced, such as scene geometry, texture map or reflection model. Sampling and compression are also discussed respectively for both categories.