We develop a progressive refinement algorithm that generates an approximate image quickly, then gradually refines it towards the final result. Our algorithm can reconstruct a high-quality image after evaluating only a small percentage of the pixels. For a typical scene, evaluating only 6 % of the pixels yields an approximate image that is visually hard to distinguish from an image with all the pixels evaluated. At this low sampling rate, previous techniques such as adaptive stochastic sampling suffer from artifacts including heavily jagged edges, missing object parts, and missing high-frequency details. A key ingredient of our algorithm is the directional coherence map (DCM), a new technique for handling general radiance discontinuities in a progressive ray tracing framework. Essentially an encoding of the directional coherence in image space, the DCM performs well on discontinuities that are usually considered extremely difficult, e.g. those involving non-polygonal geometry or caused by secondary light sources. Incorporating the DCM into a ray tracing system incurs only a negligible amount of additional computation. More importantly, the DCM uses little memory and thus it preserves the strengths of ray tracing systems in dealing with complex scenes. We have implemented our algorithm on top of RADIANCE. Our enhanced system can produce high-quality images significantly faster than RADIANCE – sometimes by orders of magnitude. Moreover, when the baseline system becomes less effective as its Monte Carlo components are challenged by difficult lighting configurations, our system will still produce high quality images by redistributing computation to the small percentage of pixels as dictated by the DCM.

Real-time audio and video data streamed over unreliable IP networks, such as the Internet, may encounter losses due to dropped packets or late arrivals. This paper reviews error-concealment schemes developed for streaming realtime audio and video data over the Internet. Based on their interactions with (video or audio) source coders, we classify existing techniques into source coder-independent schemes that treat underlying source coders as black boxes, and source coder-dependent schemes that exploit coder-specific characteristics to perform reconstruction. Last, we identify possible future research directions. 1. Introduction Increases in bandwidth and computational speed lead to growing interests in real-time audio and video transmissions over the Internet. In the Internet, packets carrying real-time data may be dropped or arrive too late to be useful because the Internet is a packet-switched, best-effort delivery service, with no guarantee on the quality of service (QoS). Traditi...

In the transmission of a precoded, stored video such as that in video-on-demand systems , it is often necessary to reduce the bit rate of the compressed video in cases that the network capacity is reduced. This paper proposes a novel block-dropping approach for rate shaping of MPEG-precompressed video. A directional spatial interpolation scheme is incorporated at the receiver to reconstruct faithfully a large percentage of the blocks that are intelligently dropped to meet the constraint of reduced bandwidth. The blocks are dropped in a way that is optimal in the rate-distortion sense. The new approach is conceptually quite different from conventional approaches in which coefficients (usually high frequency coefficients) are eliminated. Experimental results show that the proposed scheme can provide significantly higher peak signal-to-noise ratio (PSNR) and much better visual quality than conventional approaches. We also show that by jointly dropping blocks and coefficients, the bit rate...

In this paper, image compression utilizing visual redundancy is investigated. Inspired by recent advancements in image inpainting techniques, we propose an image compression framework towards visual quality rather than pixel-wise fidelity. In this framework, an original image is analyzed at the encoder side so that portions of the image are intentionally and automatically skipped. Instead, some information is extracted from these skipped regions and delivered to the decoder as assistant information in the compressed fashion. The delivered assistant information plays a key role in the proposed framework because it guides image inpainting to accurately restore these regions at the decoder side. Moreover, to fully take advantage of the assistant information, a compression-oriented edge-based inpainting algorithm is proposed for image restoration, integrating pixel-wise structure propagation and patch-wise texture synthesis. We also construct a practical system to verify the effectiveness of the compression approach in which edge map serves as assistant information and the edge extraction and region removal approaches are developed accordingly. Evaluations have been made in comparison with baseline JPEG and standard MPEG-4 AVC/H.264 intra-picture coding. Experimental results show that our system achieves up to 44 % and 33 % bits-savings, respectively, at similar visual quality levels. Our proposed framework is a promising exploration towards future image and video compression.

Error concealment (EC) is important for transmitting video over error prone networks such as the Internet or wireless networks. Different EC strategies have their own advantages for different scenarios. However, it is generally difficult for the decoder to figure out which strategy works the best for a specific case. This paper proposes to use data embedding to convey the necessary high level side information in a standard compliant way to help improve the decoder's EC performance. We show that the EC "mode" information (i.e.., whether spatial EC or temporal EC should be used) is critical for a spatial-temporal EC approach.

Quality-delay trade-o#s can be made in transmitting subband-coded images in the Internet by using either the TCP or the UDP protocol. Delivery by TCP gives superior decoding quality but with very long delays when the network is unreliable, whereas delivery by UDP has negligible delays but with degraded quality when packets are lost. Although images are delivered primarily by TCP today, we study in this paper the use of UDP to deliver multi-description reconstruction-based subband-coded images and the reconstruction of missing information at the receiver based on information received. We first determine empirically the interleaving factors that should be used in order to keep the probability of unrecoverable packet losses sufficiently small. Next, we propose a joint sender-receiver approach for designing transforms in multi-description subband coding. In the receiver, we use a simple interpolation-based reconstruction algorithm, as sophisticated concealment techniques cannot be employed in practice. In the sender, we design an optimized reconstruction-based subband transform (ORB-ST), with an objective of minimizing the mean squared error, assuming that some of the descriptions are lost and that the missing information is reconstructed by simple averaging at the destination. Experimental results show that our proposed ORB-ST performs well in real Internet tests, and UDP delivery of MDC images is an attractive alternative to TCP delivery. Keywords and Phrases: Error concealment, interpolation-based reconstruction, multidescription coding (MDC), real-time multimedia in the Internet, reconstruction-based subband image coding, single-description coding (SDC), TCP, UDP, World-Wide Web.

Video streaming over packet-loss networks faces the challenges that the networks are error-prone, transmission bandwidth is limited and fluctuating, the user device capabilities vary, and networks are heterogeneous. These challenges necessitate the need for smart adaptation of the precoded video. The focus of the thesis is error-resilient rate shaping for streaming precoded video over packet-loss networks. Given the packet-loss characteristic of the networks, the precoded video consists of channel-coded as well as source-coded bits. Error-resilient rate shaping is a filtering process that adapts the bit rates of the precoded video, in order to deliver the best video quality given the network condition at the time of delivery. We first illustrate "baseline rate shaping (BRS)" of the proposed error-resilient rate shaping as a baseline. Having introduced BRS with coarse decisions in rate adaptation, more sophisticated error-resilient rate shaping is proposed for layer-coded videos, namely, the enhancement layer video and the base layer video. "Fine-grained rate shaping (FGRS)" is proposed for streaming the enhancement layer video, and "errorconcealment aware rate shaping (ECARS)" is proposed for streaming the base layer video. FGRS and ECARS are formulated as rate-distortion (R-D) optimization problems. A two-stage R-D optimization approach is proposed to solve the R-D optimization problem in a fast and accurate manner. FGRS makes use of the fine granularity property of the MPEG-4 fine-granularityscalability bitstream and outperforms ad-hoc unequal packet-loss protection methods. ECARS takes into account error concealment (EC) performed at the receiver to deliver the part of precoded video that cannot be EC-reconstructed well. Frame dependency due to predictive coding and/...

This paper proposes a systematic rate controller (SRC) for content-aware streaming of MPEG-4 FGS video over the Internet. An active layer dropping technique is proposed to provide both coarse-grain and fine-grain scalability of smooth quality adaptation to bandwidth fluctuations and bit-rate variations of streamed video over a general timescale. The smooth quality adaptation is realized through the mode and state transition of a state machine that implements the SRC. The SRC effectively uses available bandwidth and client buffer by forward-shifting the FGS video stream. It provides protection to video segments with important content by introducing a content-aware priority-based layer model for the MPEG-4 FGS video stream.

’Eν αρχ´ην ɛ´ι o λóγoς

Distributed video coding (DVC) is a video coding paradigm allowing low complexity encoding for emerging applications such as wireless video surveillance. Side information (SI) generation is a key function in the DVC decoder, and plays a key-role in determining the performance of the codec. This paper proposes an improved SI generation for DVC, which exploits both spatial and temporal correlations in the sequences. Partially decoded Wyner-Ziv (WZ) frames, based on initial SI by motion compensated temporal interpolation, are exploited to improve the performance of the whole SI generation. More specifically, an enhanced temporal frame interpolation is proposed, including motion vector refinement and smoothing, optimal compensation mode selection, and a new matching criterion for motion estimation. The improved SI technique is also applied to a new hybrid spatial and temporal error concealment scheme to conceal errors in WZ frames. Simulation results show that the proposed scheme can achieve up to 1.0 dB improvement in rate distortion performance in WZ frames for video with high motion, when compared to state-of-the-art DVC. In addition, both the objective and perceptual qualities of the corrupted sequences are significantly improved by the proposed hybrid error concealment scheme, outperforming both spatial and temporal concealments alone. Copyright © 2009 Shuiming Ye et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 1.