JPEG core experiment for the evaluation of JPEG XR image coding

Summary

When a new technology for compression of digital still or moving pictures is submitted to the attention of the international standardization community, ad-hoc groups of experts are created to evaluate it. The evaluation usually consists of comparative studies with existing or concurrent technologies to test the compression efficiency achieved by the proposed coding algorithm, its computational complexity, and any additional functionalities  provided by the codec. The compression efficiency of di fferent coding strategies can be reliably compared only by means of subjective tests, carried out at the premises of several independent institutions and according to common evaluation methodologies de ned by experts.
The need for subjective tests is due to the fact that the compression effiiency of a coding algorithm expresses its ability to maximize the visual quality of a compressed image or video sequence versus the number of bits used to represent it. Even if a considerable e ort has been spent by the research community to develop algorithms which can objectively evaluate the quality of digital pictures, i.e. objective quality metrics, the ability of existing metrics to predict human judgement remains limited. Thus, the benchmark for any kind of quality assessments remains the subjective results collected by mean of experiments which are time consuming and have to be carefully designed.

In the paper “Subjective evaluation of JPEG XR image compression”, in Proceedings of SPIE 7443, authored by Francesca De Simone, Lutz Goldmann, Vittorio Baroncini and Touradj Ebrahimi, we present a detailed procedure for subjective quality evaluation of high resolution still pictures, by describing its application to performance evaluation of the JPEG XR compression algorithm. The new technology has been compared to existing JPEG and JPEG 2000 algorithms, considering compression of high resolution 24 bpp pictures, by mean of a campaign of subjective quality assessment tests which followed the general guidelines provided in the core experiment plan described in the document WG1N5001 by the AIC JPEG XR ad-hoc group. Sixteen naive subjects took part in experiments at EPFL and each subject participated in four test sessions, scoring a total of 186 test stimuli. A detailed procedure for the statistical analysis of subjective data is also proposed, which allows an accurate comparison of codec performance.

In order to facilitate experiment reproduction, the command line calls used for coding the pictures, are detailed below and the entire set of original, coded and decoded test and training pictures are available for free download:
 
 
The subjective results of our test campaign are also publicly available:
 


The password to access the data is jpegxrevaluation. If you download the data, it is assumed that you agree to the copyright notice reported below. If you use our data for your own publications please do not forget to reference this website and our paper.

 

Command lines used for coding the pictures

JPEG 4:2:0 coding

Software: IJG version 6b (http://www.ijg.org)

Encoding command: cjpeg –baseline –quality %q% in_RGB.bmp out.jpg

Decoding command: djpeg –bmp out.jpg out_RGB.bmp

JPEG 2000 4:2:0 coding

Software: Kakadu version 6.0 (http://www.kakadusoftware.com)

Encoding command: kdu_compress.exe -i in_Y.pgm, in_Cb.pgm, in_Cr.pgm -o out.jp2 -rate %rate% -jp2_space sYCC Cycc=no Ssigned=no Sprecision={8},{8},{8} Sdims={%w%,%h%},{%w/2%,%h/2%}, {%w/2%,%h/2%} Ssampling={1,1},{2,2},{2,2}

Decoding command: kdu_expand.exe -i out.jp2 -o out_Y.pgm, out_Cb.pgm, out_Cr.pgm

JPEG 2000 4:4:4 coding (with visual weighting)

Software: Kakadu version 6.0 (http://www.kakadusoftware.com)

Encoding command: kdu_compress.exe -i in_RGB.bmp -o out.jp2 -rate %rate%

Decoding command: kdu_expand.exe -i out.jp2 -o out_RGB.bmp

JPEG XR MS 4:2:0 coding

Software: provided by Microsoft Corporation to JPEG commettee members

Pre-encoding command: HDP_QMAP.exe in_RGB.bmp tmp.quant -q %quality% -n 3 3 -v 3 9

Encoding command: HDPEncApp.exe -i in_RGB.bmp –o out.wdp -d 1 -l 1 -D tmp.quant

Decoding command: HDPDecApp.exe -i out.wdp -o out_RGB.bmp


Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute the data provided and its documentation for research purpose only. The data provided may not be commercially distributed. In no event shall the Ecole Polytechnique Federale de Lausanne be liable to any party for direct, indirect, special, incidental, or consequential damages arising out of the use of the data and its documentation. The Ecole Polytechnique Federale de Lausanne  specifically disclaims any warranties. The data provided hereunder is on an “as is” basis and the Ecole Polytechnique Federale de Lausanne has no obligation to provide maintanance, support, updates, enhancements, or modifications.

Acknowledge

The work presented here was partially supported by the European Network of Excellence PetaMedia (FP7/2007-2011), and the Swiss National Foundation for Scientic Research in the framework of NCCR Interactive Multimodal Information Management (IM2). The authors thank Dr. Gary Sullivan and Dr. Thomas Richter for providing the JPEG XR MS software and the JPEG XR PS compressed pictures, respectively. Also, a special thank goes to the subjects who participated with high dedication to the subjective test.