Date: 2025-08-08

ISO/IEC DIS 21794-2:2025(en)

ISO/IEC JTC1/SC 29/WG 01

Secretariat: JISC

Information technology —Plenoptic image coding system (JPEG Pleno) — Part 2: Light field coding

Technologies de l'information — Système de codage d'images plénoptiques (JPEG Pleno) — Partie 2: Titre manque

© ISO/IEC 2025

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of the requester.

ISO copyright office

CP 401 • Ch. de Blandonnet 8

CH-1214 Vernier, Geneva

Phone: +41 22 749 01 11

Email: copyright@iso.org

Website: www.iso.org

Published in Switzerland

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.

The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types of documents should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).

This 2nd edition of ISO/IEC 21794-2 ("Plenoptic image coding system (JPEG Pleno) Part 2: Light field coding") integrates AMD1 of ISO/IEC 21794-2 (“Profiles and levels for JPEG Pleno Light Field Coding”) and includes the specification of an additional coding mode, entitled Slanted 4D transform mode and its associated profile.

ISO/IEC 21794-2 ("Plenoptic image coding system (JPEG Pleno) Part 2: Light field coding") specifies two coding modes: the 4D transform mode and the 4D prediction mode. The 4D prediction mode is efficient for light fields of all baselines but depends on the availability of accurate depth information. The 4D transform mode, although not relying on any sort of depth information, is only efficient for coding narrow baseline light fields. The Slanted 4D transform mode, based on 4D transformations, is efficient for light fields with both narrow and wide baselines and does not rely on the availability of depth information.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights. Details of any patent rights identified during the development of the document will be in the Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC list of patent declarations received (see patents.iec.ch).

Any trade name used in this document is information given for the convenience of users and does not constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions related to conformity assessment, as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.

A list of all parts in the ISO/IEC 21794 series can be found on the ISO website.

Any feedback or questions on this document should be directed to the user’s national standards body. A complete listing of these bodies can be found at www.iso.org/members.html and www.iec.ch/national-committees.

Introduction

This document is part of a series of standards for a system known as JPEG Pleno. This document defines the JPEG Pleno framework. It facilitates the capture, representation, exchange and visualization of plenoptic imaging modalities. A plenoptic image modality can be a light field, point cloud or hologram, which are sampled representations of the plenoptic function in the form of, respectively, a vector function that represents the radiance of a discretized set of light rays, a collection of points with position and attribute information, or a complex wavefront. The plenoptic function describes the radiance in time and in space obtained by positioning a pinhole camera at every viewpoint in 3D spatial coordinates, every viewing angle and every wavelength, resulting in a 7D function.

JPEG Pleno specifies tools for coding these modalities while providing advanced functionality at system level, such as support for data and metadata manipulation, editing, random access and interaction, protection of privacy and ownership rights.

Information technology —Plenoptic image coding system (JPEG Pleno)— Part 2: Light field coding

This document specifies a coded codestream format for storage of light field modalities as well as associated metadata descriptors that are light field modality specific. This document also provides information on the encoding tools.

The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

ITU-T Rec. T.800 | ISO/IEC 15444‑1, Information technology — JPEG 2000 image coding system — Part 1: Core coding system

ITU-T Rec. T.801 | ISO/IEC 15444‑2, Information technology — JPEG 2000 image coding system — Part 2: Extensions

ISO/IEC 21794‑1:2020, Information technology — Plenoptic image coding system (JPEG Pleno) — Part 1: Framework

ISO/IEC 60559, Information technology — Microprocessor Systems — Floating-Point arithmetic

ITU-T T.81 | ISO/IEC 10918-1, Information technology – Digital compression and coding of continuous-tone still images – Requirements and guidelines

ITU-T Rec. T.84 | ISO/IEC 10918-3, Information technology – Digital compression and coding of continuous-tone still images: Extensions

For the purposes of this document the terms and definitions given in ISO/IEC 21794-1 and the following apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— ISO Online browsing platform: available at https://www.iso.org/obp

— IEC Electropedia: available at http://www.electropedia.org/

3.1

arithmetic coder

entropy coder that converts variable length strings to variable length codes (encoding) and vice versa (decoding)

3.2

bit-plane

two-dimensional array of bits

3.3

4D bit-plane

four-dimensional array of bits

3.4

coefficient

numerical value that is the result of a transformation or linear regression

3.5

compression

reduction in the number of bits used to represent source image data

3.6

depth

distance of a point in 3D space to the camera plane

3.7

disparity view

image that for each pixel of the subaperture view contains the apparent pixel shift between two subaperture views along either horizontal or vertical axis

3.8

hexadeca-tree

division of a 4D region into 16 (sixteen) 4D subregions

3.9

pixel

collection of sample values in the spatial image domain having all the same sample coordinates

EXAMPLE A pixel may consist of three samples describing its red, green and blue value.

3.10

plenoptic function

amount of radiance in time and in space by positioning a pinhole camera at every viewpoint in 3D spatial coordinates, every viewing angle and every wavelength, resulting in a 7D representation

3.11

reference view

subaperture view that is used as one of the references to generate the intermediate views

3.12

subaperture view

subaperture image

image taken of the 3D scene by a pinhole camera positioned at a particular viewpoint and viewing angle

3.13

texture

pixel attributes

EXAMPLE Colour information, opacity, etc.

3.14

transform

transformation

mathematical mapping from one signal space to another

Integer numbers are expressed as bit patterns, hexadecimal values or decimal numbers. Bit patterns and hexadecimal values have both a numerical value and an associated particular length in bits.

Hexadecimal notation, indicated by prefixing the hexadecimal number by "0x", may be used instead of binary notation to denote a bit pattern having a length that is an integer multiple of 4. For example, 0x41 represents an eight-bit pattern having only its second most significant bit and its least significant bit equal to 1. Numerical values that are specified under a "Code" heading in tables that are referred to as "code tables" are bit pattern values (specified as a string of digits equal to 0 or 1 in which the left-most bit is considered the most-significant bit). Other numerical values not prefixed by "0x" are decimal values. When used in expressions, a hexadecimal value is interpreted as having a value equal to the value of the corresponding bit pattern evaluated as a binary representation of an unsigned integer (i.e. as the value of the number formed by prefixing the bit pattern with a sign bit equal to 0 and interpreting the result as a two's complement representation of an integer value). For example, the hexadecimal value 0xF is equivalent to the 4-bit pattern '1111' and is interpreted in expressions as being equal to the decimal number 15.

NOTE Many of the operators used in document are similar to those used in the C programming language.

Operators are listed in descending order of precedence. If several operators appear in the same line, they have equal precedence. When several operators of equal precedence appear at the same level in an expression, evaluation proceeds according to the associativity of the operator either from right to left or from left to right.

This document specifies the JPEG Pleno Light Field superbox and the JPEG Pleno light field decoding algorithm. The generic JPEG Pleno Light Field superbox syntax is specified in Annex A.

The specified light field decoding algorithm distinguishes three coding modes:

— 4D transform mode: this mode is specified in Annex B and is based on a 4D inverse discrete cosine transform (IDCT), 4D block partitioning, and 4D bit-plane hexadeca-tree decoding.

— 4D prediction mode: this mode is based the prediction of intermediate views based on reference views and normalized disparity maps. The signalling syntax and decoding of the reference views is addressed in Annex C, the normalized disparity views in Annex D, and the prediction parameters and residual views in Annex E. The intermediate views are reconstructed in a decoding process that involves view warping, view merging and prediction error correction.

— Slanted 4D transform mode: this mode is specified in Annex F and is based on changing the EPI slants/slopes in the 4D blocks by applying a geometric transformation before the conventional 4D-DCT stage of the 4D Transform mode architecture, notably to make the slopes of the lines composing the EPIs as aligned with one of the separable 4D-DCT dimensions as possible.

The overall architecture of the three coding modes (Figure 1) provides the flexibility to configure the encoding and decoding system depending on the requirements of the addressed use case.

Figure 1 — Generic JPEG Pleno light field decoder architecture.

An encoding process converts source light field data to coded light field data.

In order to conform with this document, an encoder shall conform with the codestream format syntax and file format syntax specified in the annexes for the encoding process(es) embodied by the encoder.

A decoding process converts coded light field data to reconstructed light field data. Annexes A through F describe and specify the decoding process.

A decoder is an embodiment of the decoding process. In order to conform to this document, a decoder shall convert all, or specific parts of, any coded light field data that conform to the file format syntax and codestream syntax specified in Annex A to F to a reconstructed light field.

Annex A specifies the description of the JPEG Pleno Light Field superbox.

This document specifies three approaches to represent a compressed representation of light field data: the 4D Transform mode is specified in Annex B, the 4D Prediction mode is specified in Annex C, Annex D and Annex E. Annex C details the signalling of the reference view data, Annex D the signalling of the normalized disparity views and , Annex E the signalling of the prediction parameters to generate the intermediate views and residual view data to compensate for prediction errors, and Annex F specifies the Slanted 4D Transform mode. Annex G defines the profiles and levels of the three coding modes.

1. 
   (normative)
   
   JPEG Pleno Light Field superbox
   1. General

This annex specifies the use of the JPEG Pleno Light Field superbox which is designed to contain compressed light field data and associated metadata. The listed boxes shall comply with their definitions as specified in ISO/IEC 21794-1.

This document may redefine the binary structure of some boxes defined as part of the ISO/IEC 15444-1 or ISO/IEC 15444-2 file formats. For those boxes, the definition found in this document shall be used for all JPL files.

• 1. Organization of the JPEG Pleno Light Field superbox

Figure A.1 shows the hierarchical organization of the JPEG Pleno Light Field superbox contained by a JPL file. This illustration does not specify nor imply a specific order to these boxes. In many cases, the file will contain several boxes of a particular box type. The meaning of each of those boxes is dependent on the placement and order of that particular box within the file.

This superbox is composed out of the following core elements:

— a JPEG Pleno Light Field Header box containing parameterization information about the light field such as size and colour parameters;

— a JPEG Pleno Light Field Reference View box containing the compressed reference views of the light field;

— a JPEG Pleno Light Field Disparity View box signalling disparity information for all or a subset of subaperture views;

— a JPEG Pleno Light Field Intermediate View box containing prediction parameters and eventual compressed residual signals for subaperture views not encoded as reference views.

Table A.1 lists all boxes defined as part of this document. Boxes defined as part of the ISO/IEC 15444-1 or ISO/IEC 15444-2 file formats are not listed. A box that is listed in Table A.1 as “Required” shall exist within all conforming JPL files. For the placement of and restrictions on each box, see the relevant section defining that box.

Note that the IPR, XML, and UUID boxes defined in Annex A can be signalled, as well at the level of the JPEG Pleno Light Field box, to carry light field specific metadata.

Figure A.1 — Hierarchical organization of a JPEG Pleno Light Field superbox

• 1. Defined boxes
     1. Overview

The following boxes should be interpreted properly by all conforming readers. Each of these boxes conforms to the standard box structure as defined in ISO/IEC 21794-1:2020, Annex A. The following clauses define the value of the DBox field. It is assumed that the LBox, TBox and XLBox fields exist for each box in the file as defined in ISO/IEC 21794-1:2020, Annex A.

Table A.1 — Defined boxes

• • 1. JPEG Pleno Profile and Level box

Profile and levels are defined in Annex G. The type of the JPEG Pleno Profile and Level box shall be ‘jppl’ (0x6A70686F). The contents of the box shall have the organization as in Figure A.2, and its format shall be as in Table A.2.

Key

Ppih profile of the codestream (as defined in Annex G)

Plev level of the codestream (as defined in Annex G)

Figure A.2 — Organization of the contents of a JPEG Pleno Profile and Level box

Table A.2 — Format of the contents of the JPEG Pleno Profile and Level box

• • 1. JPEG Pleno Light Field Header box
       1. General

The JPEG Pleno Header box contains generic information about the file, such as the number of components, bits per component and colour space. This box is a superbox. Within a JPL file, there shall be one and only one JPEG Pleno Header box. The JPEG Pleno Header box shall be located anywhere after the File Type box and before the Contiguous Codestream box. It also shall be at the same level as the JPEG Pleno Signature and File Type boxes. It shall not be inside any other superbox within the file.

The type of the JPEG Pleno Header box shall be 'jplh' (0x6A706C68).

This box contains several boxes. Other boxes may be defined in other documents and may be ignored by conforming readers. The boxes contained within the JPEG Pleno Header box that are defined within this document are shown in Figure A.3:

— The Light Field Header box specifies information about the reference grid geometry, bit depth and the number of components. This box shall be the first box in the JPEG Pleno Header box and is specified in A.3.3.2.

— The Bits Per Component box specifies the bit depth of the components in the file in cases where the bit depth is not constant across all components. Its structure shall be as specified in ISO/IEC 15444-1.

— The Colour Specification boxes specify the colour space of the decompressed image. Their structures shall be as specified in ISO/IEC 15444-2. There shall be at least one Colour Specification box within the JPEG Pleno Header box. The use of multiple Colour Specification boxes provides the ability for a decoder to be given multiple optimization or compatibility options for colour processing. These boxes shall be positioned anywhere in the JPEG Pleno Header box provided that they come after the Light Field Header box. All Colour Specification boxes shall be contiguous within the JPEG Pleno Header box.

— The Channel Definition box defines the channels in the image. Its structure shall be as specified in ISO/IEC 15444-1. This box shall be positioned anywhere in the JPEG Pleno Header box, provided that it comes after the Light Field Header box.

Key

lhdr Light Field Header box

bppc Bits Per Component box

colrⁱ Colour Specification boxes

cdef Channel Definition box

Figure A.3 — Organization of the contents of a JPEG Pleno Header box

• • • 1. Light Field Header box
         1. General

This box contains fixed length generic information about the light field, such as light field dimensions, subaperture image size, number of components, codec and bits per component. The contents of the JPEG Pleno Header box shall start with a Light Field Header box. Instances of this box in other places in the file shall be ignored. The length of the Light Field Header box shall be 30 bytes, including the box length and type fields. Much of the information within the Light Field Header box is redundant with information stored in the codestream itself.

All references to "the codestream" in the descriptions of fields in this Light Field Header box apply to the codestream found in the first Contiguous Codestream box in the file. Files that contain contradictory information between the Light Field Header box and the first codestream are not conforming files. However, readers may choose to attempt to read these files by using the values found within the codestream.

The type of the Light Field Header box shall be 'lhdr' (0x6C68 6472) and the contents of the box shall have the format as in Figure A.4 and Table A.3:

— ROWS (T): The value of this parameter indicates the number of rows of the subaperture view array. This field is stored as a 4-byte big-endian unsigned integer.

— COLUMNS (S): The value of this parameter indicates the number of columns of the subaperture view array. This field is stored as a 4-byte big-endian unsigned integer.

— HEIGHT (V): The value of this parameter indicates the height of the sample grid. This field is stored as a 4-byte big-endian unsigned integer.

— WIDTH (U): The value of this parameter indicates the width of the sample grid. This field is stored as a 4-byte big-endian unsigned integer.

— NC: This parameter specifies the number of components in the codestream and is stored as a 2-byte big-endian unsigned integer. The value of this field shall be equal to the value of the NC field in the LFC marker in the codestream (as defined in B.3.2.6.3). If no Channel Definition Box is available, the order of the components for colour images is R-G-B-Aux or Y-U-V-Aux.

— BPC: This parameter specifies the bit depth of the components in the codestream, minus 1, and is stored as a 1-byte field (Table A.4).

The low 7-bits of the value indicate the bit depth of the components. The MSB indicates whether the components are signed or unsigned. If the MSB is 1, then the components contain signed values. If the MSB is 0, then the components contain unsigned values. If the components vary in bit depth or sign, or both, then the value of this field shall be 255 and the Light Field Header box shall also contain a Bits Per Component box defining the bit depth of each component (as defined in A.3.3.2.2).

— C: This parameter specifies the compression algorithm used to compress the image data. It is encoded as a 1-byte unsigned integer. If the value is 0, the 4D transform mode coding is activated. If the value is 1, the 4D prediction mode is activated. If the value is 2, the Slanted 4D transform mode is activated. All other values are reserved for ISO/IEC use.

— UnkC: This field specifies if the actual colour space of the image data in the codestream is known. This field is encoded as a 1-byte unsigned integer. Legal values for this field are 0, if the colour space of the image is known and correctly specified in the Colour Space Specification boxes within the file, or 1 if the colour space of the light field is not known. A value of 1 will be used in cases such as the transcoding of legacy images where the actual colour space of the image data is not known. In these cases, while the colour space interpretation methods specified in the file may not accurately reproduce the image with respect to an original, the image should be treated as if the methods do accurately reproduce the image. Values other than 0 and 1 are reserved for ISO/IEC use.

— IPR: This parameter indicates whether this JPL file contains intellectual property rights information. If the value of this field is 0, this file does not contain rights information, and thus the file does not contain an IPR box. If the value is 1, then the file does contain rights information and thus does contain an IPR box as defined in ISO/IEC 15444-1. Other values are reserved for ISO/IEC use.

Key

Figure A.4 — Organization of the contents of a Light Field Header box

Table A.3 — Format of the contents of the Light Field Header box

Table A.4 — BPC values

• • • • 1. Bits Per Component box

The Bits Per Component box specifies the bit depth of each component. If the bit depth of all components in the codestream is the same (in both sign and precision), then this box shall not be present. Otherwise, this box specifies the bit depth of each individual component. The order of bit depth values in this box is the actual order in which those components are enumerated within the codestream. The exact location of this box within the JPEG Pleno Header box may vary provided that it follows the Light Field Header box.

There shall be one and only one Bits Per Component box inside a JPEG Pleno Header box.

The type of the Bits Per Component box shall be 'bpcc' (0x6270 6363). The contents of this box shall be as in Table A.5 and Figure A.5.

Key

BPCⁱ bits per component

Figure A.5 — Organization of the contents of a Bits Per Component box

Table A.5 — Format of the contents of the Bits Per Component box

This parameter specifies the bit depth of component i, minus 1, encoded as a 1‑byte value (Table A.6). The ordering of the components within the Bits Per Component box shall be the same as the ordering of the components within the codestream. The number of BPCⁱ fields shall be the same as the value of the NC field from the Light Field Header box. The value of this field shall be equivalent to the respective Ssizⁱ field in the LFC marker in the codestream. The low 7-bits of the value indicate the bit depth of this component. The MSB indicates whether the component is signed or unsigned. If the MSB is 1, then the component contains signed values. If the MSB is 0, then the component contains unsigned values.

Table A.6 — BPCⁱ values

• • • 1. Camera Parameter box
         1. General

The Camera Parameter box provides information on the positioning of the local reference grid in the global reference grid, its size, and calibration information about the light field. This box is optional.

Camera models can be represented by matrices with particular properties that represent the mapping of the 3D world coordinate system to the image coordinate system. This 3D to 2D transform depends on a number of parameters, known as the intrinsic and extrinsic parameters.

• • • • 1. JPEG Pleno camera parameters

As specified in ISO/IEC 21794-1, JPEG Pleno provides a mechanism to co-register plenoptic data contained by the JPL file on a 3D reference grid system. This reference grid system exists out of a global and a local reference grid. The global reference grid allows the positioning of the individual modalities in the represented 3D scene. In addition, each JPEG Pleno Light Field, Point Cloud and Hologram box shall be assigned a local reference grid to address their sampled plenoptic data. This local reference grid is specified by signalling its translation and rotation with respect to the global reference grid. The rotation angles shall be determined utilizing the right-hand rule for curve orientation.

The parameters related to the local reference grid are signalled per plenoptic object in the associated JPEG Pleno Light Field, Point Cloud or Hologram box.

Figure A.6 — The global and local reference grid

In Figure A.6, boundaries and coordinate axes of the global and one local reference grid are shown. In each case, the samples or coefficients coincident with the left and upper boundaries are included in a given bounding box, while samples or coefficients along the right and/or lower boundaries are not included in that bounding box.

• • • • 1. Camera modelling and calibration

General

The camera modelling and calibration is described based on the local reference grid (X_L, Y_L, Z_L) in Figure A.6. Both intrinsic and extrinsic parameters are being signalled to model and calibrate the camera setting and behaviour. The intrinsic parameters are the camera parameters that are internal and fixed to a particular camera/digitization setup, allowing the mapping between camera coordinates and pixel coordinates in the image plane^[1]. The extrinsic parameters are the camera parameters that are external to the camera and may change with respect to the 3D local reference grid, defining the location and orientation of the camera with respect to the 3D local reference grid coordinate system.

Intrinsic camera parameters

Considering a pinhole camera model, the centre of projection is the ‘optical centre’ (C in Figure A.7). The camera's ‘principal axis’ (Z_CAM in Figure A.7) is the line perpendicular to the image plane that passes through the pinhole. Its intersection with the image plane is known as the ‘principal point’ (p in Figure A.7) and it is the geometrical centre of the image. The parameters u₀ and v₀ are the principal points offsets, which are the coordinates of the principal point relative to the coordinate axes u and v (Figure A.7).

Figure A.7 — Pinhole camera geometry

The focal lengths and correspond to the distance between the optical centres of the cameras and their respective image planes. They are represented in terms of pixel dimensions in the u and v directions. For example, if the camera focal length is given in mm one needs to convert it to pixel dimensions using Formulae (A.1) and (A.2). For square pixels the sensor height is equal to the sensor width, and is equal to .

(pixel) = (focal length (mm) / sensor width (mm)) × image width (pixel) (A.1)

(pixel) = (focal length (mm) / sensor height (mm)) × image height (pixel) (A.2)

The axis skew parameter sk causes shear distortion in the projected image, and for most of the cameras its value is equal to zero. The parameters , ,, and completely characterize the mapping of an image point from camera to pixels coordinates. They are known as the intrinsic or internal parameters of a camera system and can be represented by the transformation matrix K in Formula (A.3):

(A.3)

The matrix K is known as the calibration matrix. In general, the mapping from 3D local reference grid to the image is linear. A camera system is said to be calibrated when its intrinsic parameters are known, otherwise it is an uncalibrated camera system.

Extrinsic camera parameters

The parameters that relate the camera orientation and position to a 3D local reference grid coordinate system are called the extrinsic or external parameters. The geometric quantities (rotational and translational components) describing the relative position and orientation of the cameras are called the extrinsic parameters of the camera system. The rotation and translation can be represented in an extrinsic matrix taking the form of a rigid transformation matrix: a 3×3 rotation matrix R, and a 3×1 translation column-vector t_M = (XCC, YCC, ZCC)^T that can be represented in a 3×4 matrix, as in Formula (A.4).

(A.4)

Matrix P (Formula (A.5)), known as the projection matrix, or camera matrix, represents the pose of the 3D local reference grid coordinates relative to the image coordinates. It contains 6 independent parameters (Degrees of Freedom - DoF): 3 for rotation and 3 for translation. These parameters are expressed in the local reference grid (Figure A.6).

(A.5)

The 3×4 matrix P relates the sensor plane 2D image coordinates u = (u, v) (= (u, v, 1) in homogeneous coordinates) to the 3D local reference grid coordinates X = (X_L, Y_L, Z_L) ( = (X_L, Y_L, Z_L, 1) in homogeneous coordinates) via Formula (A.5). The mapping between a point in the 3D world into a 2D image is given by =P, where is the image point in homogeneous coordinates, P is the camera matrix and is the 3D local reference grid point in homogeneous coordinates. The extrinsic camera parameter matrix represents the current status of the camera in the 3D scene.

For example, a rotation in 3D local reference grid space involves an axis around which to rotate, and an angle of rotation, according to the right-handed coordinates.

Formulae (A.6), (A.7) and (A.8) show the rotation matrix values for rotations around these 3 axes:

, rotation around the X_L axis, rotates Y_L, Z_L, leaving the X_L coordinates fixed (A.6)

, rotation around the Y_L axis, rotates X_L, Z_L, leaving the Y_L coordinates fixed (A.7)

, rotation around the Z_L axis, rotates X_L, Y_L, leaving the Z_L coordinates fixed. (A.8)

Any rotation can be expressed as a combination of the three rotations about the three axes, as per Formula (A.9):

(A.9)

Formula (A.10) shows the mapping of a 3-D local reference grid point (X_L, Y_L, Z_L) to the image coordinate system (u,v) , for a calibrated system:

(A.10)

• • • • 1. Camera Parameter box definition

The type of the Camera Parameter box shall be 'lfcp' (0x6C66 6370) and the contents of the box shall have the format as in Figure A.8.

If the Camera Parameter box is not signalled, all parameters specified in Figure A.8 and Table A.7 shall be initialized to zero. The scaling values S_GLX, S_GLY and S_GLZ will be initialized to 1.

Key

NOTE 1 PP indicates the floating-point precision issued for the coordinates.

NOTE 2 X_LO, Y_LO, Z_LO, , , , S_GLX, S_GLY, S_GLZ, (t, s) , (t, s) and (t, s) utilize the chosen floating-point precision.

Figure A.8 — Organization of the contents of the Camera Parameter box

The geometrical coordinates of the centre of the camera when acquiring the view (t, s) are denoted as

An example of camera centre coordinates for a planar camera array is given in Figure A.9, where both the horizontal and vertical coordinates are illustrated for five views in the camera array. The camera centres XCC and YCC are used together with the normalized disparity maps to obtain horizontal and vertical disparity maps between a pair of views in the light field. For usage examples see Annex D.4 and Annex E.4.

Figure A.9 — Organization of subaperture views and associated planar
camera calibration information

Table A.7 — Format of the contents of the Camera Parameter box

Table A.8 — Meaning of ExtInt bits

• • 1. Contiguous Codestream box

The Contiguous Codestream box contains a JPEG Pleno codestream.

The type of a Contiguous Codestream box shall be 'jp2c' (0x6A70 3263). The contents of the box shall be as in Figure A.10 and Table A.9:

Figure A.10 — Organization of the contents of the Contiguous Codestream box

Table A.9 — Format of the contents of the Contiguous Codestream box

1. 
   (normative)
   
   4D transform mode
   1. General

This annex describes an instantiation of 4D transform mode encoder. Next, the codestream syntax is specified and subsequently the light field decoding process is detailed.

• 1. 4D transform mode encoding
     1. High-level coding architecture

In the 4D transform mode,^[2] the light field is encoded with a four-step process (Figure B.1). First, the 4D light field data is divided into fixed-sized 4D blocks that are independently encoded according to a predefined and fixed scanning order. However, if any light field dimensions are not multiple of such fixed-sized 4D blocks, the sizes of the 4D blocks at the light field boundaries shall be truncated to fit in the light field dimensions. The initial blocks can be further partitioned into a set of non-overlapping 4D sub-blocks, where the optimal partitioning parameters are derived based on a rate-distortion (R-D) criterion. Each sub-block is independently transformed by a variable block-size 4D DCT. Subsequently, the transformed blocks are quantized and entropy coded using hexadeca-tree bit plane decomposition and adaptive arithmetic encoding, producing a compressed representation of the light field. This coding procedure is applied to each colour component independently.

Figure B.1 — 4D transform mode encoding architecture

A sample (pixel) of the light field is referenced in a 4D coordinate system along the t, s, v and u-axes, where t and s are representing the coordinates of the addressed subaperture view, and v and u the sample coordinates within the subaperture images as illustrated in Figure B.2. The blocks are scanned in the directions t, s, v and u, with direction u corresponding to the inner loop of the scan. In Table B.1 a pseudo-code describes the scan of the 4D blocks. Each 4D block will generate a separate codestream embedded in the codestream, which can be independently decoded in support of random access.

Figure B.2 — 4D structure

The 4D block partitioning, as well as the clustering of the bit-planes of transform coefficient – for efficient encoding – are signalled using tree structures (Figure B.3). The partitioning of the 4D blocks in sub-blocks is signalled with a binary tree using ternary flags. These flags signal whether:

— a block is transformed as is;

— is split into 4 blocks in the s,t (view) dimensions;

— is split into 4 blocks in the u,v (spatial) dimensions.

Key

Sn split node

Ln leaf node

Figure B.3 — Binary tree representing 4D-block partitioning of a t_k×s_k×v_k×u_k 4D block

Before subsequently applying the 4D-DCT on the sub-blocks, a level-shift operation is performed to reduce the dynamic range requirements of the DCT (Annex B.2.3.1). The deployed 4D DCT (Annex B.2.3.2) is separable, i.e. with 1D transforms computed separately in each of the 4 directions. An example of the computation flow of the 4D separable transform is depicted in Figure B.4. The order of the 1D transforms is arbitrary.

Figure B.4 — Separable forward 4D-DCT

After the 4D-DCT is performed, the set of transform coefficients is sliced into 4D bit-planes. A transform coefficient is considered non-significant on a 4D bit-plane if its bits belonging to higher 4D bit-planes are all zero. Otherwise, the transform coefficient is considered to be significant. A hexadeca-tree with ternary flags is used to group the non-significant transform coefficients and thus localize the significant transform coefficients (Figure B.5). The ternary flags signal that:

— either a block of transform coefficients containing a significant coefficient at the current bit-plane is split into 16 blocks in the four t, s, v, u dimensions,

— or a block of transform coefficients not containing any significant coefficient at the current bit-plane is not split,

— or a block of transform coefficients containing a significant coefficient at the current bit-plane is discarded.

The 4D bit-planes are scanned from the most significant bit-plane to the least significant one, where the least significant 4D bit-plane being determined by the desired quantization level. Both the hexadeca-tree bits and the bits from the transform coefficients are encoded using an adaptive arithmetic coder.

Key

Sn split node

Ln leaf node

Figure B.5 — Hexadeca-tree representing the clustering of bit-planes of 4D transform coefficients of a t_b×s_b×v_b×u_b 4D block

The structure that clusters the non-significant 4D transform coefficients and thus localizes the significant ones is a hexadeca-tree, that is encoded using ternary flags. They signal that a block of transform coefficients containing a significant coefficient at the current bit-plane is split into 16 blocks in the four t, s, v, u dimensions, or that a block of transform coefficients not containing any significant coefficient at the current bit-plane is not split, or that a block of transform coefficients containing a significant coefficient at the current bit-plane is discarded. The 4D bit-planes are scanned from the most significant to the least significant one, where the least significant 4D bit-plane being determined by the desired quantization level. Both the hexadeca-tree bits and the bits from the coefficients are encoded using an adaptive arithmetic coder. The hexadeca-tree structure mentioned above is depicted in Figure B.5.

The following two sections provide more insight on the partitioning strategy and the entropy and quantization steps.

• • 1. Partitioning optimization

The partitioning optimization for each block in Figure B.6 is obtained as follows: initially each block is transformed by a full-size DCT (B.2.3.2), and the Lagrangian encoding cost, J₀ is evaluated (Table B.5). This cost is defined as J₀ = D + λ R, where D is the distortion incurred when representing the original block by the quantized version and R is the rate needed to encode it. The procedure of Table B.5 is used to evaluate this cost.

Figure B.6 — 4D Block of a light field

Next, the block can be partitioned in four sub-blocks each one with approximately a quarter of the pixels in the spatial dimensions. For example, let us consider a block B of dimensions t_k×s_k×v_k×u_k. This block (pictured in Figure B.7 and Figure B.8) will be subdivided in four sub-blocks of sizes t_k×s_k×⎿v_k/2⏌×⎿u_k/2⏌, t_k×s_k×⎿v_k/2⏌×(u_k-⎿u_k/2⏌), t_k×s_k×(v_k-⎿v_k/2⏌)×⎿u_k/2⏌ and t_k×s_k×(v_k-⎿v_k/2⏌)×(u_k-⎿u_k/2⏌), respectively.

Figure B.8 shows a 4D block with dimensions t_k×s_k×v_k×u_k in the root node. When applying the spatial split (signalled with the spatialSplit flag) to the root node, the tree in Figure B.8 is obtained. Figure B.7 illustrates the four ways that a single view is partitioned using the spatialSplit flag, corresponding to four nodes of Figure B.8. The optimal partition for each sub-block is computed by means of the recursive procedure described in Table B.5 and the Lagrangian costs of the four sub-blocks are added to compute the Lagrangian cost J_S (Spatial R-D cost).

Figure B.7 — Spatial partitioning of block with spatial dimensions v_k×u_k

Key

<graphic>21794-</graphic> node

<graphic>21794-</graphic> spatialSplit flag

Figure B.8 — Hierarchical 4D partitioning of a 4D block with dimensions t_k×s_k×v_k×u_k using the spatialSplit flag

The block can be partitioned in four sub-blocks each one with approximately a quarter of the pixels in the view dimensions. For example, let us consider again a block B of dimensions t_k×s_k×v_k×u_k. This block (pictured in Figure B.9 and Figure B.10) will be subdivided in four sub-blocks of sizes ⎿t_k/2⏌×⎿s_k/2⏌×v_k×u_k ⎿t_k/2⏌×(s_k-⎿s_k/2⏌)×v_k×u_k, (t_k-⎿t_k/2⏌)×⎿s_k/2⏌×v_k×u_k, (t_k-⎿t_k/2⏌)×(s_k-⎿s_k/2⏌)×v_k×u_k, respectively Figure B.10). When applying the view split (signalled with the viewSplit flag) to the root node, the tree in Figure B.10 is obtained. Figure B.9 illustrates the four ways that a 4D block is partitioned using the viewlSplit flag, corresponding to four the nodes of Figure B.10. The optimal partition for each sub-block is computed by means of the recursive procedure described in Table B.5 and the Lagrangian costs of the four sub-blocks are added to compute the Lagrangian cost J_V (View R-D cost).

Figure B.9 — View partitioning of a 4D block of dimensions t_k×s_k×v_k×u_k

Key

<graphic>21794-</graphic> node

<graphic>21794-</graphic> viewSplit flag

Figure B.10 — Hierarchical 4D partitioning of a 4D block with dimensions t_k×s_k×v_k×u_k , using the viewSplit flag

Finally, the three Lagrangian costs (J_0, J_S and J_V) are compared (Table B.5) and the one presenting the lowest value is chosen.

The recursive partition procedure (Table B.5) keeps track of this tree (Table B.11) and returns a partitionString (Table B.5) that represents the optimal tree. The string is obtained as follows: once the lowest cost is chosen, the current value of partitionString is augmented by appending to it the flag corresponding to the lowest cost chosen. Then, the string returned by the recursive call that leads to the minimum cost is also appended to the end of the partitionString and the procedure returns both the minimum cost J₀ , J_S or J_V and the updated partitionString.

Key

<graphic>21794-</graphic> node

<graphic>21794-</graphic> spatialSplit flag

<graphic>21794-</graphic> viewSplit flag

<graphic>21794-</graphic> transform flag

NOTE The transform flag signals that the node is a leaf node and will be no further partitioned.

Figure B.11 — Hierarchical 4D partitioning using the viewSplit flag and the spatialSplit flag

After the optimal partition tree is found, the Encode Partition procedure (Table B.6) is called to encode it.

• • 1. Forward 4D-DCT
       1. Level shift

Subsequently, the subblocks are subject to a DCT. However, before processing the forward DCT for a block of source light field samples, if the samples of the component are unsigned, those samples shall be level shifted to a signed representation. if the MSB of Ssizⁱ from the LFC marker segment (see Annex B.3.2.6.3) is zero, all samples x(u,v,s,t) of the ith component are level shifted by subtracting the same quantity from each sample as follows:

• • • 1. Forward 4D-DCT function

First all components have to be converted to the same precision (bit-depth). Each colour component c is by multiplied by before the forward 4D-DCT.

For a given light field x(u,v,s,t) the corresponding 4D-DCT representation X(i,j,p,q) is computed by transforming the block along each dimension as follows.

where and , where N is the size of the transform. Output coefficients are represented as 32-bit integers. As indicated earlier, the transform order is arbitrary.

• • 1. Quantization and entropy encoding

The quantization and entropy encoding rely on the R-D optimized hexadeca-tree structure, which is constructed based on the procedure listed in Table B.7 and which is further discussed in this paragraph. This tree is uniquely represented by a series of ternary flags: lowerBitplane, splitBlock and zeroBlock (Table B.44). The hexadeca-tree is built by recursively subdividing a 4D block until all sub-blocks reach a 1×1×1×1 4D block-size. The hexadeca-tree is built by recursively subdividing a 4D block. Starting from a 4D block of size t_b×s_b×v_b×u_b, and a bit-plane initially set to maxBitplane (Table B.11), 3 operations are performed:

The procedure described in Table B.7 recursively subdivides the input block, as determined by the ternary flags in the segmentation string until a 1×1×1×1 4D block-size is reached.

Given a particular hexadeca-tree, specified by a unique segmentationString of ternary segmentation flags together with a particular block, the data can be encoded by means of the recursive procedure described in Table B.8. The inputs to this procedure are the transformed block to be encoded and the optimal partition string. It recursively subdivides the input block, as determined by the ternary flags in the segmentation string. Then the magnitude of the single coefficient of this block is encoded one bit at a time, ranging from the current bit plane to the minimumBitPlane (Table B.8), using an arithmetic encoder with a different context information for each bit. If the coefficient is not zero valued, its signal is encoded as well.

The rate and the distortion achieved depend heavily on the choice of the segmentation tree as well as the data itself, and those should be matched. The procedure described in Table B.7, recursively chooses the optimal segmentation tree to use in the encoding of a given block in a rate-distortion sense. The optimization works as follows: it starts with bitplane=maximumBitplane, segmentationString=null, variables J₀ and J₁ both set to infinity and the full transformed input block. The transformed block is scanned and all its coefficients are compared to a threshold given by 2^bitplane. If the magnitudes of all of them are less than the threshold, the optimization procedure is recursively called with the same block as input, but with a bitplane value decreased by one (bitplane-1). The values returned by this recursive call are the new Lagrangian cost J₀, and a rate-distortion optimized segmentation string lowerSegmentationString. However, if any coefficient is above the threshold, the transformed block is segmented into up to 16 sub-blocks as previously described. The optimization procedure is called recursively for each sub-block and the returned Lagrangian costs are added to obtain the new Lagrangian cost J₁. The segmentation strings returned from these calls are concatenated to form the splitSegmentationString.

Another Lagrangian cost J₂ is evaluated considering the resulting cost if the block was replaced by a block entirely composed of zeros. The lowest cost is chosen, and the segmentation string is updated as follows:

— If the minimum cost is J₀, the input segmentation string is augmented by appending a flag lowerBitplane followed by the lowerSegmentationString.

— If the minimum cost is J₁, the input segmentation string is augmented by appending a flag splitBlock followed by the splitSegmentationString.

— If the minimum cost is J₂, the input segmentation string is augmented by appending a flag zeroBlock.

The procedure returns the lowest cost and the resulting associated segmentation string.

The 4D coefficients, flags, and probability context information generated during the encoding process, are input to the arithmetic encoder, that generates the compressed representation of the light field (Table B.4). The adaptive statistical binary arithmetic coding is detailed in Table B.12, Table B.13 and Table B.14. The arithmetic coding requires transmitting only the information needed to allow a decoder to determine the particular fractional interval between 0 and 1 to which the sequence is mapped, adapting to changing statistics in the codestream.

• • 1. Sample encoding procedure

In this subclause, a sample encoding algorithm is provided for informative purposes. The main procedure processing the individual 4D blocks is listed in Table B.1.

Table B.1 — 4D block scan procedure for 4D-Transform mode

Table B.2 — 4D block padding procedure

Table B.3 — 4D-DCT block coefficient component scaling procedure

Table B.4 — 4D block encoding procedure for 4D-Transform mode

Table B.5 — 4D block partition optimization procedure

Table B.6 — 4D block encode partition procedure

Table B.7 — 4D block hexadeca-tree optimization procedure

Table B.8 — 4D block hexadeca-tree encoding procedure

Table B.9 — 4D block hexadeca-tree encode coefficient procedure

Table B.10 — 4D-block hexadeca-tree encode minimum bit-plane procedure

Table B.11— 4D block optimal bit-plane procedure

Table B.12 —Encode bit procedure

Table B.13 —Init encoder procedure

Table B.14 —Flush encoder procedure

• 1. 4D transform mode decoding
     1. General

In this subclause, the decoding procedure of the codestreams for light field data encoded with the 4D Transform mode is specified. The codestream is signalled as payload of the Contiguous Codestream box defined in Annex A.3.4. Figure B.12 illustrates the decoder architecture.

Figure B.12 — 4D transform mode light field decoder architecture

• • 1. Codestream syntax
       1. General

This section specifies the marker and marker segment syntax and semantics defined by this document. These markers and marker segments provide codestream information for this document. Further, this subclause provides a marker and marker segment syntax that is designed to be used in future specifications that include this document as a normative reference.

This document does not include a definition of conformance. The parameter values of the syntax described in this annex are not intended to portray the capabilities required to be compliant.

• • • 1. Markers, marker segments, and headers

This document uses markers and marker segments to delimit and signal the characteristics of the source image and codestream. This set of markers and marker segments is the minimal information needed to achieve the features of this document and is not a file format. A minimal file format is offered in Annexes A and B.

Main header is collections of markers and marker segments. The main header is found at the beginning of the codestream.

Every marker is two bytes long. The first byte consists of a single 0xFF byte. The second byte denotes the specific marker and can have any value in the range 0x01 to 0xFE. Many of these markers are already used in ITU-T Rec. T.81 | ISO/IEC 10918-1, ITU-T Rec. T.84 | ISO/IEC 10918-3, ITU-T Rec. T.800 | ISO/IEC 15444-1 and ITU-T Rec. T.801 | ISO/IEC 15444-2 and shall be regarded as reserved unless specifically used.

A marker segment includes a marker and associated parameters, called marker segment parameters. In every marker segment the first two bytes after the marker shall be an unsigned value that denotes the length in bytes of the marker segment parameters (including the two bytes of this length parameter but not the two bytes of the marker itself). When a marker segment that is not specified in the document appears in a codestream, the decoder shall use the length parameter to discard the marker segment.

• • • 1. Key to graphical descriptions

Each marker segment is described in terms of its function, usage, and length. The function describes the information contained in the marker segment. The usage describes the logical location and frequency of this marker segment in the codestream. The length describes which parameters determine the length of the marker segment.

These descriptions are followed by a figure that shows the order and relationship of the parameters in the marker segment. Figure B.13 shows an example of this type of figure. The marker segments are designated by the three-letter code of the marker associated with the marker segment. The parameter symbols have capital letter designations followed by the marker's symbol in lower-case letters. A rectangle is used to indicate a parameter's location in the marker segment. The width of the rectangle is proportional to the number of bytes of the parameter. A shaded rectangle (diagonal stripes) indicates that the parameter is of varying size. Two parameters with superscripts and a grey area between indicate a run of several of these parameters.

Figure B.13 — Example of the marker segment description figures

The figure is followed by a list that describes the meaning of each parameter in the marker segment. If parameters are repeated, the length and nature of the run of parameters is defined. As an example, in Figure B.13, the first rectangle represents the marker with the symbol MAR. The second rectangle represents the size of the length parameter SLmar (Table B.15). The third rectangle represents the length parameter Lmar. Parameters Amar, Bmar, Cmar, and Dmar are 8-, 16-, 32-bit and variable length respectively. The notation Emarⁱ implies that there are n different parameters, Emarⁱ, in a row.

After the list is a table that either describes the allowed parameter values or provides references to other tables that describe these values. Tables for individual parameters are provided to describe any parameter without a simple numerical value. In some cases, these parameters are described by a bit value in a bit field. In this case, an "x" is used to denote bits that are not included in the specification of the parameter or sub-parameter in the corresponding row of the table.

Table B.15 — Size parameters for the SLlfc, SLscc and SLpnt

• • • 1. Defined marker segments

Table B.16 lists the markers specified in this document.

Table B.16 — List of defined marker segments

• • • 1. Construction of the codestream

Figure B.14 shows the construction of the codestream. All of the solid lines show required marker segments. The following markers and marker segments are required to be in a specific location: SOC, LFC, PNT, SOB and EOC. The dashed lines show optional or possibly not required marker segments.

Figure B.14 — Codestream structure

• • • 1. Delimiting markers and marker segments
         1. General

The delimiting marker and marker segments shall be present in all codestreams conforming to this document. Each codestream has only one SOC marker, one EOC marker, and contains at least one 4D block. Each 4D block has one SOB and one EOB marker. The SOC, SOB, and EOC are delimiting markers, not marker segments, and have no explicit length information or other parameters.

• • • • 1. Start of codestream (SOC)

Function: Marks the beginning of a codestream specified in this document (Table B.17).

Usage: Main header. This is the first marker in the codestream. There shall be only one SOC per codestream.

Length: Fixed.

Table B.17 — Start of codestream parameter values

• • • • 1. Light field configuration (LFC)

Function: Provides information about the uncompressed light field such as the width and height of the subaperture views, number of subaperture views in rows and columns, number of components, component bit depth, number of 4D blocks and size of the 4D blocks, component bit depth for transform coefficients (Figure B.15):

— Llfc: The value of this parameter is determined as .

— ROWS (T): The value of this parameter indicates the number of rows of the subaperture view array. This field is stored as a 4-byte big-endian unsigned integer.

— COLUMNS (S): The value of this parameter indicates the number of columns of the subaperture view array. This field is stored as a 4-byte big-endian unsigned integer.

— HEIGHT (V): The value of this parameter indicates the height of the sample grid. This field is stored as a 4-byte big-endian unsigned integer.

— WIDTH (U): The value of this parameter indicates the width of the sample grid. This field is stored as a 4-byte big-endian unsigned integer.

— NC: This parameter specifies the number of components in the codestream and is stored as a 2-byte big-endian unsigned integer. The value of this field shall be equal to the value of the NC field in the Light Field Header box. If no Channel Definition Box is available, the order of the components for colour images is R-G-B-Aux or Y-U-V-Aux.

— Ssizi: The precision is the precision of the component samples before DC level shifting is performed (i.e. the precision of the original component samples before any processing is performed). If the component sample values are signed, then the range of component sample values is –2(Ssiz+1 AND 0x7F)–1 ≤ component sample value ≤ 2(Ssiz+1 AND 0x7F)–1 – 1. There is one occurrence of this parameter for each component. The order corresponds to the component's index, starting with zero.

— TRNC: If unset (TRNC=0), all 4D blocks will have initially the same fixed-sizes and a padding procedure is applied.

Usage: Main header. There shall be one and only one in the main header immediately after the SOC marker segment. There shall be only one LFC per codestream.

Length: Fixed.

Key

Figure B.15 — Light field configuration syntax

Table B.18 — Format of the contents of configuration parameter set for the 4D coding

Table B.19 — Component Ssiz parameter

• • • • 1. Colour component scaling (SCC)

Function: Describes the global scaling factor used for a colour component (Table B.20). If this marker segment is not signalled for a specific colour component, the value of the global scaling factor for this colour component shall be 1.

Usage: Main header. No more than one per any given component may be present. Optional.

Length: Variable depending on the number of colour components.

Table B.20 — Colour component scale parameter values

Table B.21 — Quantization values for the Spscc parameter

• • • • 1. Codestream pointer set (PNT)

Function: Provides pointers to the codestream associated each 4D block to facilitate efficient access (Table B.22).

Usage: Optional. If present, must be included in the main header between the LFC marker segment and the first SOB marker segment defined in this document. Only one PNT marker segment shall be embedded in the main header (Figure B.16).

Length: Variable.

Key

PNT marker code

SLpnt size of Lpnt parameter

Lpnt length of marker segment in bytes (not including the marker)

Spnt size of the PPnt parameter

PPnt^i,c pointer to codestream of 4D block and colour component c

NOTE 1 The value of the Lpnt parameter is determined as follows:

where N_4D is defined in the LFC marker segment.

NOTE 2 The PPnt^i,c pointer indicates the position of the addressed 4D block codestream – its SOB marker – in the Contiguous Codestream box counting from the beginning of this box, i.e. the LBox field.

Figure B.16 — Codestream pointer set syntax

Table B.22 — Format of the contents of the codestream pointer set for the 4D coding

Table B.23 — Size parameters for Spnt

• • • • 1. Start of block (SOB)

Function: Marks the beginning of a 4D block (Table B.24).

Usage: Every 4D block header. Shall be the first marker segment in a 4D Block header. There shall be at least one SOB in a codestream. There shall be only one SOB per 4D block.

Length: Fixed.

Table B.24 — Format of the contents of the start of block

• • • • 1. End of codestream (EOC)

Function: Indicates the end of the codestream (Table B.25).

NOTE 1 This marker shares the same code as the EOI marker in ITU-T Rec. T.81 | ISO/IEC 10918-1 and the EOC marker in ITU-T Rec. T.800 | ISO/IEC 15444-1.

Usage: Shall be the last marker in a codestream. There shall be one EOC per codestream.

NOTE 2 In the case a file has been corrupted, it is possible that a decoder could extract much useful compressed image data without encountering an EOC marker.

Length: Fixed.

Table B.25 — Format of the contents of end of codestream

• • 1. Codestream parsing

The procedure to parse and decode a light field codestream as contained by the Contiguous Codestream box (Annex A.3.4), with dimensions (max_t, max_s, max_v, max_u) defined as ROW, COLUMN, HEIGHT and WIDTH in Figure B.15 (Table B.18), with block dimensions (t_k, s_k, v_k, u_k) defined as BLOCK-SIZE_t, BLOCK-SIZE_s, BLOCK-SIZE_v and BLOCK-SIZE_u in Figure B.15 (Table B.18), with a maximum number of bit-planes of max_bitplane (defined in Figure B.15 (Table B.18)), is described in the pseudo-code (all variables are integers). The scan order is in the order t, s, v, u, as described in Table B.26.

The coordinate set (t, s, v, u) refers to the left, superior corner of the 4D block. The procedure “ResetArithmeticDecoder()” resets all the context model probabilities of the arithmetic decoder. The procedure “LocateContiguousCodestream” reads the pointer corresponding to position (t,s,v,u) and delivers the respective codestream to procedure “DecodeContiguous Codestream()”. The procedure “DecodeContiguousCodestream()” decodes the components of the block contained in the codestream enabling the sequential decoding of light field. The codestreams of the components are decoded sequentially.

Table B.26 — JPEG Pleno (JPL) codestream structure for the 4D transform mode

• • 1. 4D partitioning general structure
       1. General

The datasets (all texture views in Figure B.1) are composed by 4D light fields of dimensions t×s×v×u. The views are addressed by the s,t coordinates pair, while the u,v pair addresses a pixel within each s,t view, as pictured in Figure B.6.

• • • 1. Partition tree decoding

The root node of the tree corresponds to a full t×s×v×u transform. The partition tree is represented by a series of ternary flags:

NOTE The transform flag signals that the node is a leaf node and will be no further partitioned.

— Figure B.11 shows six nodes marked by a transform flag, two nodes split (segmented) accordingly the spatialSplit and the viewSplit flags, and the 9^th node, (⎿t_k/2⏌×(s_k-⎿s_k/2⏌)×⎿v_k/2⏌×⎿u_k/2⏌, which can be further segmented using either the spatialSplit flag or the viewSplit flag.

Figure B.17 depicts the result of a 4D block with dimensions of 9×9×434×625 (t_k×s_k×v_k×u_k) partitioned, using the spatialSplit flag, into a sub-block with dimensions of 9×9×217×312, in grey, (t_k×s_k×(v_k −⎿v_k /2⏌)×⎿u_k/2⏌). Please note that, in Figure B.17 the partitioned v and u dimensions are: 217 = 434 −⎿434/2⏌ (v_k −⎿v_k /2⏌) and 312 =⎿625/2⏌ (⎿u_k/2⏌).

Figure B.18 depicts the result of a 4D block with dimensions of 9×9×434×625 (t_k×s_k×v_k×u_k) partitioned, using the viewSplit flag, into a sub-block with dimensions of 5×4×434×625, in grey, ((t_k−⎿t_k/2⏌)×⎿s_k/2⏌×v_k×u_k). Please note that, the partitioned t and s dimensions are: (t_k−⎿t_k/2⏌) and 4 =⎿9/2⏌ (⎿s_k/2⏌).

Figure B.17 — Example of a 4D block with dimensions t_k×s_k×(v_k −)× (in grey) superimposed to a 4D block with dimensions t_k×s_k×v_k×u_k

Figure B.18 — Example of a 4D block with dimensions (t_k )× ×v_k×u_k (in grey) superimposed on a 4D block with dimensions t_k×s_k×v_k×u_k

• • • 1. Contiguous codestream of a 4D-block

For a contiguous codestream of a 4D-block with size tk×sk×vk×uk, a 4D-block partitioning decoding procedure is performed (DecodeContiguousCodestream), that uses the recursive procedure “Procedure DecodePartitionStep”, both defined in Table B.27 and Table B.28.

Table B.27 — Decode contiguous codestream procedure

Table B.28 — 4D-DCT block coefficient component inverse scaling

Table B.29 — Decode partition step procedure

Table B.30 — Lists of partition flags representations

• • 1. Arithmetic decoding procedure
       1. General

Figure B.19 shows a simple block diagram of a binary adaptive arithmetic decoder. The compressed light field data cd and a context cx from the decoder's model unit (not shown) are input to the arithmetic decoder. The arithmetic decoder's output is the decision d. The encoder and decoder model units need to supply exactly the same context cx for each given decision. The decoding process should be initialized. The contexts (cx) and bytes of compressed light field data (as needed) are read and passed on to the decoder until all contexts have been read. The decoding process computes the binary decision d and returns a value of either 0 or 1. The estimation procedures for the probability models, which provide adaptive estimates of the probability for each context are part of the decoding process.

Figure B.19 — Arithmetic decoder inputs and output

• • • 1. Probability models

The contexts of the arithmetic coder are defined in Table B.31. When the adaptive flag is Off the probability model is fixed.

Table B.31 — Arithmetic coder contexts

• • • 1. Procedures to read symbols

Each call to the arithmetic decoder to read a variable is performed according to the following syntax:

symbol=read(Context, AdaptiveFlag);

Each call to read a symbol from stream, may update arithmetic decoder models if the adaptive flag is On. The procedure read(Context, AdaptiveFlag) is defined in Table B.32 and the variable Context is defined in Table B.31.

Table B.32 — Read bit using arithmetic decoder procedure

Table B.33 — Read magnitude procedure

Table B.34 — Read minimum bitplane procedure

If a DCT coefficient is different from zero its sign must be read from the stream with the procedure in Table B.35.

Table B.35 — Read sign procedure

Every ternary hexadecatree flag shall be read using the procedure in Table B.36:

Table B.36 — Read hexadeca-tree flag procedure

Every ternary partition flag must be read using the procedure in Table B.37.

Table B.37 — Read partition tree flag procedure

• • • 1. Arithmetic decoder procedures and definitions

The arithmetic decoder employs probabilistic models to decode symbols from the codestream.

Definitions and procedures to reset the arithmetic decoder, to update probabilistic models and read information from the stream are detailed in this section.

The procedures use two vectors (acumFreq_0 and acumFreq_1), defined with length MAX_NUMBER_OF_MODELS.

The codec uses 99 models numbered from 0 to 98 (i.e. MAX_NUMBER_OF_MODELS=98).

At initialization the decoder limits are set and first bits are read from the stream as specified in the procedure in Table B.38.

Table B.38 — Initialize decoder procedure

A probabilistic model is initialized executing the procedure in Table B.39.

Table B.39 — Initialize probabilistic model procedure

The arithmetic decoder is reset, for every model, by calling the “InitDecoder” procedure (Table B.38), followed by the “InitProbabilistModel” (Table B.39 and Table B.40).

Table B.40 — Reset arithmetic decoder procedure

The procedure to decode a bit from the stream is described in Table B.41.

Table B.41 — Decode bit procedure