sample model linear access prediction mode for video coding

专利摘要:
The techniques are described in the prediction mode by linear model (lm). In one or a few instances, a video encoder or video decoder may limit the number of neighboring samples that are fetched for subsampling, thereby increasing the speed at which the video encoder or video decoder is able to complete encoding. or prediction decoding by lm.
公开号:BR112019011883A2
申请号:R112019011883
申请日:2017-12-19
公开日:2019-10-22
发明作者:Chen Jianle；Zhang Kai；Zhang Li；Karczewicz Marta
申请人:Qualcomm Inc；
IPC主号:

专利说明:

MODE OF PREDICTION BY LINEAR MODEL WITH SAMPLE ACCESS FOR VIDEO CODING [001] This application claims the benefit of provisional US patent application No. 62 / 436,319 filed on December 19, 2016, the entire content of which is incorporated herein through reference title.
TECHNICAL FIELD [002] This disclosure refers to video encoding and decoding.
FUNDAMENTALS [003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct transmission systems, wireless transmission systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, recording devices, digital media players, video game devices, video game consoles, cell phones or satellite radio phones, called smartphones, video teleconferencing devices, video streaming devices and similar. Digital video devices implement video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Video encoding Advanced Video (AVC), the High Efficiency Video Coding (HEVC) standard currently under development, and extensions to such standards. Video devices can transmit, receive, encode, decode and / or store
Petition 870190053680, of 12/06/2019, p. 6/171
2/119 digital video information more efficiently by implementing such video compression techniques.
[004] Video compression techniques perform spatial (intra-image) and / or temporal (inter-image) prediction to reduce or remove the redundancy inherent in video sequences. For block-based video encoding, a video slice (that is, a video frame or a portion of a video frame) can be partitioned into video blocks. The video blocks in an intra-coded slice (I) of an image are encoded using spatial prediction in relation to reference samples in neighboring blocks in the same image. Video blocks in an intercoded image slice (P or B) can use spatial prediction in relation to reference samples in neighboring blocks in the same image or temporal prediction in relation to reference samples in other reference images. Spatial or temporal prediction results in a predictive block for a block that will be encoded. The residual data represents pixel differences between the original block that will be encoded and the predictive block. An intercodified block is coded according to a motion vector that points to a reference sample block that forms the predictive block, and the residual data indicates the difference between the coded block and the predictive block. An intra-encoded block is encoded according to an intra-encoding mode and residual data. For additional compression, residual data can be transformed from the pixel domain to a transform domain, resulting in
Petition 870190053680, of 12/06/2019, p. 7/171
3/119 residual coefficients, which can then be quantized.
SUMMARY [005] In general, aspects of disclosure refer to techniques for subsampling neighboring luma samples for linear model prediction (LM). As described in more detail, a video encoder or video decoder can be configured to search for neighboring sample samples as part of the subsampling to build a chroma prediction block in LM prediction mode. In some examples, the video encoder or video decoder can be configured not to seek certain neighboring luma samples as part of the subsampling to construct the chroma prediction block in LM prediction mode. By not searching for certain neighboring luma samples, the examples of techniques can promote efficient processing and use of memory bandwidth.
[006] In one example, the disclosure describes a method of decoding video data, the method comprising determining a luma block that corresponds to a chroma block, searching for neighboring luma samples for the subsampling of neighboring luma samples , in which the neighboring luma samples sought comprise a plurality of luma samples that are above the luma block and exclude the luma samples that are above and to the left of an upper left luma sample of the luma block, determining a plurality of luma samples subsampled based on the samples of
Petition 870190053680, of 12/06/2019, p. 8/171
4/119 neighboring luma searched, in which one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left luma sample, determine one or more scheduling parameters based on the subsampled luma samples, determine a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) that decodes the chroma block based on the predictive block.
[007] In one example, the disclosure describes a device for decoding video data, the device comprising a video data memory, and a video decoder that comprises at least one of the set of fixed function circuits or set programmable circuits. The video decoder is configured to determine a block of luma that corresponds to a chroma block, to search, from the video data memory, for samples of neighboring luma for the subsampling of the samples of neighboring luma, in which the samples of luma searched neighbors comprise a plurality of luma samples that are above the luma block and exclude the luma samples that are above and to the left of an upper left luma sample from the luma block, determining a plurality of subsampled luma samples based on in the neighboring luma samples searched, where one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left luma sample, determine one or more scheduling parameters based on
Petition 870190053680, of 12/06/2019, p. 9/171
5/119 in the subsampled luma samples, determine a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) decodes the chroma block based on the predictive block.
[008] In one example, the disclosure describes a method of encoding video data, the method comprising determining a luma block that corresponds to a chroma block, searching for neighboring luma samples for the subsampling of neighboring luma samples , in which the neighboring luma samples sought comprise a plurality of luma samples that are above the luma block and exclude the luma samples that are above and to the left of an upper left luma sample of the luma block, determining a plurality of subsampled luma samples based on the neighboring luma samples searched, where one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left luma sample, determine one or more escalation parameters based on the samples of sub-samples, determine a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) that c odifies the chroma block based on the predictive block.
[009] In one example, the disclosure describes a device for encoding video data, the device comprising a video data memory, and a video encoder comprising at least one of the set of fixed function circuits or set programmable circuits. The video encoder is configured
Petition 870190053680, of 12/06/2019, p. 10/171
6/119 to determine a luma block that corresponds to a chroma block, to search, from the video data memory, for neighboring luma samples for the sub-sampling of neighboring luma samples, in which the searched luma samples comprise a plurality of luma samples that are above the luma block and exclude the luma samples that are above and to the left of an upper left luma sample from the luma block, determine a plurality of subsampled luma samples based on the samples of determined neighboring luma, where one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left luma sample, determine one or more scheduling parameters based on the subsampled luma samples, determine a predictive block based on one or more scheduling parameters, and linear model prediction (LM) encodes the chroma block based on the predictive block.
[0010] In one example, the disclosure describes a method of decoding video data, the method comprising determining a luma block that corresponds to a chroma block, searching for neighboring luma samples for the subsampling of neighboring luma samples , in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are more than a limit number of samples to the left of the luma block and below one upper left luma sample from the luma block, determine a plurality of subsampled luma samples based on
Petition 870190053680, of 12/06/2019, p. 11/171
7/119 neighboring luma searched, where one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block, determine one or more scheduling parameters based on the subsampled luma samples , determine a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) that decodes the chroma block based on the predictive block.
[0011] In one example, the disclosure describes a device for decoding video data, the device comprising a video data memory, and a video decoder comprising at least one of the set of fixed function circuits or set programmable circuits. The video decoder is configured to determine a luma block that corresponds to a chroma block, to search for neighboring luma samples for the subsampling of neighboring luma samples, in which the searched neighboring luma samples comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are more than a limit number of samples to the left of the luma block and below an upper left luma sample to the luma block, determine a plurality of luma samples subsampled based on the neighboring luma samples searched, where one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block, determine one or more parameters of
Petition 870190053680, of 12/06/2019, p. 12/171
8/119 scheduling based on the subsampled luma samples, determining a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) decodes the chroma block based on the predictive block.
[0012] In one example, the disclosure describes a method of encoding video data, the method comprising determining a luma block that corresponds to a chroma block, searching for neighboring luma samples for the subsampling of neighboring luma samples , in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are more than a limit number of samples to the left of the luma block and below one upper left luma sample of the luma block, determine a plurality of subsampled luma samples based on the neighboring luma samples searched, in which one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to left of the luma block, determine one or more scheduling parameters based on the subsampled luma samples, determine a predictive block based on one or more s scheduling parameters, and prediction by linear model (LM) that encodes the chroma block based on the predictive block.
[0013] In one example, the disclosure describes a device for encoding video data, the device comprising a video data memory, and a video encoder comprising at least one of the
Petition 870190053680, of 12/06/2019, p. 13/171
9/119 fixed function circuit pack or programmable circuit pack. The video encoder is configured to determine a luma block that corresponds to a chroma block, to search for neighboring luma samples for the subsampling of neighboring luma samples, in which the searched neighboring luma samples comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are more than a limit number of samples to the left of the luma block and below an upper left luma sample to the luma block, determine a plurality of luma samples sub-sampled based on the neighboring luma samples searched, where one of the sub-sampled luma samples corresponds to a sub-sampled luma sample which is the limit number of samples to the left of the luma block, determine one or more scheduling parameters based on the samples samples, determine a predictive block based on one or more scheduling parameters, and prediction by linear model (LM) encodes the chroma block based on the predictive block.
[0014] Details of one or more aspects of the disclosure are presented in the attached drawings and in the description below. Other resources, objectives and advantages of the disclosure will become evident from the description and drawings, and from the claims.
BRIEF DESCRIPTION OF THE DRAWINGS [0015] Figure 1 is a block diagram that illustrates an example of a video encoding system that can use the techniques described in this disclosure.
Petition 870190053680, of 12/06/2019, p. 14/171
10/119 [0016] Figure 2 is a block diagram that illustrates an example of a video encoder that can implement the techniques described in this disclosure.
[0017] Figure 3 is a block diagram that illustrates an example of a video decoder that can implement the techniques described in this disclosure.
[0018] Figure 4 is a conceptual diagram that illustrates nominal vertical and horizontal relative locations of luma and chroma samples.
[0019] Figure 5 is a conceptual diagram that illustrates examples of locations from which the scheduling parameters used to scale the reconstructed and subsampled luma block are derived.
[0020] Figure 6 is a conceptual diagram that illustrates an example of luma positions and chroma positions for the subsampling of samples from a luma block to generate a predictive block.
[0021] Figure 7 is a conceptual diagram that illustrates another example of luma positions and chroma positions for the subsampling of samples from a luma block to generate a predictive block.
[0022] Figures 8A to 8D are conceptual diagrams that illustrate neighboring samples used in the subsampling.
[0023] Figure 9 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample.
[0024] Figure 10 is a conceptual diagram that illustrates an example of luma samples that are sought
Petition 870190053680, of 12/06/2019, p. 15/171
11/119 to obtain a subsampled luma sample.
[0025] Figures 11A and 11B are conceptual diagrams that illustrate different filter lengths for samples sub-sampled in different positions.
[0026] Figures 12A and 12B are conceptual diagrams that illustrate filtering with filled samples.
[0027] Figure 13 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample.
[0028] Figure 14 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample.
[0029] Figures 15A and 15B are conceptual diagrams that illustrate different filter lengths for samples sub-sampled in different positions.
[0030] At Figures 16A and 16B are diagrams conceptual that illustrate the filtration with samples filled. [0031] THE Figure 17 it is a diagram conceptual
which illustrates neighboring blocks of the current block.
[0032] Figure 18 is a flow chart illustrating an example of a method of decoding video data.
[0033] Figure 19 is a flow chart that illustrates an example of a method for encoding video data.
DETAILED DESCRIPTION [0034] This disclosure describes video encoding and compression techniques. In particular, this disclosure describes techniques for the mode of
Petition 870190053680, of 12/06/2019, p. 16/171
12/119 prediction by linear model (LM). In LM prediction video encoding mode, a chroma block is predicted from a corresponding scaled, subsampled, reconstructed luma block (ie, that corresponding scaled, subsampled, reconstructed luma block forms a predictive block used for predict the chroma block).
[0035] In some examples, the subsampling of the corresponding reconstructed luma block includes filtering. This disclosure describes examples of ways in which such filtering is carried out. In addition, the techniques described in this disclosure can also be applied to situations where the luma samples used in the LM prediction mode are located on different tiles.
[0036] Consequently, the techniques described in this disclosure refer to a prediction mode by Linear Model (LM), which is used to reduce the redundancy between components in the video encoding. The techniques described in this disclosure can be used in the context of advanced video codecs, such as extensions to the high-efficiency video encoding (HEVC) video encoding standard or the next generation of video encoding standards.
[0037] When performing prediction encoding or decoding by LM, a video encoder or video decoder, respectively, looks for neighboring luma samples from the video data memory for subsampling in order to determine the scaling parameters used for stagger a corresponding subsampled luma block. If a type of filter used to sub-sample neighboring luma samples use samples
Petition 870190053680, of 12/06/2019, p. 17/171
13/119 of neighboring luma that are outside the range of neighboring luma samples that are locally stored (for example, in the local encoding circuitry memory), processing time and memory bandwidth can be wasted by the encoder. video or video decoder retrieving luma samples from an external memory. For example, in video encoding technology, instead of generic video encoding, there may be problems where performing LM prediction mode operations requires finding sample luma values from memory that may require processing time and additional memory bandwidth. This disclosure describes examples for reducing the number of sample values that are fetched, which could require relatively high amounts of processing time and memory bandwidth.
[0038] As an example, when searching for neighboring luma samples to subsample, the video encoder and video decoder can exclude certain luma samples (for example, luma samples not stored in local memory or luma samples that have not yet been generated) of the search. Thus, in one example, the search does not cause the video encoder and the video decoder to access non-local memory. Instead, in this example, the video encoder or video decoder fetches only luma samples from local memory, for example, for use in LM prediction mode operations.
[0039] In some examples, the video encoder
Petition 870190053680, of 12/06/2019, p. 18/171
14/119 video and the video decoder can be configured to subsample using different filters. For example, the video encoder and video decoder can apply a first filter when none of the neighboring luma samples that are excluded are needed for subsampling according to the first filter. However, if applying the first filter requires searching for the excluded luma samples, then the video encoder and video decoder can apply a second filter that is different from the first filter.
[0040] Figure 1 is a block diagram illustrating an example of a video encoding system 10 that can use the techniques of this disclosure. As used herein, the term video encoder generally refers to both video encoders and video decoders. In this disclosure, the terms video encoding or encoding may, in general, refer to video encoding or video decoding. Video encoder 20 and video decoder 30 of video encoding system 10 represent examples of devices that can be configured to perform techniques for video encoding based on linear model prediction (LM) according to several examples described in this revelation. For example, video encoder 20 and video decoder 30 can be configured to encode a chroma block using staggered, sub-sampled luma samples, reconstructed from a corresponding luma block, as described in this disclosure.
[0041] As shown in Figure 1, the
Petition 870190053680, of 12/06/2019, p. 19/171
The video encoding system 10 includes a source device 12 and a destination device 14. The source device 12 generates encoded video data. Consequently, the source device 12 can be called a video encoding device or a video encoding device. The target device 14 can decode the encoded video data generated by the source device 12. Consequently, the target device 14 can be called a video decoding device or a video decoding device. The source device 12 and the target device 14 can be examples of video encoding devices or video encoding devices.
[0042] The source device 12 and the target device 14 can comprise a wide range of devices, including desktop computers, mobile computing devices, notebook computers (e.g. laptop), tablet computers, set-top boxes, telephone sets such as so-called smart phones, televisions, cameras, display devices, digital media players, game consoles, car computers, or the like.
[0043] The target device 14 can receive encoded video data from the source device 12 via a channel 16. The channel 16 can comprise one or more means or devices capable of moving the encoded video data from the source device 12 to target device 14.
Petition 870190053680, of 12/06/2019, p. 20/171
11/16
In one example, channel 16 may comprise one or more communication means that allow the source device 12 to transmit encoded video data directly to the destination device 14 in real time. In this example, the source device 12 can modulate the video data encoded according to a communication standard, such as a wireless communication protocol, and can transmit the modulated video data to the destination device 14. 0 one or more media Communication channels may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. One or more means of communication may be part of a packet-based network, such as a local area network, a wide area network, or a global network (for example, the Internet). One or more means of communication may include routers, switches, base stations, or other equipment that facilitate communication from the source device 12 to the destination device 14.
[0044] In another example, channel 16 can include a storage medium that stores encoded video data generated by the source device 12. In this example, the destination device 14 can access the storage medium, for example, through access disk or card access. The storage medium may include a variety of locally accessed data storage media such as Blu-ray discs, DVDs, CD-ROMs, flash memory, or other digital storage media suitable for storing encoded video data.
[0045] In an additional example, channel 16
Petition 870190053680, of 12/06/2019, p. 21/171
17/119 can include a file server or other intermediate storage device that stores encoded video data generated by the source device 12. In this example, destination device 14 can access encoded video data stored on the file server or another device intermediate storage via streaming or download. The file server can be a type of server capable of storing encoded video data and transmitting encoded video data to the target device 14. Examples of file servers include web servers (for example, for a website) , file transfer protocol (FTP) servers, network-attached storage devices (NAS), or local disk drives.
[0046] The target device 14 can access the encoded video data through a standard data connection, such as an Internet connection. Examples of types of data connections may include wireless channels (for example, Wi-Fi connections), wired connections (for example, DSL, cable modem, etc.), or combinations of both that are suitable for accessing data from encoded video stored on a file server. The transmission of encoded video data from the file server can be a streaming transmission, a downloadable transmission, or a combination of both.
[0047] The techniques of this revelation no are limited applications or configurations without thread. At techniques can be applied to the coding of video in support a variety of multimedia applications, such as
Petition 870190053680, of 12/06/2019, p. 22/171
18/119 terrestrial television broadcasts, cable television broadcasts, satellite television broadcasts, streaming video broadcasts, for example, over the Internet, encoding video data for storage in a data storage medium, decoding video data stored on a data storage medium, or other applications. In some examples, the video encoding system 10 can be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video playback, video transmission and / or video telephony.
[0048] The video encoding system 10 illustrated in Figure 1 is merely an example and the techniques of this disclosure may apply to video encoding configurations (for example, video encoding or video decoding) that do not necessarily include any communication data between encoding and decoding devices. In some instances, data is retrieved from local memory, streamed over a network, or the like. A video encoding device can encode and store data in memory and / or a video decoding device can retrieve and decode data from memory. In many instances, encoding and decoding is performed by devices that do not communicate with each other, but simply encode data in memory and / or retrieve and decode data from memory.
[0049] In the example in Figure 1, the device
Petition 870190053680, of 12/06/2019, p. 23/171
11/199
in source 12 includes a video source 18, one encoder in video 20 and an interface output 22. In some examples, The interface about to leave 22 can include a
modulator / demodulator (modem) and / or a transmitter. The video source 18 can include a video capture device (for example, a video camera), a video file containing previously captured video data, a video feed interface for receiving video data from a content provider video and / or a computer graphics system for generating video data, or a combination of such video data sources.
[0050] Video encoder 20 can encode video data from video source 18. In some examples, source device 12 directly transmits encoded video data to destination device 14 via output interface 22. In other examples , the encoded video data can also be stored on a storage medium or a file server for later access by the target device 14 for decoding and / or playback.
[0051] In the example of Figure 1, the target device 14 includes an input interface 28, a video decodifleader 30, and a display device 32. In some examples, input interface 28 includes a receiver and / or a modem. Input interface 28 can receive encoded video data via channel 16. Display device 32 can be integrated or external to destination device 14. In general, display device 32 displays video data
Petition 870190053680, of 12/06/2019, p. 24/171
11/20 decoded. The display device 32 may comprise a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display or other type of display device.
[0052] Video encoder 20 and video decoder 30 can be implemented by one or more processors formed by any one of a variety of suitable circuitry (e.g., fixed function and / or programmable circuitry) as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuit (ASICs), field programmable port arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the techniques are partially implemented in the software, a device can store instructions for the software on a suitable non-temporary computer-readable storage medium and can execute instructions on hardware using one or more processors to perform the techniques in this disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) can be considered as one or more processors. Each video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device.
[0053] This revelation can refer, in
Petition 870190053680, of 12/06/2019, p. 25/171
21/119 generally, the video encoder 20 that signals or transmits certain information to another device, such as the video decoder 30. The term signaling or transmission can refer, in general, to the communication of elements of syntax and / or other data used to decode compressed video data. Such communication can occur in real time or almost in real time. Alternatively, such communication can occur over a period of time, as it can when syntax elements are stored in a computer-readable storage medium in a bit stream encoded at the time of encoding which can then be retrieved by a decoding device at any time after being stored on that medium.
[0054] In some examples, the video encoder 20 and the video decoder 30 may operate according to a video compression standard. Examples of video encoding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG- 4 Visual and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its Scalable Video Encoding (SVC) and Multivista Video Encoding (MVC) extensions.
[0055] In addition, a new video coding standard, ie, High Efficiency Video Coding (HEVC), was recently developed by the ITU-T Video Coding Experts Group's Joint Collaboration Team on Video Coding (JCT-VC) (VCEG) and ISO / IEC Motion Picture Experts Group (MPEG). A HEVC specification project,
Petition 870190053680, of 12/06/2019, p. 26/171
22/119 called HEVC WD later in this document, is available from http://phenix.intevry.fr/jct/doc_end_user/documents/14_Vienna/wgl 1 / JCTVCN1003-VÍ .zip. The HEVC standard is also presented together in Recommendation ITU-T H.265 and International Standard ISO / IEC 23008-2, both entitled High efficiency video coding, and both published in October 2014.
[0056] The HEVC specification and its extensions including Format Range Extensions (RExt), Scalability (SHVC), and Multivista (MV-HEVC) is available from http://phenix.intevry.fr/j et / doc_end_user / documents / 18_Sapporo / wgl 1 / JCTVCR1013- v6. zip.
[0057] Video encoding can be performed based on color space and color format. For example, color video plays an essential role in multimedia systems, where multiple color spaces are used to represent color efficiently. A color space specifies color with numerical values using multiple components. A popular color space is the RGB color space, where the color is represented as a combination of three primary color component values (that is, red, green and blue). For color video compression, the YCbCr color space was widely used, as described in A. Ford and A. Roberts, Color space conversions, University of Westminster, London, Tech. Rep., August 1998.
[0058] YCbCr can be easily converted from
Petition 870190053680, of 12/06/2019, p. 27/171
23/119 RGB color space through linear transformation and redundancy between different components, that is, cross-component redundancy, is significantly reduced in the YCbCr color space. An advantage of YCbCr is the backward compatibility with black and white TV as the Y signal transmits luminance information. In addition, the chrominance bandwidth can be reduced by sub-sampling components Cb and Cr in the 4: 2: 0 chroma sampling format with significantly less subjective impact than RGB sub-sampling. Due to these advantages, YCbCr has been the main color space in video compression. There are also other color spaces, such as YCoCg, used in video compression. In this development, regardless of the actual color space used, Y, Cb, Cr is used to represent the three color components in the video compression scheme.
[0059] In the sampling 4: 2: 0, each of the two chroma matrices is half the height and half the width of the luma matrix. The relative vertical and horizontal nominal locations of luma and chroma samples in the images are shown in Figure 4.
[0060] In HEVC and other video encoding standards, a video sequence typically includes a series of images. The images can also be called frames. The reconstructed image can include three sample matrices, denoted SL, Sob and Scr. S _L is a two-dimensional matrix (ie, a block) of luma samples. S _C b is a two-dimensional matrix of Cb chrominance samples. S _cr is a two-dimensional matrix of chrominance samples
Petition 870190053680, of 12/06/2019, p. 28/171
11/24
Cr. Chrominance samples can also be referred to in this document as chroma samples. In other cases, an image may be monochromatic and may include only a matrix of luma samples.
[0061] To generate an encoded representation of an image, video encoder 20 can generate a set of tree encoding units (CTUs). Each of the CTUs can be a luma sample tree coding block, two corresponding chroma sample tree coding blocks, and syntax structures used to encode the tree coding block samples. A tree coding block can be an NxN block of samples. A CTU can also be called a tree block or a large coding unit (LCU). HEVC CTUs can be broadly analogous to macroblocks from other standards, such as H.264 / AVC. However, a CTU is not necessarily limited to a specific size and can include one or more encoding units (CUs). A slice can include an integer number of CTUs sorted consecutively in the scanned scan.
[0062] To generate an encoded CTU, the video encoder 20 can recursively partition the tree of quadrants into the tree coding blocks of a CTU to divide the tree coding blocks into coding blocks, then the name units tree coding. A coding block is an NxN block of samples. A CU can be a luma sample coding block and two corresponding chroma sample
Petition 870190053680, of 12/06/2019, p. 29/171
25/119 image that has an array of luma samples, an array of Cb samples and an array of Cr samples, and syntax structures used to encode the samples in the coding blocks. The video encoder 20 can partition a CU encoding block into one or more prediction blocks. A prediction block can be a rectangular block (that is, square or non-square) of samples to which the same prediction is applied. A CU prediction unit (PU) can be a luma sample prediction block, two corresponding prediction blocks of chroma samples from an image, and syntax structures used to predict the prediction block samples. The video encoder 20 can generate predictive blocks of luma, Cb and Cr for prediction blocks of luma, Cb and Cr of each PU of the CU.
[0063] The video encoder 20 can use intraprediction, interpretation or prediction by linear model (LM), as some examples, to generate (for example, determine) the predictive blocks for a PU. If the video encoder 20 uses intraprediction to generate the predictive blocks of a PU, the video encoder 20 can generate the predictive blocks of the PU based on decoded samples of the image associated with the PU.
[0064] If the video encoder 20 uses interpretation to generate (for example, determine) the predictive blocks of a PU, the video encoder 20 can generate the predictive blocks of the PU based on decoded samples of one or more images except the image associated with PU. The video encoder 20 can use
Petition 870190053680, of 12/06/2019, p. 30/171
11/26 uniprediction or biprediction to generate the predictive blocks of a PU. When video encoder 20 uses uniprediction to generate the predictive blocks for a PU, the PU can have a single motion vector (MV). When video encoder 20 uses biprediction to generate the predictive blocks for a PU, the PU can have two MVs.
[0065] After the video encoder 20 generates predictive blocks of luma, Cb and Cr for one or more PUs of a CU, the video encoder 20 can generate a residual block of luma for the CU. Each sample in the CU residual luma block indicates a difference between a luma sample in one of CU's predictive luma blocks and a corresponding sample in the original CU luma coding block. In addition, video encoder 20 can generate a residual block of Cb for the CU. Each sample in the residual CU Cb block can indicate a difference between a Cb sample in one of CU's predictive Cb blocks and a corresponding sample in the original CU Cb coding block. The video encoder 20 can also generate a residual block of Cr for the CU. Each sample in the residual CU Cr block can indicate a difference between a Cr sample in one of CU's predictive Cr blocks and a corresponding sample in the original CU Cr coding block.
[0066] For a chroma block, instead of determining a predictive block for intra- or interpredition, the video encoder 20 can determine a predictive block based on a corresponding reconstructed luma block, for LM prediction mode. O
Petition 870190053680, of 12/06/2019, p. 31/171
27/119 video decoder 30 can similarly determine a predictive block based on a corresponding reconstructed luma block. The corresponding luma block refers to the luma block that was part of the unit (for example, coding unit or prediction unit) from which the current chroma block was determined. The video encoder 20 can determine the residual between the chroma block and that predictive block generated from a corresponding reconstructed foam block.
[0067] In addition, video encoder 20 can use the quadrants tree partition to decompose the residual blocks of luma, Cb and Cr of a CU into one or more blocks of luma transform, Cb and Cr. A transform block can be a rectangular block of samples to which the same transform is applied. A transform unit (TU) of a CU can be a luma sample transform block, two corresponding chroma sample transform blocks, and syntax structures used to transform the transform block samples. Thus, each CU of a CU can be associated with a luma transform block, a Cb transform block and a Cr transform block. The luma transform block associated with the TU can be a sub-block of the CU residual luma block. The Cb transform block can be a sub-block of the residual Cb block of CU. The Cr transform block can be a subblock of the residual Cr block of CU.
[0068] The video encoder 20 can apply one or more transforms to a transformer block of
Petition 870190053680, of 12/06/2019, p. 32/171
28/119 luma of a TU to generate a luma coefficient block for the TU. A coefficient block can be a two-dimensional matrix of transform coefficients. A transform coefficient can be a scalar quantity. The video encoder 20 can apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. The video encoder 20 can apply one or more transforms to a Cr transform block of a TU to generate a Cr coefficient block for the TU.
[0069] After generating a coefficient block (for example, a luma coefficient block, a Cb coefficient block or a Cr coefficient block), the video encoder 20 can quantize the coefficient block. In general, quantization refers to a process in which the transform coefficients are quantized possibly to reduce the amount of data used to represent the transform coefficients, providing additional compression. After the video encoder 20 quantizes a coefficient block, the video encoder 20 can perform entropy encoding of syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform Context Adaptive Binary Arithmetic Coding (CAB AC) on the syntax elements indicating the quantized transform coefficients. The video encoder 20 can send the entropy-encoded syntax elements in a bit stream.
[0070] The video encoder 20 can emit
Petition 870190053680, of 12/06/2019, p. 33/171
29/119 a bit stream that includes entropy-encoded syntax elements. The bit stream may include an encoded representation of video data. For example, the bit stream may include a bit stream that forms a representation of encoded images and data
associated companies. 0 flow bits can understand a sequence of units in layer in abstraction of network (NAL) Each unity of NAL includes one header of unity of NAL and encapsulates a useful load of sequence in bytes rough
(RBSP). The NAL unit header may include a syntax element indicating an NAL unit type code. The NAL unit type code specified by the NAL unit header of an NAL unit indicates the type of the NAL unit. An RBSP can be a syntax structure containing an integer number of bytes that is encapsulated within an NAL unit. In some cases, an RBSP includes zero bits.
[0071] Different types of NAL units can encapsulate different types of RBSPs. For example, a first type of NAL unit can encapsulate an RBSP for a set of image parameters (PPS), a second type of NAL unit can encapsulate an RBSP for an encoded slice, a third type of NAL unit can encapsulate an RBSP for supplementary improvement information (SEI), and so on. NAL units that encapsulate RBSPs for video encoding data (as opposed to RBSPs for parameter sets and SEI messages) can be called video encoding layer (VCL) NAL units.
Petition 870190053680, of 12/06/2019, p. 34/171
30/119 [0072] The video decoder 30 can receive a stream of bits generated by the video encoder 20. In addition, the video decoder 30 can analyze the bit stream to decode syntax elements of the bit stream. The video decoder 30 can reconstruct the images of the video data based at least in part on the syntax elements decoded from the bit stream. The process for reconstructing the video data can generally be reciprocal to the process performed by the video encoder 20. For example, the video decoder 30 can use PU MVs to determine predictive blocks for the PUs of a current CU. As another example, for the LM prediction mode, the video decoder 30 can determine the predictive block for a chroma block based on the reconstructed samples of a corresponding luma block. In addition, the video decoder 30 can inversely quantize the transform coefficient blocks associated with TUs of the current CU. The video decoder 30 can perform reverse transforms in the transform coefficient blocks to reconstruct the transform blocks associated with the TUs of the current CU.
[0073] The video decoder 30 can reconstruct the encoding blocks of the current CU by adding the samples of the predictive blocks for PUs of the current CU to corresponding samples of the transform blocks of the current CU's TUs. With the reconstruction of the encoding blocks for each CU of an image, the video decoder 30 can reconstruct the image.
[0074] In some examples, the video encoder
Petition 870190053680, of 12/06/2019, p. 35/171
31/119 video 20 and video decoder 30 can be configured to perform linear model based (LM) encoding. The following is a description of prediction coding based on LM. For example, although cross-complement redundancy is significantly reduced in the YCbCr color space, there is still a correlation between three color components. Several methods have been studied to improve the performance of video encoding by further reducing the correlation.
[0075] In 4: 2: 0 chroma video coding, a method called Linear Model (LM) prediction mode was well studied, during the development of HEVC pattern. See J. Chen, V. Seregin, W.-J. Han, J.-S. Kim, B.-M. Joen, CE6.a.4: Chroma intra prediction by reconstructed luma samples, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC UTC1 / SC29 / WG11, UCTVC-E266, 5th Meeting : Geneva, March 16-23, 2011, available from http://phenix.intevry.fr/j ct / doc_end_user / current_document.php Id = 219 6, and is called UCTVC-E266 below.
[0076] With the LM prediction mode, chroma samples are predicted based on reconstructed luma samples from the same block using a linear model as follows:
«Treç (D where prdec (i, j) represents the prediction of chroma samples in a block and recL represents the reconstructed luma samples subsampled from the same block. The α parameters
Petition 870190053680, of 12/06/2019, p. 36/171
32/119 and β are derived from reconstructed causal samples around the current block (for example, neighboring luma samples). Indicate the chroma block size in NxN; then, both i and j are within the range [0, N).
[0077] The parameters α and β in equation (1) are derived by minimizing the regression error between the neighboring reconstructed luma and chroma samples around the current block.
(2)
The parameters α and β are resolved as follows:
where x ± is the rebuilt Luma reference sample, y ± is the reconstructed Chroma reference sample, and I is a quantity (for example, number) of the reference samples. For a target NxN chroma block, when both causal samples on the left and above are available, the total number of samples involved I is equal to 2N; when only the causal samples on the left and above are available, the total number of samples involved I is equal to N.
[0078] Figure 5 is a conceptual diagram that illustrates examples of locations from which the scheduling parameters used to scale the reconstructed and subsampled luma block are derived. For example,
Petition 870190053680, of 12/06/2019, p. 37/171
11/33
Figure 5 illustrates one sampling example 4: 2: 0, and the parameters scheduling are α and β. [0079] In general when the way of prediction per LM for applied, O encoder video 20 and O
video decoder 30 can invoke the following steps. Video encoder 20 and video decoder 30 can sub-sample neighboring lumen samples. The video encoder 20 and the video decoder 30 can derive linear parameters (i.e., a and β) (also called scaling parameters). Video encoder 20 and video decoder 30 can subsample the current luma block and derive the prediction (e.g., predictive block) from the subsampled luma block and linear parameters (i.e., scaling parameters).
[0080] There can be several forms of subsampling. The following description presents examples of ways in which subsampling can be performed.
[0081] In JCTVC-E266, as described above, when the LM prediction mode is performed, the current subsampled luma block and the subsampled neighboring luma block are required. The current subsampled luma block is used to derive the prediction block for chroma coding while the subsampled neighboring luma block is used for the derivation of parameters (ie, a, α and β).
[0082] Since the typical sampling rate of chroma components is half the luma component and has a sample phase difference of 0.5 in the vertical direction in 4: 2: 0 sampling, the luma sample
Petition 870190053680, of 12/06/2019, p. 38/171
34/119 reconstructed from a current block is subsampled in the vertical direction and subsampled in the horizontal direction to match the size and phase of the chroma signal, as follows:
j) = (Re | 2ú 2 / j Ψ Re [2í 2j + I]) »1 (5) where Rec _LOr ig [] indicates the original reconstructed luma sample.
[0083] Figure 6 is a conceptual diagram that illustrates an example of luma positions and chroma positions for the subsampling of samples from a luma block to generate a predictive block for a luma block. As shown in Figure 6, a chroma sample, represented by the filled triangle (ie solid black), is predicted from two samples of luma, represented by the two filled circles, applying a filter [1, 1]. The filter [1, 1] is an example of a 2-lead filter.
[0084] For the subsampling of a neighboring luma block, when the neighboring samples are on the current luma block, the subsampling process is defined as:
(A -1) - Re (2d -1] (6) [0085] When the neighboring samples are to the left of the current luma block, the subsampling process is defined as:
(-1, J) - (Re 1-2,2 /] ã Re [-2,2 / +1 ']) »1 (7) [0086] The 2-lead filter, that is, [1,
Petition 870190053680, of 12/06/2019, p. 39/171
11/35
1], can be the same as that used in the example illustrated in Figure 6.
[0087] Other descending sampling techniques have also been proposed. In Yi-Jen Chiu, Yu Han, Lidong Xu, Wenhao Zhang, Hong Jiang, Cross-channel techniques to improve intra chroma prediction, Joint Collaborative Team on Video Coding (JCT-VC) of ITU- T SGI 6 WP3 and ISO / IEC JTC1 / SC29 / WG11, JCTVC-F502, 6th Meeting: Torino, IT, 14-22 July 2011, available from http://phenix.intevry.fr/j ct / doc_end_user / current_document.php Id = 2 97 9, and called JCTVC-F502, instead of using the two-lead filter, two-dimensional 6-lead filtering is applied to both the current luma block and the neighboring luma block. The set of two-dimensional filtering coefficients is:
[0088] In other words, the subsampled luma samples are derived by the following equation (9):
_n-;: /) - (R _and <..... J 2i 2/1 * 2 Re v | / 2. / 4 1 | 2'2 and R / 4 j -1 · Re [. .. 2í + Ç 2j] * 2. r Re [2 / * 1,2y +1] ψ Re [2 + I, 2 / -1]) »3 (9)
[0089] Figure 7 is a conceptual diagram that illustrates another example of luma positions and chroma positions for the subsampling of samples from a luma block to generate a predictive block. As shown in Figure 7, a chroma sample, represented by the triangle
Petition 870190053680, of 12/06/2019, p. 40/171
36/119 filled (ie solid black), it is predicted from six luma samples, represented by the six filled circles, applying a 6-lead filter.
[0090] Since the chroma sample predictor is derived using the linear function, as defined in formula (1), it could be observed that when the 6-lead filter is applied, the chroma sample predictor depends on the six neighboring luma samples. When
combine the equations (In ( 9), the result is as follows equation (10):jwá _e (i, /) - a> (Re <. ' _{i <w} | ; 2r, 2j] * 2 + Rec1] -í ' J 2 /, 2 / -1] 10)1 / + L 2 /] * 2-rRe <+ l> Re £, _íte J2i + 1,2 / -1]) »3) + β ( [0091] No text The next, the sample subsampled recL (i, j) is called sample luma subsampled corresponding for the sample of chroma located in (i, j). [0092] No Request Patent Provisional US No. 62 / 395,145, deposited in 15 September 2016 ( Request Ί45, below) i, and US Order No. 15/705 .029, deposited on 14
From September 2017 (Order Ό29, below), more subsampling filters are described, and more neighboring samples can be used to derive the linear models as shown in Figures 8A to 8D. In general, Figures 8A to 8D are conceptual diagrams that illustrate neighboring samples used for subsampling. In Figures 8A to 8D, the filled circles are examples of neighboring chroma samples. Their corresponding luma samples can be subsampled as part of subsampling neighboring luma samples to determine
Petition 870190053680, of 12/06/2019, p. 41/171
37/119 staggering.
[0093] The following description shows the signaling in LM mode. In JCTVC-E266 and JCTVC-F502 (both mentioned above), a flag is signaled to indicate whether LM prediction is used for the current block. Order '145 and Order' 029 describe more LM prediction modes, and a sophisticated signaling method can be described. LM modes and a Non-LM mode are organized into an ordered list. The order of modes depends on the chroma intraprediction modes of neighboring blocks. The index of the chroma intraprediction mode encoded in the ordered list of the current block is signaled by the video encoder 20.
[0094] There may be problems with some examples of techniques. The following description presents some of these possible problems. The subsampling process invoked in the LM prediction mode proposed in JCTVC-F502 may be more efficient, but it accesses more neighboring blocks (for example, neighboring luma samples), which can increase the cost of in-line buffering in the design of hardware. The signaling method proposed in Order '145 and Order' 029 can optimize the coding performance. Examples of techniques in Order '145 and Order' 029 may also include the overhead of reordering the mode list. Also, examples of techniques in Order '145 and Order' 029 can increase the cost of online buffering by accessing more neighboring blocks (for example, neighboring luma samples).
[0095] Consequently, there may be problems
Petition 870190053680, of 12/06/2019, p. 42/171
38/119 technologies present in some techniques for the subsampling of neighboring luma samples. The video encoder 20 and the video decoder 30 each include respective line buffers that are local to the respective video encoder 20 and video decoder 30. For example, the video encoder processing circuitry 20 you may not need a system bus external to the video encoder 20 to access the video encoder line buffer 20, and the video decoder processing circuitry 30 may not need a system bus external to the decoder video decoder 30 to access the video decoder line buffer 30. The video encoder line buffer 20 can be on the same integrated circuit (IC) chip as the video encoder 20, and the line buffer of the video decoder 20 video 30 can be on the same IC chip as video decoder 30. Video encoder 20 and video decoder 30 may also be able to search for data stored in the respective line buffers in a relatively short period of time, as the availability of line buffer access is not delayed due to some other component accessing the line buffer (for example, the respective line buffers may not be used by components except the video encoder 20 and the video decoder 30).
[0096] For example, of all the components that may be present in the IC that includes the video encoder 20 or the IC that includes the video decoder 30,
Petition 870190053680, of 12/06/2019, p. 43/171
39/119 only video encoder 20 or video decoder 30, or a limited number of IC components, can access the line buffer. There may be memory in the IC that includes video encoder 20 or video decoder 30. However, that memory can be accessible by all components in the IC. The line buffer, in some, but not necessarily all, examples may differ from the memory available to all components in the IC, and may be available for video encoder 20 or video decoder 30 (as applicable) and none another component or a limited number of components in the CI. The various components of the IC can access this memory via a system bus, but access to the line buffer may not require video encoder 20 or video decoder 30 to use the same system bus.
[0097] In some examples, the line buffer, as in the video decoder 30, can provide a relatively small amount of storage space. For example, a line buffer can store a limited number of luma and chroma sample values, and may not provide enough storage to store an entire image of luma and chroma sample values. The line buffer of the video decoder 30 can store reconstructed luma and chroma sample values as luma and chroma sample values that are close (e.g., neighboring) to a current block being encoded or decoded. If video decoder 30 fetches sample values from its
Petition 870190053680, of 12/06/2019, p. 44/171
40/119 line, the video decoder 30 may be able to retrieve sample values relatively quickly and with low power consumption. However, if the video decoder 30 fetches the sample values not stored within its line buffer, the video decoder 30 can spend energy and time accessing the system memory external to the video decoder 30 to retrieve the sample values. Consequently, there may be technical advantages to limiting the video decoder 30's access to memory other than line buffer memory when possible.
[0098] The line buffer memory of video encoder 20 may not necessarily be limited to a relatively small amount of storage space. However, due to the fact that the video decoder 30 performs the reverse operations of the video encoder 20, if the video encoder 20 depends on sample values for subsampling that could be outside the line buffer of the video decoder 30, there may be an impact on how quickly the video decoder 30 is able to reconstruct the blocks of an image.
[0099] The description above the line buffer is provided to aid understanding, and should not be considered limiting. The examples of techniques described in this disclosure can be applicable to examples where a larger line buffer memory is used, or when no line buffer memory is used. For ease, in Figures 2 and 3, reference is made to a data memory of
Petition 870190053680, of 12/06/2019, p. 45/171
41/119 video, and an example of video data memory is a line buffer. However, the video data memory may differ from the line buffer.
[00100] As described in more detail, in one or more examples of techniques, for the subsampling of neighboring luma samples for LM prediction mode, video encoder 20 and video decoder 30 can use filtering techniques of subsampling that exclude neighboring luma samples that are stored outside the line buffer. In some examples, if the luma samples used to sub-sample neighboring luma samples to determine the scheduling parameters are stored within the line buffer, then the video encoder 20 and video decoder 30 can perform a sub-sample accordingly. with a first filter. However, if the luma samples used to sub-sample neighboring luma samples to determine the scheduling parameters are not stored within the line buffer, then the video encoder 20 and video decoder 30 can perform a sub-sample according to a second filter. The second filter can be different from the first filter. In some examples, if the luma samples used to subsample the neighboring luma samples to determine the scheduling parameters are not stored within the line buffer, then the video encoder 20 and video decoder 30 can perform a subsampling of according to the first filter, but generate luma sample values for values not stored in the line buffer, instead of
Petition 870190053680, of 12/06/2019, p. 46/171
42/119 fetch those luma sample values from the memory external to video encoder 20 or video decodifleader 30.
[00101] The following describes techniques according to this disclosure. Examples of techniques can solve the problems mentioned above, but they do not necessarily need to solve the problems mentioned above. Detailed techniques can be applied individually. Alternatively, any combination of them can be applied.
[00102] In the technical examples, the coordinate (position) of the upper left sample of the current block is indicated by (xO, yO). Examples of techniques are described excluding luma samples from luma samples whose values are sought for subsampling. For ease, the luma samples that are excluded are
described in relation to the sample of upper left luma of current block. [00103] During the derivation process in samples in luma subsamples corresponding to an sample in chroma, samples sought luma are limited The a certain area depending on the position gives
chroma sample. As an example, during the derivation process of luma samples corresponding to chroma samples, a luma sample with the coordinate (x, y), where x <x0 and y <yO, is not involved in the subsampling process of neighboring luma samples above the current block. For example, in the search for neighboring luma samples for subsampling luma samples
Petition 870190053680, of 12/06/2019, p. 47/171
43/119 neighbors, video encoder 20 and video decoder 30 can exclude the luma samples that are above and to the left of the upper left luma sample from the luma block, and fetch the luma samples that are above the block of luma.
[00104] Figure 9 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample. Figure 10 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample. Figure 9 and Figure 10 demonstrate the difference between the JCTVC-F502 subsampling methods (Figure 9) and the techniques described in this disclosure (Figure 10). To obtain the sample of a neighboring luma above the most left, JCTVC-F502 describes that a process will search two samples of luma above the left of the current block; while the video encoder 20 and video decoder 30 cannot search for these two luma samples according to the example techniques described in this disclosure.
[00105] In one example, the length of the filter (ie number of samples to which the filter extends) to derive the corresponding subsample luma sample from a chroma sample is shorter when the filtering process involves luma samples with the coordinate (x, y), where x <x0 and y <y0. In other words, video encoder 20 and video decoder 30 can use different filters for the subsampling process in which neighboring lumen samples are involved in the
Petition 870190053680, of 12/06/2019, p. 48/171
44/119 subsampling.
[00106] Figures 11A and 11B are conceptual diagrams that illustrate different filter lengths for samples sub-sampled in different positions. The subsampling filter proposed in JCTVC-F502 is used when there is no luma sample with the coordinate (x, y), where x <x0, y <yO, involved in the filtering process, as shown in Figure 11A. On the other hand, the subsampling filter proposed in JCTVC-E266 is used when there is a sample of luma with the coordinate (x, y), where x <x0, y <yO, involved in the filtering process JCTVC-F502, as shown in Figure 11B. The xO and yO values can be coordinate values within the image, such as the coordinates in the given image.
[00107] In one example, the filters for deriving the corresponding subsampled luma sample from a chroma sample are the same, regardless of whether the filtering process involves luma samples with the coordinate (x, y), where x < x0 and y <yO. However, when the filtering process involves luma samples with the coordinate (x, y), where <x0 and y <yO, these luma samples are filled with adjacent luma samples instead of being fetched from the corresponding locations.
[00108] Figures 12A and 12B are conceptual diagrams that illustrate filtering with filled samples. In Figures 12A and 12B, the subsampling filter proposed in JCTVC-F502 is always used, regardless of whether a sample of luma with the coordinate (x, y), in which x <x0, y <yO, is involved or not
Petition 870190053680, of 12/06/2019, p. 49/171
45/119 filtration process. However, when luma samples with the coordinate (x, y), where x <xO and y <yO, are involved, these samples are filled in instead of being fetched as shown in Figure 12B. Equation (11) shows a filling process in the form of a formula.
mr (ã - (Re í 2 /, 2 f] * 2 * Re ^r ,. ·>. [2 / ™ 12 | + Re í ^. [2í + L 2 + ReQ, _w Ph 2_ / + l ] * 2 * -1 2y +1] tRcJ2; -el _s 2 / 4-1]) »3 where
Re2: -1 ,. 2J] - Re [2 /, 2J]
Re e _cw [2i -L 2, / 11] - Re c [2p 2/41] (11) [00109] Consequently, Figures 10 to 12B illustrate examples of which samples of the video encoder 20 and decoder of video 30 seek luma samples for the subsampling. For example, the neighboring luma samples sought include a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of the upper left luma sample of the luma block. The video encoder 20 and video decoder 30 can determine a plurality of sub-sampled luma samples based on the searched neighboring luma samples. One of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left sample.
Petition 870190053680, of 12/06/2019, p. 50/171
46/119 [00110] For example, Figures 10, 11B and 12B illustrate a subsampled luma sample that is immediately above the upper left luma sample. Figures 11A and 12A illustrate a subsampled luma sample that is above the luma block, but not immediately above the upper left luma sample. Consequently, Figures 10 to 12B illustrate examples of ways in which video encoder 20 and video decoder 30 can determine a plurality of subsampled luma samples, where the subsampled luma sample in Figures 10, 11B and 12B is a among the plurality of subsampled luma samples, and the subsampled luma sample in Figures 11A and 12A is another among the plurality of subsampled luma samples.
[00111] As shown in Figures 10, 11B and 12B, one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left sample. The neighboring luma samples sought that are subsampled to generate the subsampled luma sample of Figures 10, 11B and 12B exclude the luma samples that are above and to the left of an upper left luma sample of the luma block. In contrast, the neighboring luma samples sought that are subsampled to generate the subsampled luma sample of Figure 9 include the luma samples that are above and to the left of an upper left luma sample of the luma block. In some examples, luma samples that are above and to the left of an upper left sample of the luma block may not be stored in the line buffer memory.
Petition 870190053680, of 12/06/2019, p. 51/171
47/119
Therefore, access to these luma samples can be inefficient in terms of processing and bandwidth. Therefore, the examples of techniques can promote processing efficiency and bandwidth by excluding, from the luma samples sought, the samples that are above and to the left of an upper left sample of the luma block.
[00112] However, in some examples, all the luma samples necessary for the subsampling according to the filter described in JCTVC-F502 can be stored in the line buffer and in such examples, the video encoder 20 and the video decoder 30 can search for all neighboring luma samples necessary to perform the subsampling according to the JCTVC-F502 filter. If all the luma samples needed for subsampling according to the JCTVC-F502 filter are not available (for example, due to the fact that these luma samples are located above and to the left of the upper left sample), then the video encoder 20 and video decoder 30 can fetch samples different from those used in the JCTVC-F502 filter. It should be understood that the reference to the JCTVC-F502 filter is used merely as an example, and the techniques are applicable to other filters as well.
[00113] As shown in Figure 11A, video encoder 20 and video decoder 30 can apply a first filter to a first set of neighboring lumen samples sought to determine a first lumen sample from the plurality of lumen samples. , with no sample of
Petition 870190053680, of 12/06/2019, p. 52/171
48/119 luma that is above and to the left of the upper left luma sample is necessary for subsampling according to the first filter. For example, in Figures 11A and 12A, the first filter uses three luma samples from a first line that is above the luma block, and uses three luma samples from a second line that is above the first line. In Figures 11A and 12A, the first set of neighboring luma samples sought consists of the examples of luma samples used to generate the illustrated subsampled luma sample.
[00114] As shown in Figures 10 and 11B, video encoder 20 and video decoder 30 can apply a second filter to a second set of neighboring lumen samples sought to determine a second lumen sample subsampled from the plurality of sample samples. luma subsamples, when at least one luma sample which is above and to the left of the upper left luma sample of the luma block is required for subsampling according to the first filter. It should be understood that when the video encoder 20 or the video decoder 30 applies the second filter, the video encoder 20 and the video decoder 30 do not necessarily determine (although it is possible) that the use of the first filter could result in search for at least one luma sample that is above and to the left of the upper left luma sample. Preferably, in the condition that the use of the first filter could result in the search for at least one luma sample that is above and to the left of the upper left luma sample, the
Petition 870190053680, of 12/06/2019, p. 53/171
49/119 video 20 and ο video decoder 30 can use a second filter for subsampling that does not require the search for luma samples that are above and to the left of the upper left luma sample.
[00115] For example, in Figures 10 and 11B, if ο the first filter (for example, which is used in Figure 11A) is used, then video encoder 20 and video decoder 30 might be needed to fetch samples of luma that are above and to the left of the upper left luma sample. To avoid searching for luma samples that are above and to the left of the upper left luma samples, video encoder 20 and video decoder 30 can apply the second filter. In some examples, the second filter uses less than three luma samples from a first line that is above the luma block (for example, two luma samples in Figure 10 and one luma sample in Figure 11B), and uses less than three luma samples from a second line that is above the first line (for example, two luma samples in Figure 10, and one luma sample in Figure 11B).
[00116] In Figures 10, 11A and 11B, video encoder 20 and video decoder 30 can use different filters based on whether the search for luma samples could result in the search for luma samples that are above and to the left of the upper left luma sample. However, the examples of techniques are not so limited.
[00117] In some examples, video encoder 20 and video decoder 30 can use a
Petition 870190053680, of 12/06/2019, p. 54/171
50/119 filter. In the condition that the use of that filter for the subsampling could result in the search for video encoder 20 and video decoder 30 for luma samples that are above and to the left of the upper left luma sample, instead of searching for luma samples. which are above and to the left of the upper left luma sample, the video encoder 20 and the video decoder 30 can generate luma values (for example, fill values described above). For example, as shown in Figure 12B, video encoder 20 and video decoder 30 can generate luma values (for example, fill values) without searching that correspond to luma samples located above and to the left of the sample upper left luma of the luma block. The video encoder 20 and the video decoder can apply a filter (e.g., JCTVC-F502 filter) to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[00118] The examples provided above for the subsampling of neighboring luma samples that are above the luma block. The following description presents examples for the subsampling of neighboring luma samples that are to the left of the luma block. The following examples for the subsampling of neighboring luma samples on the left and the above examples for the subsampling of neighboring luma samples that are above the luma block can be used together or separately.
[00119] For example, video encoder 20
Petition 870190053680, of 12/06/2019, p. 55/171
51/119 and video decoder 30 can be configured to perform one or more exemplary techniques described in this disclosure in relation to the subsampling above the surrounding luma samples, and may not be configured to perform one or more exemplary techniques described in this disclosure in relation to subsampling of neighboring luma samples on the left. The video encoder 20 and video decoder 30 may not be configured to perform one or more of the exemplary techniques described in this disclosure in relation to the subsampling above of the surrounding luma samples, and may be configured to perform one or more of the exemplary techniques described in this disclosure. revelation in relation to the subsampling of luma samples neighboring on the left. The video encoder 20 and video decoder 30 can be configured to perform one or more exemplary techniques described in this disclosure in relation to the subsampling above the surrounding luma samples, and can be configured to perform one or more exemplary techniques described in this disclosure in regarding the subsampling of luma samples neighboring on the left.
[00120] For luma samples neighboring on the left, as an example, during the derivation process of corresponding luma samples of chroma samples, a luma sample with the coordinate (x, y), where x <x0- k ey> = y0, is not involved in the subsampling process of luma samples neighboring to the left of the current block. Here, k is an integer greater than 0. In an example, k is equal to 4.
Petition 870190053680, of 12/06/2019, p. 56/171
52/119 [00121] Figure 13 is a conceptual diagram that illustrates an example of luma samples that are sought to obtain a subsampled luma sample. Figure 14 is a conceptual diagram that illustrates another example of luma samples that are sought to obtain a subsampled luma sample. Figure 13 and Figure 14 demonstrate the difference between the JCTVC-F502 subsampling methods (Figure 13) and the techniques described in this disclosure (Figure 14) respectively, when k = 4. To obtain the sample of the neighboring left luma subsampled from the second column, as described in Order '145 and Order' 029, the subsampling method in JCTVC-F502 will search for two luma samples with the coordinates (x0-5, yO) and ( x0-5, yO + 1); while video encoder 20 and video decoder 30 cannot fetch these two luma samples according to examples in this disclosure.
[00122] In one example, the length of the filter (that is, the number of samples the filter used as an input) to derive the corresponding subsampled luma sample from a chroma sample is shorter when the filtering process involves luma samples with the coordinate (x, y), where x <x0-k and y> = y0.
[00123] Figures 15A and 15B are conceptual diagrams that illustrate different filter lengths for samples sub-sampled in different positions. The subsampling filter proposed in JCTVC-F502 is used when no luma sample with the coordinate (x, y), where x <x0-k and y> = y0, is involved in the filtering process, as shown in Figure 15 A . For another
Petition 870190053680, of 12/06/2019, p. 57/171
53/119 side, the subsampling filter proposed in JCTVC-E266 is used when luma samples with the coordinate (x, y), where x <xO-k and y> = yO, are involved in the JCTVC- F502, as shown in Figure 15B.
[00124] In one example, the filters for deriving the corresponding subsampled luma sample from a chroma sample are the same, regardless of whether or not the filtering process involves luma samples with the coordinate (x, y), where x < x0-k. However, when the filtering process involves luma samples with the coordinate (x, y), where x <x0-k and y> = y0, these luma samples are filled with adjacent luma samples instead of being fetched from locations corresponding.
[00125] Figures 16A and 16B are conceptual diagrams that illustrate filtering with filled samples. In Figures 16A and 16B, the subsampling filter proposed in JCTVC-F502 is always used, regardless of whether a sample of luma with the coordinate (x, y), in which x <x0-k and y> = y0, is involved or not in the filtering process. However, when luma samples with the coordinate (x, y), where x <x0-k and y> = y0, are involved, these samples are filled in instead of being fetched as shown in Figure 16B. Equation (11) shows a filling process in the form of a formula.
my (f,. /) = (Re [2f, 2 / | * 2 + Re c - ζ 2 / J r Re ^ [2i + l _f 2 /]
-η Re [2L 2j +!] * 2 -η Ree [2 / -1,2 / +1] + Re [2 / ζ 2. / 4 · I]) »3
Petition 870190053680, of 12/06/2019, p. 58/171
54/119 where
Re Ú 12 / · ~ 1,2 / · ± rj - Re _ÍÍ ', _iXa [2ς 2/41] [00126] Consequently, Figures 14 to 16B illustrate examples of which luma samples the video encoder 20 and decoder of video 30 seek luma samples for the subsampling. For example, the neighboring luma samples searched for include a plurality of luma samples that are to the left of the luma block and exclude luma samples that are to the left and that are more than a limit number (for example, k) of samples to the left of the luma block and below a sample of the upper left luma block. The video encoder 20 and the video decoder 30 can determine a plurality
samples of luma subsamples with based on samples of luma neighbors sought. An of samples in luma subsampled corresponds The an sample in luma
subsample that is the limit number of samples to the left of the luma block.
[00127] For example, Figures 14, 15B and 15B illustrate a subsampled luma sample that is four samples to the left of the luma block, and in this example, the limit number of samples to the left of the luma block is four (for example , k = 4). Then, Figures 14, 15B and 16B illustrate examples in which the neighboring luma samples sought include samples that are to the left of the luma block and exclude luma samples that are more than the limit number of samples to the left of the luma block. luma e
Petition 870190053680, of 12/06/2019, p. 59/171
55/119 are below an upper left luma sample, where one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block.
[00128] Figures 15A and 16A illustrate a subsampled luma sample that is to the left of the luma block, but not a limit number of luma samples to the left of the luma block. For example, if the limit number of luma samples on the left, beyond which luma samples are excluded from the search, is four, then in Figures 15A and 16A, the subsampled luma samples are illustrated as two luma samples on the left of the luma block (for example, in a column that is two samples of luma to the left of the luma block). Accordingly, Figures 14 to 16B illustrate examples of ways in which the video encoder 20 and video decoder 30 can determine a plurality of subsampled luma samples, where the subsampled luma sample in Figures 14, 15B and 16B is a among the plurality of subsampled luma samples, and the subsampled luma sample in Figures 15A and 16A is another among the plurality of subsampled luma samples.
[00129] As shown in Figures 14, 15B and 15B, one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of luma samples to the left of the luma block (for example, it corresponds to a sample of l a subsample of more than two columns since the limit number is four). The neighboring luma samples sought that are subsampled
Petition 870190053680, of 12/06/2019, p. 60/171
56/119 to generate the luma sample subsampled from Figures 14, 15B and 16B exclude luma samples that are more than a limit number (for example, k = 4) of samples to the left of the luma block and below a upper left sample of the luma block. In contrast, the neighboring luma samples sought that are subsampled to generate the subsampled luma sample in Figure 13 include luma samples that are more than a limit number of samples to the left of the luma block. In some examples, luma samples that are more than a limit number of samples to the left of the luma block and are below an upper left sample of the luma block may not be stored in the line buffer memory. Therefore, access to these luma samples can be inefficient in terms of processing and bandwidth. So, examples of techniques can promote processing efficiency and bandwidth by excluding, from the search, luma samples that are more than a limit number of samples to the left of the luma block and are below an upper luma sample. left of the luma block.
[00130] However, in some examples, all the luma samples necessary for the subsampling according to the filter described in JCTVC-F502 can be stored in the line buffer and in such examples, the video encoder 20 and the video decoder 30 can search for all neighboring luma samples necessary to perform the subsampling according to the JCTVC-F502 filter. If all luma samples required for subsampling according to the JCTVC-F502 filter are not
Petition 870190053680, of 12/06/2019, p. 61/171
57/119 available (for example, due to the fact that these luma samples are located more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block), then the video encoder 20 and video decoder 30 can fetch samples different from those used in the JCTVC-F502 filter. It should be understood that the reference to the JCTVC-F502 filter is used merely as an example, and the techniques are applicable to other filters as well.
[00131] As shown in Figure 15A, video encoder 20 and video decoder 30 can apply a first filter to a first set of neighboring lumen samples sought to determine a first lumen sample from the plurality of lumen samples. , when no luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample is necessary for subsampling according to the first filter. The first sample of a subsampled luma can be in a column that is a first number of columns to the left of the luma block. For example, in Figures 15A and 16A, the first filter uses three luma samples from a first line that is to the left of the luma block, and uses three luma samples from a second line that is below the first line. Also in Figures 15A and 16A, the subsampled luma sample is in a column that is two columns to the left of the luma block. In Figures 15A and 16A, the first set of neighboring luma samples sought consists of examples of
Petition 870190053680, of 12/06/2019, p. 62/171
58/119 luma samples used to generate the illustrated subsampled luma sample.
[00132] As shown in Figures 14 and 15, video encoder 20 and video decodifleader 30 can apply a second filter to a second set of neighboring lumen samples sought to determine a second lumen sample subsampled from the plurality of sample samples. luma subsamples, when at least one luma sample which is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block could be required for subsampling according to the first filter. It should be understood that when the video encoder 20 or the video decoder 30 applies the second filter, the video encoder 20 and the video decoder 30 do not necessarily determine (although it is possible) that the use of the first filter could result in search for at least one luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample. Preferably, in the condition that the use of the first filter could result in the search for at least one sample of luma that is more than the limit number of samples to the left of the luma block and below the upper left luma sample, the encoder video 20 and video decoder 30 can use a second filter for subsampling that does not require the search for luma samples that are more than the limit number of samples to the left of the luma block and below the upper luma sample
Petition 870190053680, of 12/06/2019, p. 63/171
59/119 left.
[00133] For example, in Figures 14 and 15B, if the first filter (for example, the one used in Figure 11A) is used, then the video encoder 20 and the video decoder 30 might be needed to fetch the samples from luma that are more than the limit number of luma samples to the left of the luma block and below the upper left luma sample. To avoid searching for luma samples that are more than the limit number of luma samples to the left of the luma block and below the upper left luma sample, video encoder 20 and video decoder 30 can apply the second filter. In some examples, the second filter uses less than three luma samples from a first line that is to the left of the luma block (for example, two luma samples in Figure 14, and one luma sample in Figure 15B), and uses less than three luma samples from a second line that is above the first line (for example, two luma samples in Figure 14, and one luma sample in Figure 15B).
[00134] In Figures 10, 15A and 15B, the video encoder 20 and the video decoder 30 can use different filters based on whether the search for the luma sample could result in the search for luma samples that are longer than the limit number of luma samples to the left of the luma block and below the upper left luma sample. However, the examples of techniques are not so limited.
[00135] In some examples, the video encoder
Petition 870190053680, of 12/06/2019, p. 64/171
60/119 video 20 and the video decodifleader 30 can use a filter. In the condition that the use of that filter for the subsampling could result in the search for video encoder 20 and video decoder 30 for luma samples that are more than the limit number of luma samples to the left of the luma block and below the upper left luma sample, instead of fetching those luma samples, the video encoder 20 and video decoder 30 can generate luma values (e.g., fill values described above). For example, as shown in Figure 12B, video encoder 20 and video decoder 30 can generate luma values (for example, fill values) that match luma samples located more than the limit number without searching. of luma samples to the left of the luma block and below the upper left luma sample of the luma block. The video encoder 20 and the video decoder can apply a filter (e.g., JCTVC-F502 filter) to the generated luma values to determine at least one subsampled lumen sample from the plurality of subsampled lumen samples.
[00136] The techniques mentioned above may not be limited to the specific subsampling filters described above, and can be used in conjunction with any subsampling filter. Also, the aforementioned techniques may not be limited to video / image encoding in a yuv420 chroma format. As an example, they can also be used in video / image encoding in the yuv422 chroma format.
Petition 870190053680, of 12/06/2019, p. 65/171
61/119 [00137] A cluster flag called the LM flag, to indicate whether the LM mode (such as the LM mode proposed in JCTVC-E266 or one of the multiple LM modes proposed in Order '145 and Order' 029) is encoded per video encoder 20 or video decoder 30, as applicable. The coding context of this flag may depend on the encoded / decoded LM flags of one or more neighboring blocks.
[00138] In one example, such a flag is first encoded. If this flag indicates that an LM mode is used, the LM mode index can be additionally signaled in the bit stream encoded by the video encoder 20. Otherwise, other chroma modes excluding LM modes can be additionally signaled. As an example, there can be 3 contexts for the LM flag: LM ctx [0], LM ctx [l] and LM ctx [2]. A variable ctx is calculated as ctx = LM_flag_A + LM_flag_B, where the LM A flag and the LM B flag are the LM flags of neighboring blocks. In one example, the neighboring blocks denoted by A and B are shown in Figure 17, respectively (for example, A, A0, Al, B, B0, BI and B2). Figure 17 is a conceptual diagram that illustrates neighboring blocks of the current block.
In an example furthermore when the
Block X (X being one of the neighboring blocks) is not coded by intraprediction or does not exist (that is, it is not available), the flag LM X is equal to 0, where X is A or B. In an example, in addition , when a neighboring block is located above the current block, it can be
Petition 870190053680, of 12/06/2019, p. 66/171
62/119 the neighboring block must be within the same LCU. If the neighboring block is outside the current LCU, it can be treated as unavailable.
[00140] Consequently, in one example, video decoder 30 can decode a flag for a current block that includes the flash block and the chroma block. The flag (for example, LM flag) indicates that LM prediction coding is enabled for the chroma block. The video decoder 30 can be configured to decode the flag based on a context comprising one or more flags indicating whether LM prediction encoding is enabled for neighboring blocks. Similarly, video encoder 20 can encode a flag for a current block that includes the flash block and the chroma block. The flag (for example, LM flag) indicates that LM prediction coding is enabled for the chroma block. Video encoder 20 can encode the flag based on a context comprising one or more flags indicating whether LM prediction encoding is enabled for neighboring blocks.
[00141] In this disclosure, the examples of techniques described in relation to the LM flag can be applied separately and independently of the examples of related techniques with which luma samples are subsampled for LM prediction. In some examples, video encoder 20 and video decoder 30 can be configured to exclude neighboring luma samples from the subsampling, as described above, and encode or
Petition 870190053680, of 12/06/2019, p. 67/171
63/119 decode the LM flag as described above. In some examples, video encoder 20 and video decoder 30 can be configured to exclude neighboring luma samples from the subsample, as described above, but do not encode or decode the LM flag as described above. In some examples, video encoder 20 and video decoder 30 may not be configured to exclude neighboring luma samples from the subsample, as described above, but may be configured to encode or decode the LM flag as described above.
[00142] The following description presents an example for implementing one or more examples of techniques described in this disclosure. The example is for the examples of downfiltering techniques (for example, sub-sampling) of the neighboring luma samples above, or for the examples of downfiltering techniques (for example, sub-sampling) of the neighboring luma samples on the left. Assuming that ReciOng [0, 0] represents the upper left luma sample of the current block, the subsampling process for the neighboring luma samples above is defined as follows:
- if the chroma sample is no longer located to the left of the neighboring line above, that is, when i> 0:
regO. · ''!) 2% -2] * 2 <Rec, _A> J [2f 2] 'l'Req ^ J 2 / H ~ 2]
-s-Rec ,. _v 42í. -l] + Re ^. [2í + l.-q + #) «)» 3
...... '..... * ...... (12)
- Otherwise, that is, when i is equal to 0:
Petition 870190053680, of 12/06/2019, p. 68/171
64/119 mç (j ,. -1) s' ^: ((Re u, [2 /, - 2 | ♦ 3 4- Re c- _tiW [22 9-1 - 2 |
4-Reeq ^ pl -91-1] -9i9 $ fe / 1.) >> 3 (13)
[00143] In An example, offset0 and offsetl are both equal to 0. In An example, offset0 and offsetl are both equal to 4. Per example, for subsampling of
above luma samples according to the example in Figure 10, video encoder 20 and ο video decoder 30 can apply equation (13) with a displacement of 0. For the subsampling of the left luma samples according to the example of Figure 14, the video encoder 20 and ο video decoder 30 can apply equation (13) with an offset of 4. In this example, the offset is 4, since the luma samples more than 4 samples to the left of the luma block are excluded from subsampling. The offset can be different from 4 if less than or more than 4 samples to the left of the luma block are excluded from the subsample.
[00144] For example, video encoder 20 and video decoder 30 can perform the operations of equation (12) (for example, applying a first filter to a first set of neighboring luma samples searched) to determine a first sample subsampled luma when no luma sample that is above and to the left of the upper left luma sample is necessary for subsampling according to the operations of equation (12). As an example, video encoder 20 and video decoder 30 can perform the operations of equation (12) to determine the sample of a subsampled luma
Petition 870190053680, of 12/06/2019, p. 69/171
65/119 illustrated in Figure 11A using the neighboring luma samples (for example, a first set of the searched neighboring luma samples) illustrated in Figure 11A.
[00145] The video encoder 20 and the video decoder 30 can perform the operations of equation (13) (for example, applying a second filter to a second set of neighboring luma samples searched) to determine a second subsampled luma sample when at least one luma sample which is above and to the left of the upper left luma sample of the luma block is required for subsampling according to the operations of equation (12) (for example, according to the first filter). For example, if the first filter is used, then at least one sample of luma that is above and to the left of the upper left sample of the luma block might be necessary for the subsampling. As an example, video encoder 20 and video decoder 30 can perform the operations of equation (13) to determine the subsampled luma sample illustrated in Figure 10 using the neighboring luma samples (for example, a second set of samples neighboring luma species) illustrated in Figure 10.
[00146] For example, video encoder 20 and video decoder 30 can perform the operations of equation (12) (for example, applying a first filter to a first set of neighboring luma samples searched) to determine a first sample subsampled luma when no luma sample that is more than the limit number of samples (eg k samples) to
Petition 870190053680, of 12/06/2019, p. 70/171
66/119 left of the luma block and below the upper left luma sample is necessary for subsampling according to the operations of equation (12). The first sample of a subsampled luma is in a column that is a first number of columns to the left of the luma block. As an example, video encoder 20 and video decoder 30 can perform the operations of equation (12) to determine the sample of a subsampled luma illustrated in Figure 16A using the neighboring luma samples (for example, a first set of samples neighboring luma
sought ) illustrated in Figure 15A. As illustrated, the sample of luma subsampled in Figure 15A is two columns to the left of luma block. [00147] 0 encoder of video 2 0 and the
video decoder 30 can perform the operations of equation (13) (for example, applying a second filter to a second set of neighboring luma samples searched) to determine a second lumen sampled luma when at least one luma sample is more than the limit number of samples (for example, k samples) to the left of the luma block and below the upper left luma sample of the luma block is required for subsampling according to the operations of equation (12) (for example, according to the first filter). For example, if the first filter is used, then at least one sample of luma that is more than the limit number of samples (for example, k samples) to the left of the luma block and below the upper left luma sample of the luma block could be required for subsampling. THE
Petition 870190053680, of 12/06/2019, p. 71/171
67/119 second sample of a subsampled luma is in a column that is a second number of columns to the left of the luma block, and the second number is greater than the first number. As an example, video encoder 20 and video decoder 30 can perform the operations of equation (13) to determine the sample of a subsampled luma illustrated in Figure 14 using the neighboring luma samples (for example, a second set of samples neighboring luma groups) illustrated in Figure 14.
[00148] Figure 2 is a block diagram illustrating an example of video encoder 20 that can implement the techniques of this disclosure. Figure 2 is provided for explanatory purposes and should not be considered as limiting the techniques as widely exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video encoder 20 in the context of HEVC encoding. However, the techniques in this disclosure may apply to other coding standards or methods, including those under development today or in the future.
[00149] Video encoder 20 represents an example of a device that can be configured to perform techniques for LM-based video encoding according to several examples described in this disclosure. For example, video encoder 20 can be configured to encode one or more blocks using LM video encoding.
[00150] In the example of Figure 2, video encoder 20 includes a prediction processing unit
Petition 870190053680, of 12/06/2019, p. 72/171
68/119
100, video data memory 101, a residual generation unit 102, a transform processing unit 104, a quantization unit 106, a reverse quantization unit 108, a reverse transform processing unit 110, a reconstruction unit 112, a filter unit 114, a decoded image buffer (DPB) 116 and an entropy encoding unit 118. The prediction processing unit 100 includes an interpreting processing unit 120 and an intraprediction processing unit 126. Interpretation processing unit 120 includes a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 also includes a linear model-based (LM) encoding unit 122 configured to perform various aspects of the LM-based encoding techniques described in this disclosure. In other examples, the video encoder 20 may include more, less or different functional components.
[00151] Video data memory 101 can store video data that will be encoded by video encoder components 20. Video data stored in video data memory 101 can be obtained, for example, from the video source video 18. The DPB 116 can be a reference image memory that stores reference video data for use in encoding video data by video encoder 20 (for example, in intra or intercoding modes). Video data memory 101 and DPB 116 can be formed
Petition 870190053680, of 12/06/2019, p. 73/171
69/119 by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices . Video data memory 101 and DPB 116 can be provided by the same memory device or separate memory devices. In several examples, the video data memory 101 may be on the chip with other video encoder components 20, or off the chip with respect to those components.
[00152] In this disclosure, reference to video data memory 101 should not be construed as limited to the internal memory of video encoder 20, unless specifically described as such, or memory external to video encoder 20, unless specifically described as such. The reference to the video data memory 101 is to be understood as referring to the memory that stores video data that the video encoder 20 receives for encoding (for example, video data for a current block that must be encoded). The video data memory 101 can also provide temporary storage of outputs of the various video encoder units 20.
[00153] As an example, the video data memory 101 is an example of a line buffer that is internal to the video encoder 20 and stores samples from neighboring blocks that are adjacent to the current block being encoded. As another example, part of DPB 116 can be the line buffer that is internal to the video encoder 20, and
Petition 870190053680, of 12/06/2019, p. 74/171
70/119 part of DPB 116 may be the memory that is external to the video encoder 20, as part of the system memory of the IC chip that includes the video encoder 20. As another example, the line buffer may be the memory video encoder cache 20.
[00154] The various units in Figure 2 are illustrated to help understand the operations performed by the video encoder 20. The units can be implemented as fixed function circuits, programmable circuits, or a combination of them. Fixed-function circuits refer to circuits that provide specific functionality, and are present in operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, and provide flexible functionality in the operations that can be performed. For example, programmable circuits can run software or firmware that causes programmable circuits to operate in the manner defined by instructions in the software or firmware. Fixed-function circuits can execute software instructions (for example, to receive parameters or output parameters), but the types of operations that fixed-function circuits perform are generally immutable. In some examples, the one or more units can be separate circuit blocks (fixed or programmable function), and in some examples, the one or more units can be integrated circuits.
[00155] The video encoder 20 can include logical and arithmetic units (ALUs), function units
Petition 870190053680, of 12/06/2019, p. 75/171
71/119 elementary (EFUs), digital circuits, analog circuits and / or programmable cores, formed from programmable circuits. In examples where video encoder 20 operations are performed by software executed by programmable circuits, video data memory 101 can store the software object code that video encoder 20 receives and executes, or other memory (not shown) can store such instructions.
[00156] Video encoder 20 can receive video data. The video encoder 20 can encode each CTU into a slice of a reconstructed image of the video data. Each of the CTUs can be associated with equally sized luma coding tree blocks (CTBs) and corresponding CTBs of the image. As part of coding a CTU, the prediction processing unit 100 can perform quadtree partitioning to divide CTU CTBs into progressively smaller blocks. The smallest block can be CU coding blocks. For example, the prediction processing unit 100 can partition a CTB associated with a CTU into four equally sized sub-blocks, partition one or more of the sub-blocks into four equally sized sub-blocks, and so on.
[00157] The video encoder 20 can encode the CUs of a CTU to generate encoded representations of the CUs (i.e., encoded Cus). As part of the encoding of a CU, the prediction processing unit 100 can partition the encoding blocks associated with the CU between one or more PUs of the CU. Thus,
Petition 870190053680, of 12/06/2019, p. 76/171
72/119 each PU can be associated with a luma prediction block and corresponding chroma prediction blocks. The video encoder 20 and the video decoder 30 can support PUs that have various sizes. As indicated above, the size of a CU can refer to the size of the CU's luma coding block and the size of a PU can refer to the size of a PU's prediction block. Assuming that the size of a specific CU is 2Nx2N, the video encoder 20 and video decoder 30 can support PU sizes of 2Nx2N or NxN for intraprediction, and symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, or similar for interpretation. The video encoder 20 and video decoder 30 can also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N and nRx2N for interpretation.
[00158] Interpretation processing unit 120 can generate predictive data for a PU by performing the interpretation on each PU of a CU. Predictive data for PU can include predictive blocks for PU and motion information for PU. Interpretation processing unit 120 can perform different operations for a PU from a CU depending on whether the PU is in a slice I, a slice P or a slice B. In a slice I, all PUs are intrapredicted. So, if the PU is in a slice I, the interpreting processing unit 120 does not perform interpreting in the PU. Thus, for blocks encoded in mode I, the predicted block is formed using the spatial prediction of neighboring blocks previously encoded within the same structure.
Petition 870190053680, of 12/06/2019, p. 77/171
73/119 [00159] If a PU is in a P slice, the Interpretation Processing Unit Motion Estimation Unit 120 can fetch the reference images from a list of reference images (for example, RefPicListO) for a region reference for the PU. The reference region for the PU can be a region, within a reference image, that contains the sample blocks that best correspond to the sample blocks of the PU. The movement estimation unit can generate a reference index that indicates a position in RefPicListO of the reference image containing the reference region for the PU. In addition, the motion estimation unit can generate a MV that indicates a spatial shift between a PU coding block and a reference location associated with the reference region. For example, the MV can be a two-dimensional vector that provides an offset from the coordinates in the current decoded image to the coordinates in a reference image. The motion estimation unit can send the reference index and the MV as the PU motion information. The Interpretation Processing Unit Motion Compensation Unit 120 can generate the PU predictive blocks based on actual or interpellated samples at the reference location indicated by the PU motion vector.
[00160] If a PU is in a B slice, the Interpretation Processing Unit 120 motion estimation unit can perform uniprediction or biprediction for the PU. To perform the
Petition 870190053680, of 12/06/2019, p. 78/171
74/119 unipredição de PU, the movement estimation unit can fetch the reference images from RefPicListO or a second list of reference images (RefPicListl) for a reference region of the PU. The movement estimation unit can send, like the PU movement information, a reference index that indicates a position in RefPicListO or RefPicListl of the reference image that contains the reference region, a MV that indicates a spatial displacement between a block of PU prediction and a reference location associated with the reference region and one or more prediction direction indicators that indicate whether the reference image is in RefPicListO or RefPicListl. The Interpretation Processing Unit Motion Compensation Unit 120 can generate PU predictive blocks based at least in part on actual samples or interpellated in the reference region indicated by
motion vector from PU • [00161] For accomplish The interpretation bidirectional to an PU, the unit in Estimate of movement can seek the images in reference in
RefPicListO by a reference region for the PU and you can also search the reference images in RefPicListl for another reference region for the PU. The motion estimation unit can generate reference image indices that indicate positions in RefPicListO and RefPicListl of the reference images that contain the reference regions. In addition, the motion estimation unit can generate MVs that indicate spatial displacements between the reference location associated with the
Petition 870190053680, of 12/06/2019, p. 79/171
75/119 reference regions and a PU sample block. PU movement information can include PU reference indexes and VMs. The Interpretation Processing Unit Motion Compensation Unit 120 can generate the PU predictive blocks based at least in part on actual samples or interpellated in the reference regions indicated by the PU motion vectors.
[00162] The encoding unit based on LM 122 can perform encoding by prediction by linear model (LM). For example, the LM 122-based coding unit can sub-sample the reconstructed luma samples from a luma block that corresponds to a current chroma block being encoded. The LM 122-based coding unit can scale the reconstructed luma samples subsampled from the luma block to generate a predictive block. Residual generation unit 102 can determine a residual block indicating a difference in sample values between the chroma block and the predictive block. In some instances, the LM 122-based coding unit may apply such techniques as part of residual cross-component prediction. In that case, the chroma block is a residual chroma block and the luma block is a residual luma block.
[00163] For example, the coding unit based on LM 122 can determine a luma block that corresponds to a chroma block. The coding unit based on LM 122 can search for neighboring luma samples for subsampling neighboring luma samples using the examples of techniques described in this
Petition 870190053680, of 12/06/2019, p. 80/171
76/119 development, and determining a plurality of subsampled luma samples based on the neighboring luma samples sought. The coding unit based on LM 122 can determine one or more scheduling parameters based on the subsampled luma samples (for example, α and β, described above). The coding unit based on LM 122 can determine a predictive block based on one or more scheduling parameters. For example, the coding unit based on LM 122 can determine * recL (i, j) + β, where recL (ij) is a subsampled version of the luma block that corresponds to the chroma block, and α and β are escalation factors determined from sub-sampled neighboring luma samples. The LM 122-based coding unit can sub-sample neighboring luma samples according to one or more examples of techniques described in this disclosure as techniques for limiting access to neighboring luma samples stored outside the line buffer.
[00164] The video encoder 20 can then encode the chroma block by LM prediction based on the predictive block. For example, as described below, video encoder 20 can subtract the predictive block from the chroma block to generate a residual block used by the video decoder 30 to reconstruct the chroma block.
[00165] In the techniques described in this disclosure, the LM 122-based coding unit can implement one or more of the above technical examples as part of the subsampling of the luma samples from a luma block.
Petition 870190053680, of 12/06/2019, p. 81/171
77/119
As an example, the LM 122-based coding unit can determine a luma block that corresponds to a chroma block being encoded, where a coordinate of an upper left sample of the luma block is (xO, yO), determine the luma samples, stored in DPB 116 or video data memory 101, for subsampling, with the determined luma samples excluding luma samples that have an x coordinate less than xO and a y coordinate less than yO, subsample the luma block based on the determined luma samples, determine a predictive block based on the subsampled luma block and code by linear model prediction (LM) the chroma block based on the predictive block. For example, determining luma samples for subsampling includes excluding luma samples that have an x coordinate less than xO and a y coordinate less than yO.
[00166] For example, the coding unit based on LM 122 can search for neighboring luma samples for the subsampling of neighboring luma samples. The neighboring luma samples sought include a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of an upper left luma sample of the luma block. For example, the luma samples illustrated in Figure 9 (for example, to the left of the dashed line) that are above and to the left of an upper left luma sample from the luma block are excluded from the luma samples that are searched for.
[00167] As another example, the unit of
Petition 870190053680, of 12/06/2019, p. 82/171
78/119 coding based on LM 122 can determine a luma block that corresponds to a chroma block being encoded, in which a coordinate of an upper left sample of the luma block is (xO, yO), determine the samples of luma, stored in DPB 116 or video data memory 101, for subsampling, with determined luma samples excluding luma samples that have an x coordinate less than (χθ-k) and a y coordinate greater than or equal to yO , where k is an integer greater than 0, subsample the luma block based on the determined luma samples, determine a predictive block based on the subsampled luma block and code by linear model prediction (LM) the chroma block based on the predictive block. For example, determining luma samples for subsampling includes excluding luma samples that have an x coordinate less than (χθ-k) and a y coordinate greater than or equal to yO.
[00168] For example, the coding unit based on LM 122 can search for neighboring luma samples for the subsampling of neighboring luma samples. The neighboring luma samples sought include a plurality of luma samples that are to the left of the luma block and exclude luma samples that are to the left and that are more than a limit number of samples to the left of the luma block and below a top left luma sample from the luma block. For example, the luma samples illustrated in Figure 13 (for example, to the left of the dashed line) that are more than a limit number of samples to the left of the luma block (for example, more than 4 samples
Petition 870190053680, of 12/06/2019, p. 83/171
79/119 on the left) and below the upper left sample of the luma block are excluded from the luma samples that are sought.
[00169] In one case, the coding unit based on LM 122 can apply a first filter (for example, that of JCTVC-F502) when no sample of luma having an x coordinate less than xO and a y coordinate less than yO is required for subsampling according to the first filter, and apply a different second filter (for example, that of JCTVC-E266 or equation 13) when luma samples that have an x coordinate less than xO and a y coordinate less than yO are necessary for subsampling according to the first filter. In these examples, in another case, the coding unit based on LM 122 can determine whether applying a filter could result in the use of luma samples that have an x coordinate less than xO and a y coordinate less than yO, generating values of luma for luma samples that have x coordinate less than xO and y coordinate less than yO without looking for luma values for luma samples. For example, the LM-based coding unit can populate values based on adjacent luma samples as shown in Figure 12B, and apply a filter using the generated luma values to subsample the luma block, where the filter is the same filter used for subsampling if the application of the filter could not result in the use of luma samples with an x coordinate less than xO and a y coordinate less than yO.
[00170] For luma samples neighboring the
Petition 870190053680, of 12/06/2019, p. 84/171
80/119 left, the LM 122-based coding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsampled from the plurality of subsampled luma samples, when no luma sample that is above and to the left of the upper left sample is necessary for subsampling according to the first filter. The LM 122-based coding unit can apply a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper left sample of the block of luma is necessary for subsampling according to the first filter.
[00171] For neighboring luma samples on the left, the LM 122-based coding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first lumen sample from the plurality of lumen samples , when no luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter. The first sample of a subsampled luma is in a column that is a first number of columns to the left of the luma block. The encoding unit based on LM 122 can apply a second
Petition 870190053680, of 12/06/2019, p. 85/171
81/119 different filter to a second set of neighboring luma samples sought to determine a second luma sample subsampled from the plurality of subsampled luma samples, when at least one luma sample is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block is necessary for subsampling according to the first filter. The second sample of a subsampled luma is in a column that is a second number of columns to the left of the luma block, and the second number is greater than the first number.
[00172] In an example, for the neighboring luma samples above, the LM 122-based encoding unit can generate, without searching from the video data memory 101, the luma values corresponding to the luma samples located above and to the left of the upper left luma sample of the luma block, and apply a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples. In one example, for the neighboring luma samples on the left, the LM 122-based encoding unit can generate, without searching from the video data memory 101, the luma values that correspond to more than one luma samples located the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block, and apply a filter to the generated luma values to determine at least one luma sample subsampled from the plurality of subsampled luma samples.
Petition 870190053680, of 12/06/2019, p. 86/171
82/119 [00173] The intraprediction processing unit 126 can generate predictive data for a PU by performing intraprediction on the PU. Predictive data for the PU can include predictive blocks for the PU and various syntax elements. The intraprediction processing unit 126 can perform intraprediction on PUs in slices I, slices P and slices B.
[00174] To perform intraprediction on a PU, the intraprediction processing unit 126 can use multiple intraprediction modes to generate multiple predictive data sets for the PU. The intraprediction processing unit 126 can use sample block samples from neighboring PUs to generate a predictive block for a PU. Neighboring PUs can be above, above and to the right, above and to the left and left, or to the left of the PU, assuming a top to bottom coding order from left to right for PUs, CUs and CTUs. The intraprediction processing unit 126 can use various numbers of intraprediction modes, for example, 33 directional intraprediction modes. In some instances, the number of intrapredictive modes may depend on the size of the PU-associated region.
[00175] The prediction processing unit 100 can select the predictive data for PUs from a CU among the predictive data generated by the interpretation processing unit 120 for the PUs, the predictive data generated by the intraprediction processing unit 126 for the PUs or the predictive data generated by the LM 122-based coding unit.
Petition 870190053680, of 12/06/2019, p. 87/171
83/119 For some examples, the prediction processing unit 100 selects predictive data for CU PUs based on the rate / distortion metric of the predictive data sets. The predictive blocks of the selected predictive data can be called selected predictive blocks.
[00176] Residual generation unit 102 can generate, based on the CU coding blocks of luma, Cb and Cr and the predictive luma, Cb and Cr blocks selected from CU's PU, the residual luma blocks, Cb and Cr da CU. For example, residual generation unit 102 can generate residual CU blocks so that each sample in the residual blocks has a value equal to a difference between a sample in a CU coding block and a corresponding sample in a selected predictive block corresponding value of a CU PU.
[00177] Transform processing unit 104 can perform quadtree partitioning to partition residual blocks associated with a CU into transform blocks associated with CU's TUs. Thus, a TU can be associated with a transform block and two chroma transform blocks. The sizes and positions of the CU luma and chroma transform blocks of a CU may or may not be based on the sizes and positions of the CU PU prediction blocks. A quadtree structure known as a residual quadtree (RQT) can include nodes associated with each of the regions. The CU's TUs can correspond to RQT leaf nodes.
Petition 870190053680, of 12/06/2019, p. 88/171
84/119 [00178] Transform processing unit 104 can generate transform coefficient blocks for each TU of a CU by applying one or more transforms to the transform blocks of the TU. The transform processing unit 104 can apply several transforms to a transform block associated with a TU. For example, transform processing unit 104 can apply a discrete cosine transform (DCT), a directional transform or a transform conceptually similar to a transform block. In some examples, transform processing unit 104 does not apply transforms to a transform block. In such examples, the transform block can be treated as a transform coefficient block.
[00179] The quantization unit 106 can quantize the transform coefficients in a coefficient block. The quantization process can reduce the bit depth associated with some or all of the transform coefficients. For example, a n-bit transform coefficient can be rounded down to a m-bit transform coefficient during quantization, where n is greater than m. The quantization unit 106 can quantize a coefficient block associated with a CU's TU based on a quantization parameter (QP) value associated with the CU. The video encoder 20 can adjust the degree of quantization applied to the coefficient blocks associated with a CU, by adjusting the QP value associated with the CU. Quantization can introduce loss of information; thus, the coefficients of
Petition 870190053680, of 12/06/2019, p. 89/171
85/119 quantized transformations may have a lower accuracy than the originals.
[00180] The reverse guanting unit 108 and the reverse transform processing unit 110 can apply reverse quantization and inverse transforms to a coefficient block, respectively, to reconstruct a residual block of the coefficient block. The reconstruction unit 112 can add the reconstructed residual block to corresponding samples of one or more predictive blocks generated by the prediction processing unit 100 to produce a reconstructed transform block associated with a TU. With the reconstruction of transform blocks for each CU of a CU in this way, the video encoder 20 can reconstruct the CU coding blocks.
[00181] Filter unit 114 can perform one or more unlock operations to reduce blocking artifacts in the coding blocks associated with a CU. DPB 116 can store the reconstructed coding blocks after the filter unit 114 performs one or more unlock operations on the reconstructed coding blocks. Interpretation processing unit 120 may use a reference image containing the reconstructed coding blocks to perform PU interpretation of other images. In addition, the intraprediction processing unit 126 can use reconstructed encoding blocks in the decoded image buffer 116 to perform intraprediction on other PUs in the same image as the CU. In addition, the
Petition 870190053680, of 12/06/2019, p. 90/171
86/119 coding based on LM 122 can use the reconstructed luma blocks in DPB 116 for linear model (LM) prediction that encodes a chroma block (where the luma block can include video data in some examples or can be a residual luma block and the chroma block can include video data in some examples or it can be a residual chroma block).
[00182] Entropy coding unit 118 can receive data from other functional components of video encoder 20. For example, entropy coding unit 118 can receive coefficient blocks from quantization unit 106 and can receive syntax elements from prediction processing unit 100. Entropy coding unit 118 can perform one or more entropy coding operations on the data to generate entropy encoded data. For example, the entropy coding unit 118 can perform a context-adaptive variable-length coding operation (CAVLC), a CAB CA operation, a variable-length to variable coding operation (V2V), a binary arithmetic coding operation adaptive to the context based on syntax (SB AC) a probability interval partitioning (PIPE) entropy coding operation, an Exponential-Golomb coding operation or other type of entropy coding operation on the data. The video encoder 20 can send a bit stream that includes entropy-encoded data generated by the entropy-encoding unit 118. For example, the
Petition 870190053680, of 12/06/2019, p. 91/171
87/119 bits can include data that represents an RQT of a CU.
[00183] Figure 3 is a block diagram illustrating an example of video decoder 30 that is configured to implement the techniques of this disclosure. Figure 3 is provided for explanatory purposes and should not limit the techniques as widely exemplified and described in this disclosure. For the sake of explanation, this disclosure describes the video decoder 30 in the context of HEVC encoding. However, the techniques of this disclosure may be applicable to other standards or coding methods.
[00184] Video encoder 30 represents an example of a device that can be configured to perform techniques for LM-based video encoding according to the various examples described in this disclosure. For example, video encoder 30 can be configured to encode one or more blocks using an LM video encoding mode (i.e., decoding by linear model prediction (LM) of one or more blocks).
[00185] In the example of Figure 3, the video decoder 30 includes an entropy decoding unit 150, video data memory 151, a prediction processing unit 152, an inverse quantization unit 154, a data processing unit inverse transformed 156, a reconstruction unit 158, a filter unit 160 and a decoded image buffer (DPB) 162. The prediction processing unit 152 includes the motion compensation unit 164 and an intraprediction processing unit 166. The decoder of
Petition 870190053680, of 12/06/2019, p. 92/171
88/119 video 30 also includes a linear model-based (LM) decoding unit 165 configured to perform various aspects of the LM-based encoding techniques described in this disclosure. In other examples, the video decoder 30 may include more, less or different functional components.
[00186] The video data memory 151 can store video data, such as an encoded video bit stream, which will be decoded by the video decoder components 30. The video data stored in the video data memory 151 can be obtained, for example, from a computer-readable medium 16 (for example, from a local video source, such as a camera, through wired or wireless network communication of video data, or by accessing storage media physical data). The video data memory 151 can form an encoded image buffer (CPB) that stores encoded video data from an encoded video bit stream. The DPB 162 can be a reference image memory that stores video data from reference for use in decoding video data by video decoder 30, (for example, in intra or intercoding modes). The video data memory 151 and DPB 162 can be formed by any one of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM ( RRAM), or other types of memory devices. Video data memory 151 and DPB 162 can be provided by the same
Petition 870190053680, of 12/06/2019, p. 93/171
89/119 memory device or separate memory devices. In several examples, the video data memory 151 can be on the chip with other video decoder components 30, or off the chip with respect to those components.
[00187] In this disclosure, the reference to video data memory 151 should not be construed as limited to the internal memory of the video decoder 30, unless specifically described as such, or memory external to the video decoder 30, unless specifically described as such. The reference to the video data memory 151 is to be understood as a reference memory that stores video data that the video decoder 30 receives for decoding (for example, video data for a current block that must be encoded). The video data memory 151 can also provide temporary storage of outputs of the various video decoder units 30.
[00188] As an example, the video data memory 151 is an example of a line buffer that is internal to the video decoder 30 and stores samples from neighboring blocks that are adjacent to the current block being decoded. As another example, part of DPB 162 can be the line buffer that is internal to the video decoder 30, and part of DPB 162 can be the memory that is external to the video decoder 30, as part of the system memory of the video chip. IC that includes the video decoder 30. As another example, the line buffer may be the cache memory of the video decoder 30.
Petition 870190053680, of 12/06/2019, p. 94/171
90/119 [00189] The various units in Figure 3 are illustrated to help understand the operations performed by the video decoder 30. The units can be implemented as fixed-function circuits, programmable circuits, or a combination of them. Fixed-function circuits refer to circuits that provide specific functionality, and are present in operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, and provide flexible functionality in the operations that can be performed. For example, programmable circuits can run software or firmware that causes programmable circuits to operate in the manner defined by instructions in the software or firmware. Fixed-function circuits can execute software instructions (for example, to receive parameters or output parameters), but the types of operations that fixed-function circuits perform are generally immutable. In some examples, the one or more units can be separate circuit blocks (fixed or programmable function), and in some examples, the one or more units can be integrated circuits.
[00190] The video encoder 20 can include logic and arithmetic units (ALUs), elementary function units (EFUs), digital circuits, analog circuits and / or programmable cores, formed from programmable circuits. In examples where video decoder operations 30 are performed by software executed by programmable circuits, the data memory
Petition 870190053680, of 12/06/2019, p. 95/171
91/119 of video 151 can store the software object code that video decoder 30 receives and executes, or other memory (not shown) can store such instructions.
[00191] An encoded image buffer (CPB) can receive and store encoded video data (for example, NAL units) from a bit stream. The entropy decoding unit 150 can receive encoded video data (e.g., NAL units) from the CPB and analyze the NAL units to decode syntax elements. The entropy decoding unit 150 can decode the entropy encoded syntax elements in the NAL units. Prediction processing unit 152, inverse quantization unit 154, inverse transform processing unit 156, reconstruction unit 158, and filter unit 160 can generate decoded video data based on the syntax elements extracted from the bit stream.
[00192] Bit stream NAL units may include encoded slice NAL units. As part of the bit stream decoding, the entropy decoding unit 150 can extract and entropy decode the syntax elements from the encoded slice NAL units. Each of the encoded slices can include a slice header and slice data. The slice header can contain syntax elements belonging to a slice. The syntax elements in the slice header can include a syntax element that identifies a PPS associated with an image that contains the slice.
[00193] In addition to decoding elements of
Petition 870190053680, of 12/06/2019, p. 96/171
92/119 bit stream syntax, the video decoder 30 can perform a reconstruction operation on an unpartitioned CU. To perform the reconstruction operation on a non-partitioned CU, the video decoder 30 can perform a reconstruction operation on each CU of the CU. When performing the reconstruction operation for each CU of the CU, the video decoder 30 can reconstruct residual CU blocks.
[00194] As part of performing a reconstruction operation on a CU's TU, the inverse quantization unit 154 can inversely quantize, that is, decantify, coefficient blocks associated with the TU. The inverse quantization unit 154 can use a QP value associated with the CU of the TU to determine a degree of quantization and, similarly, a degree of inverse quantization for the inverse quantization unit 154 to be applied. That is, the compaction rate, that is, the ratio of the number of bits used to represent the original and the compressed sequence, can be controlled by adjusting the value of the QP used when the transform coefficients are quantized. The compaction ratio can also depend on the entropy coding method employed.
[00195] After the inverse quantization unit 154 performs the inverse quantization of a coefficient block, the inverse transform processing unit 156 can apply one or more inverse transforms to the coefficient block in order to generate a residual block associated with the TU. For example, the
Petition 870190053680, of 12/06/2019, p. 97/171
93/119 inverse transform 156 can apply an inverse DCT, an entire inverse transform, a Karhunen-Loeve inverse transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform for the coefficient block.
[00196] If a PU is encoded using intraprediction, the intraprediction processing unit 166 can perform intraprediction to generate predictive blocks for the PU. The intraprediction processing unit 166 can use an intraprediction mode to generate the predictive blocks of luma, Cb and Cr for the PU based on the prediction blocks of spatially neighboring PUs. The intraprediction processing unit 166 can determine the intraprediction mode for the PU based on one or more syntax elements decoded from the bit stream.
[00197] The prediction processing unit 152 can build a first list of reference images (RefPicList0) and a second list of reference images (RefPicList1) based on the syntax elements extracted from the bit stream. In addition, if a PU is encoded using interpredition, the entropy decoding unit 150 can extract the motion information for the PU. The motion compensation unit 164 can determine, based on the movement information of the PU, one or more reference regions for the PU. The motion compensation unit 164 can generate, based on sample blocks in the one or more reference blocks for the PU, predictive luma blocks, Cb and Cr
Petition 870190053680, of 12/06/2019, p. 98/171
94/119 for the PU.
[00198] The reconstruction unit 158 can use the luma transform blocks, Cb and Cr associated with CU's TUs and the luma, Cb and Cr predictive blocks of CU's PUs, that is, intrapredictive data or data of interpretation, as applicable, to reconstruct the luma, Cb and Cr encoding blocks of CU. For example, the reconstruction unit 158 can add samples from the luma transform blocks, Cb and Cr to corresponding samples from the luma, Cb and Cr predictive blocks to reconstruct CU's luma, Cb and Cr coding blocks.
[00199] Filter unit 160 can perform an unlocking operation to reduce the blocking artifacts associated with CU's luma, Cb and Cr coding blocks. The video decoder can store the CU luma, Cb and Cr encoding blocks in the decoded image buffer 162. The decoded image buffer 162 can provide images for subsequent motion compensation, intraprediction and presentation on a display device, such as display device 32 of Figure 1. For example, the video decoder 30 can perform, on the basis of the luma blocks, Cb and Cr in DPB 162, intrapredictive or interpreted operations on PUs of other CUs.
[00200] According to several examples of this disclosure, the video decoder 30 can be configured to perform encoding based on LM. The decoding unit based on LM 165, for example, can perform decoding by prediction by linear model (LM). Per
Petition 870190053680, of 12/06/2019, p. 99/171
95/119 example, the LM 165-based decoding unit can sub-sample the reconstructed luma samples from a luma block that corresponds to a current chroma block being decoded. The LM 165-based decoding unit can scale the reconstructed luma samples subsampled from the luma block to generate a predictive block. The reconstruction unit 158 can then add the generated predictive block to the decoded residual data for the chroma block in order to reconstruct the chroma block. In some instances, the LM 165-based decoding unit may apply such techniques as part of residual cross-component prediction. In that case, the chroma block is a residual chroma block and the luma block is a residual luma block.
[00201] For example, the decoding unit based on LM 165 can determine a luma block that corresponds to a chroma block. The LM 165-based decoding unit can search for neighboring luma samples for subsampling neighboring luma samples using the examples of techniques described in this disclosure, and determining a plurality of subsampled luma samples based on the searched neighboring luma samples. The LM 165-based decoding unit can determine one or more scheduling parameters based on the subsampled luma samples (for example, α and β, described above). The LM 165-based decoding unit can determine a predictive block based on one or more scheduling parameters. For example, the LM 165-based decoding unit can determine the * rec _L (i, j)
Petition 870190053680, of 12/06/2019, p. 100/171
96/119 + β, where recL (ij) is a subsampled version of the luma block that corresponds to the chroma block, and α and β are scaling factors determined from neighboring sub-sampled luma samples. The LM 165-based decoding unit can sub-sample neighboring luma samples according to one or more examples of techniques described in this disclosure as techniques for limiting access to neighboring luma samples stored outside the line buffer.
[00202] The video decoder 30 can then decode the chroma block by LM prediction based on the predictive block. For example, as described below, the video decoder 30 can add the predictive block to a residual block to reconstruct the chroma block.
[00203] In the techniques described in this disclosure, the LM 165-based decoding unit can implement one or more of the above technical examples as part of the subsampling of the luma samples from a luma block. As an example, the LM 165-based decoding unit can determine a luma block that corresponds to a chroma block being encoded, where a coordinate of an upper left sample of the luma block is (x0, y0), determine the luma samples, stored in DPB 162 or video data memory 151, for subsampling, with the determined luma samples excluding luma samples that have an x coordinate less than xO and a y coordinate less than yO, subsample the luma block based on the luma samples
Petition 870190053680, of 12/06/2019, p. 101/171
97/119 determined, determine a predictive block based on the subsampled luma block and perform the prediction by linear model (LM) to decode the chroma block based on the predictive block. For example, determining luma samples for subsampling includes excluding luma samples that have an x coordinate less than xO and a y coordinate less than yO.
[00204] For example, the LM 165-based decoding unit can fetch neighboring luma samples for sub-sampling of neighboring luma samples. The neighboring luma samples sought include a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of an upper left luma sample of the luma block. For example, the luma samples illustrated in Figure 9 (for example, to the left of the dashed line) that are above and to the left of an upper left luma sample from the luma block are excluded from the luma samples that are searched for.
[00205] As another example, the LM 165-based decoding unit can determine a luma block that corresponds to a chroma block being encoded, where a coordinate of an upper left sample of the luma block is (x0, y0), determine the luma samples, stored in DPB 162 or video data memory 151, for subsampling, and the determined luma samples exclude the luma samples that have an x coordinate less than (χθ-k) and a y coordinate greater than or equal to yO, where k is an integer greater than
Petition 870190053680, of 12/06/2019, p. 102/171
98/119 than 0, subsample the luma block based on the determined luma samples, determine a predictive block based on the subsampled luma block and decode by chromatic model prediction (LM) the chroma block based on the predictive block. For example, determining luma samples for subsampling includes excluding luma samples that have an x coordinate less than (χθ-k) and a y coordinate greater than or equal to yO.
[00206] For example, the LM 165-based decoding unit can fetch neighboring luma samples for sub-sampling of neighboring luma samples. The neighboring luma samples sought include a plurality of luma samples that are to the left of the luma block and exclude luma samples that are to the left and that are more than a limit number of samples to the left of the luma block and below a top left luma sample from the luma block. For example, the luma samples shown in Figure 13 (for example, to the left of the dashed line) that are more than a limit number of samples to the left of the luma block (for example, more than 4 samples to the left) and below the upper left sample of the luma block are excluded from the luma samples that are fetched.
[00207] In one case, the LM 165-based decoding unit can apply a first filter (for example, that of JCTVC-F502) when no luma sample that has an x coordinate less than xO and a y coordinate less than yO is necessary for subsampling according to the first filter, and apply a second filter
Petition 870190053680, of 12/06/2019, p. 103/171
99/119 different (for example, that of JCTVC-E266 or equation 13) when luma samples that have an x coordinate less than xO and a y coordinate less than yO are required for subsampling according to the first filter. In these examples, in another case, the LM 165-based decoding unit can determine whether applying a filter could result in the use of luma samples that have an x coordinate less than xO and a y coordinate less than yO, generating values of luma for luma samples that have x coordinate less than xO and y coordinate less than yO without fetching luma values for luma samples (for example, fill values based on adjacent luma samples as shown in Figure 16B) , and apply a filter using the generated luma values to subsample the luma block, where the filter is the same filter used for subsampling neighboring luma samples if applying the filter could not result in the use of luma samples with an x coordinate less than xO and a y coordinate less than yO.
[00208] For neighboring luma samples on the left, the LM 165-based decoding unit can apply a first filter to a first set of neighboring luma samples searched to determine a first subsampled luma sample from the plurality of subsampled luma samples , when no luma sample that is above and to the left of the upper left luma sample is required for subsampling according to the first filter. The LM 165-based decoding unit can apply a second, different filter
Petition 870190053680, of 12/06/2019, p. 104/171
100/119 to a second set of neighboring luma samples sought to determine a second luma sample subsampled from the plurality of luma samples subsampled, when at least one luma sample is above and to the left of the upper left luma sample of the block of luma is required for subsampling according to the first filter.
[00209] For neighboring luma samples on the left, the LM 165-based decoding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first lumen sample from the plurality of lumen samples , when no luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter. The first sample of a subsampled luma is in a column that is a first number of columns to the left of the luma block. The LM 165-based decoding unit can apply a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples from which at least one luma sample is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block is necessary for subsampling according to the first filter. The second sample of a subsampled luma is in a column which is a second number of
Petition 870190053680, of 12/06/2019, p. 105/171
101/119 columns to the left of the luma block, and the second number is greater than the first number.
[00210] In one example, for the neighboring luma samples above, the LM 165-based decoding unit can generate, without searching from the video data memory 151, the luma values corresponding to the luma samples located above and to the left of the upper left luma sample of the luma block, and apply a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples. In one example, for the neighboring luma samples on the left, the LM 165-based decoding unit can generate, without searching from the video data memory 151, the luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block, and apply a filter to the generated luma values to determine at least one luma sample subsampled from the plurality of subsampled luma samples.
[00211] Figure 18 is a flow chart illustrating an example of a method of decoding video data. As illustrated, the LM 165-based decoding unit can determine a luma block that corresponds to a chroma block (180). As described above, a block can include a luma block and two chroma blocks. A luma block that corresponds to a chroma block means that the luma block and the corresponding chroma block are from the same block.
Petition 870190053680, of 12/06/2019, p. 106/171
102/119 [00212] The LM 165-based decoding unit can fetch neighboring luma samples for the subsampling of neighboring luma samples (182). In one or more examples, the LM 165-based decoding unit can exclude certain neighboring luma samples from the search. Luma samples that are not searched for can be described in relation to the upper left sample of a luma block that has the coordinate (xO, yO).
[00213] For example, for the example where neighboring luma samples are above the luma block, the LM 165-based decoding unit can exclude luma samples that are above and to the left of an upper left luma sample. of the luma pad. As an example, the luma samples to the left of the dashed line in Figure 9 may not be searched, as they are above and to the left of an upper left luma sample of the luma block. In this example, the neighboring luma samples searched exclude luma samples that have an x coordinate less than xO and a y coordinate less than yO. Instead, luma samples such as those illustrated in Figures 10, 11A and 11B can be searched.
[00214] For the example where neighboring luma samples are to the left of the luma block, the LM 165-based decoding unit can exclude luma samples that are on the left and which are more than a limit number of samples to the left of the luma block and below a sample of the upper left luma block. As an example, the luma samples to the left of the dashed lines in Figure 13 may not be searched, as
Petition 870190053680, of 12/06/2019, p. 107/171
103/119 they are more than a limit number of samples (for example, more than 4 samples) to the left of the luma block and below an upper left luma sample of the luma block. In this example, the neighboring luma samples searched exclude luma samples that have an x coordinate less than (xO-k), and a y coordinate greater than yO, where k is the limit number and an integer greater than 0 ( for example, 4). Instead, luma samples such as those illustrated in Figures 14, 15A and 15B can be searched.
[00215] The decoding unit based on LM 165 can determine a plurality of luma samples subsampled based on the neighboring luma samples searched (184). For the example where neighboring samples are above the luma block, one of the subsampled luma samples corresponds to a subsampled luma sample immediately above the upper left luma sample. For example, as shown in Figures 10, 11B and 12B, the subsampled luma sample is immediately above the upper left luma sample. For the example where neighboring samples are to the left of the luma block, one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block. Figures 11A and 12A illustrate an additional example of a subsampled luma sample. For example, as shown in Figures 14, 15B and 16B, the subsampled luma sample is 4 samples to the left of the luma block, and in this example, the limit number of samples to the left of the
Petition 870190053680, of 12/06/2019, p. 108/171
104/119 luma block is 4. In these examples, the subsampled luma sample is more than two columns to the left of the upper left luma sample (for example, the subsampled luma sample is 4 columns to the left, which is more than 2 columns on the left). Figures 15A and 16A illustrate a further example of a subsampled luma sample.
[00216] For the example where neighboring luma samples are above the luma block, the LM 165-based decoding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsample of the plurality of subsampled luma samples, when no luma sample that is above and to the left of the upper left luma sample is required for subsampling according to the first filter. For example, the LM 165-based decoding unit can apply a filter like those of JCTVC-F502 (as an example), and as illustrated in Figures 11A and 12A. The LM 165-based decoding unit can apply a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper left luma sample of the luma block if necessary (for example, it could be necessary if it is first used) for subsampling according to the first filter. For example, if using the filter described in JCTVC-F502
Petition 870190053680, of 12/06/2019, p. 109/171
105/119 would result in access to luma samples that are above and to the left of the upper left sample, so the LM 165-based decoding unit can apply a filter like the filter described in JCTVC-E266 (for example, Figure 11B) or according to equation (13) described above (for example, Figure 10).
[00217] For the example where the neighboring luma samples to the left of the luma block, the LM 165-based decoding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsample of the plurality of luma samples subsampled, when no luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter. For example, the LM 165-based decoding unit can apply a filter like those of JCTVC-F502 (as an example), and as illustrated in Figures 15A and 16A. The LM 165-based decoding unit can apply a different second filter to a second set of neighboring luma samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block is necessary (for example, it could be necessary if it is first used) for subsampling according to the first filter. Per
Petition 870190053680, of 12/06/2019, p. 110/171
106/119 example, if using the filter described in JCTVC-F502 would result in access to luma samples that are more than the limit number of samples to the left of the luma block and below the upper left sample, then the decoding unit based on LM 165 can apply a filter like the filter described in JCTVC-E266 (for example, as shown in Figure 15B) or according to equation (13) described above (for example, as shown in Figure 14).
[00218] In the examples above, the first filter uses three luma samples from a first line that is above the luma block, and uses three luma samples from a second line that is above the first line. The second filter uses less than three luma samples from a first line (for example, two samples in Figures 9 and 14, and one sample in Figures 11B and 15B) which is to the left of the luma block, and uses less than three samples of a second line luma (for example, two samples in Figures 9 and 14, and a sample in Figures 11B and 15B) which is below the first line.
[00219] In some examples, instead of using different filters, the decoding unit based on LM 165 can use the same filter, but apply the padding (for example, generate luma sample values) instead of fetching. For example, for the examples of neighboring luma samples above, the LM 165-based decoding unit can generate, without searching, luma values that correspond to luma samples located above and to the left of the upper left luma sample of the block.
Petition 870190053680, of 12/06/2019, p. 111/171
107/119 luma, and can apply a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples (for example, the example in Figure 12B). For the examples of luma samples neighboring on the left, the LM 165-based decoding unit can generate, without searching,
values in luma what correspond to samples of luma located more than the limit number of samples to left of block in luma and below gives sample of luma higher left of block of luma, , and can apply a filter to values of luma generated for to determine fur
at least one luma sample subsampled from the plurality of luma samples subsampled (for example, the example of 16B).
[00220] The decoding unit based on LM 165 can determine one or more scheduling parameters based on the subsampled luma samples (186). For example, the LM 165-based decoding unit can determine α and β based on the subsampled luma samples. The LM 165-based decoding unit can determine a predictive block based on one or more scheduling parameters (188). For example, the LM 165-based decoding unit can determine the predictive block as * recL (i, j) + β, where recL (ij) is a subsampled version of the luma block that corresponds to the
chroma block, and α and β are factors escalation determined to from samples of luma neighbors subsampled. [00221] 0 decoder video 30 may
Petition 870190053680, of 12/06/2019, p. 112/171
108/119 decode by chroma prediction by LM based on the predictive block (190). For example, reconstruction unit 158 can add the predictive block to a residual block to reconstruct the chroma block.
[00222] Figure 19 is a flow chart illustrating an example of a method for encoding video data. As illustrated, the LM 122-based coding unit can determine a luma block that corresponds to a chroma block (200). As described above, a block can include a luma block and two chroma blocks. A luma block that corresponds to a chroma block means that the luma block and the corresponding chroma block are from the same block.
[00223] The coding unit based on LM 122 can search for neighboring luma samples for the subsampling of neighboring luma samples (202). In one or more examples, the LM 122-based coding unit may exclude certain neighboring luma samples from the search. Luma samples that are not searched for can be described in relation to the upper left sample of a luma block that has the coordinate (xO, yO).
[00224] For example, for the example where neighboring luma samples are above the luma block, the LM 122-based coding unit can exclude luma samples that are above and to the left of an upper left luma sample. of the luma pad. As an example, the luma samples to the left of the dashed line in Figure 9 may not be searched, as they are above and to the left of an upper left luma sample.
Petition 870190053680, of 12/06/2019, p. 113/171
109/119 of the luma block. In this example, the neighboring luma samples searched exclude luma samples that have an x coordinate less than xO and a y coordinate less than yO. Instead, luma samples such as those illustrated in Figures 10, HA and 11B can be searched.
[00225] For the example where the neighboring luma samples are to the left of the luma block, the LM 122-based coding unit can exclude the luma samples that are on the left and which are more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block. As an example, the luma samples to the left of the dashed lines in Figure 13 may not be searched, as they are more than a limit number of samples (for example, more than 4 samples) to the left of the luma block and below of a top left luma sample from the luma block. In this example, the neighboring luma samples searched exclude luma samples that have an x coordinate less than (xO-k), and a y coordinate greater than yO, where k is the limit number and an integer greater than 0 ( for example, 4). Instead, luma samples such as those illustrated in Figures 14, 15A and 15B can be searched.
[00226] The coding unit based on LM 122 can determine a plurality of luma samples subsampled based on the luma samples sought next (204). For the example where neighboring samples are above the luma block, one of the subsampled luma samples corresponds to a luma sample
Petition 870190053680, of 12/06/2019, p. 114/171
110/119 subsampled immediately above the upper left luma sample. For example, as shown in Figures 10, 11B and 12B, the subsampled luma sample is immediately above the upper left luma sample. For the example where neighboring samples are to the left of the luma block, one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block. Figures 11A and 12A illustrate an additional example of a subsampled luma sample. For example, as shown in Figures 14, 15B and 16B, the subsampled luma sample is 4 samples to the left of the luma block, and in this example, the limit number of samples to the left of the luma block is 4. In these examples, the a subsampled luma sample is more than two columns to the left of the upper left luma sample (for example, the subsampled luma sample is 4 columns to the left, which is more than 2 columns to the left). Figures 15A and 16A illustrate an additional example of a subsampled luma sample.
[00227] For the example where the neighboring luma samples are above the luma block, the LM 122-based coding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsample of the plurality of subsampled luma samples, when no luma sample that is above and to the left of the upper left luma sample is required for subsampling according to the first
Petition 870190053680, of 12/06/2019, p. 115/171
111/119 filter. For example, the LM 122-based coding unit can apply a filter like those of JCTVC-F502 (as an example), and as illustrated in Figures 11A and 12A. The LM 122-based coding unit can apply a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper left luma sample of the luma block it could be necessary for subsampling according to the first filter. For example, if using the filter described in JCTVC-F502 would result in access to luma samples that are above and to the left of the upper left sample, then the LM 122-based coding unit can apply a filter like JCTVC-E266 (for example, example, Figure 11B) or according to equation (13) described above (for example, Figure 10).
[00228] For the example where the neighboring luma samples to the left of the luma block, the LM 122-based coding unit can apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsample of the plurality of sub-sampled luma samples, when no luma sample that is more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter. For example, the LM 122-based coding unit can apply a filter like those in JCTVC-F502 (as
Petition 870190053680, of 12/06/2019, p. 116/171
112/119 an example), and as illustrated in Figures 15A and 16A. The LM 122-based coding unit can apply a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples from the sample, when at least one luma sample is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block could be necessary for subsampling according to the first filter. For example, if using the filter described in JCTVC-F502 would result in access to luma samples that are more than the limit number of samples to the left of the luma block and below the upper left sample, then the LM 165-based decoding unit you can apply a filter like JCTVC-E266 (for example, Figure 15B) or according to equation (13) described above (for example, Figure 14).
[00229] In the examples above, the first filter uses three luma samples from a first line that is above the luma block, and uses three luma samples from a second line that is above the first line. The second filter uses less than three luma samples from a first line (for example, two samples in Figures 9 and 14, and one sample in Figures 11B and 15B) which is to the left of the luma block, and uses less than three samples of a second line luma (for example, two samples in Figures 9 and 14, and a sample in Figures 11B and 15B) which is below the first line.
[00230] In some examples, instead of using
Petition 870190053680, of 12/06/2019, p. 117/171
113/119 different filters, the LM 122-based coding unit can use the same filter, but apply the padding (for example, generate luma sample values) instead of fetching. For example, for the examples of neighboring luma samples above, the LM 122-based encoding unit can generate, without searching, luma values that correspond to luma samples located above and to the left of the upper left luma sample of the block. luma, and you can apply a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples (for example, the example in Figure 12B). For examples of luma samples neighboring on the left, the LM 122-based coding unit can generate, without searching, luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block, and you can apply a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples (e.g., example 16B).
[00231] The coding unit based on LM 122 can determine one or more scheduling parameters based on the subsampled luma samples (206). For example, the coding unit based on LM 122 can determine α and β based on the subsampled luma samples. The coding unit based on LM 122 can determine a predictive block based on one or more scheduling parameters (208). For example, the unit
Petition 870190053680, of 12/06/2019, p. 118/171
114/119 coding based on LM 122 can determine the block
predictive like * recL ( i, j) + β, where recL (ij) is a subsampled version of block of luma that run responds to chroma block, and α and β are factors of staggering determined to leave samples of luma neighbors subsampled.[00232] 0 encoder of video can code by prediction by LM o block of chroma with base on the block
predictive (210). For example, the residual generation unit 102 can subtract the predictive block from the chroma block to generate a residual block used by the video decoder 30 to reconstruct the chroma block.
[00233] The techniques described above can be performed by video encoder 20 (Figures 1 and 2) and / or video decoder 30 (Figures 1 and 3), both of which can generally be termed as a video encoder. Similarly, video encoding can refer to video encoding or video decoding, as applicable. In addition, video encoding and video decoding can generally be referred to as video data processing.
[00234] It should be understood that all the techniques described in this document can be used individually or in combination. This disclosure includes several signaling methods that can change depending on certain factors such as block size, type of slice, etc. Such variation in the signaling or inference of the syntax elements can be known for the a-priori encoder and decoder or can be signaled to the
Petition 870190053680, of 12/06/2019, p. 119/171
115/119 decoder explicitly in the video parameter set (VPS), sequence parameter set (SPS), image parameter set (PPS), slice header, on a tile level or elsewhere.
[00235] It should be recognized that, depending on the example, certain actions or events of any of the techniques described in this document can be performed in a different sequence, can be added, merged or ignored (for example, not all actions or described events are necessary for the practice of the techniques). In addition, in certain examples, actions or events can be executed simultaneously, for example, through multi-threaded processing, interrupt processing or multiple processors, instead of sequentially. In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for the sake of clarity, it should be understood that the techniques of this disclosure can be performed by a combination of units or modules associated with a video encoder.
[00236] Although specific combinations of various aspects of the techniques are described above, these combinations are provided merely to illustrate examples of the techniques described in this disclosure. Consequently, the techniques of this disclosure should not be limited to these examples of combinations and may cover any conceivable combination of the various aspects of the techniques described in this disclosure.
[00237] In one or more examples, the functions
Petition 870190053680, of 12/06/2019, p. 120/171
116/119 described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, functions can be stored or transmitted as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium that corresponds to a tangible medium such as data storage media, or means of communication including any medium that facilitates the transfer of a computer program from one location to another, for example, according to a communication protocol. Thus, computer-readable media can, in general, correspond to (1) tangible computer-readable storage media that are non-temporary or (2) a communication medium such as a signal or carrier wave. The data storage means can be any available means that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.
[00238] By way of example, and not by way of limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other means that can
Petition 870190053680, of 12/06/2019, p. 121/171
117/119 can be used to store the desired program code in the form of instructions or data structures that can be accessed by a computer. Also, any connection is properly called a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and micro- waves, then, coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are instead directed to tangible, non-temporary storage media. The disk and floppy disk, as used herein, includes compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk on which floppy disks magnetically reproduce data, while disks reproduce optically with lasers. Combinations of these may also be included within the scope of computer-readable media.
[00239] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuit (ASICs), field programmable logic matrices (FPGAs) or other
Petition 870190053680, of 12/06/2019, p. 122/171
118/119 equivalent integrated or distinct logic circuitry. Consequently, the term processor, as used in this document, can refer to any prior structure or any other structure suitable for implementing the techniques described in this document. In addition, in some respects, the functionality described in this document may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
[00240] The techniques of this disclosure can be implemented in a wide variety of devices or devices, including a wireless device, an integrated circuit (TC) or a set of ICs (for example, a chip set). Various components, modules or units are described in this disclosure to emphasize functional aspects of devices configured to perform the techniques disclosed, but do not necessarily require realization by different hardware units. Instead, as described above, multiple units can be combined into one codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with appropriate software and / or firmware.
[00241] Several examples have been described. These and other examples are within the scope of the claims
Petition 870190053680, of 12/06/2019, p. 123/171
119/119 below.

权利要求:
Claims (67)
[1]
1. Method of decoding video data, the method comprising:
determining a luma block that corresponds to a chroma block;
search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of a top left luma sample from the luma block;
determining a plurality of luma samples subjected to descending sampling based on the neighboring luma samples sought, in which one of the luma samples subjected to descending sampling corresponds to a luma sample subjected to descending sampling immediately above the upper left luma sample;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and decode by chromatic model prediction (LM) the chroma block based on the predictive block.
[2]
2. Method according to claim 1, in which a coordinate of the upper left luma sample of the luma block is (xO, yO), and in which the neighboring luma samples sought exclude luma samples that have a
Petition 870190053680, of 12/06/2019, p. 125/171
2/26 x coordinate less than xO and a y coordinate less than yO.
[3]
A method according to claim 1, wherein determining the plurality of sub-sampled luma samples comprises:
apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsampled from the plurality of subsampled luma samples, when no luma sample that is above and to the left of the upper left luma sample is needed for subsampling according to the first filter; and applying a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper lumen sample left of the luma block is necessary for subsampling according to the first filter.
[4]
A method according to claim 3, wherein the first filter uses three luma samples from a first line that is above the luma block and three luma samples from a second line that is above the first line.
[5]
A method according to claim 3, wherein the second filter uses less than three luma samples from a first line that is above the luma block and less than three luma samples from a second line that is above the first line.
Petition 870190053680, of 12/06/2019, p. 126/171
3/26
[6]
A method according to claim 1, wherein determining the plurality of sub-sampled luma samples comprises:
generate, without searching, luma values that correspond to luma samples located above and to the left of the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[7]
A method according to claim 1, wherein the LM prediction that decodes the chroma block based on the predictive block comprises adding the predictive block to a residual block to reconstruct the chroma block.
[8]
8. Method according to claim 1, which further comprises:
decode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag decoding comprises decoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[9]
9. Device for decoding video data, the device comprising:
a video data memory; and a video decoder that comprises at least
Petition 870190053680, of 12/06/2019, p. 127/171
4/26 minus a fixed function circuit pack or programmable circuit pack, where the video decoder is configured to:
determining a luma block that corresponds to a chroma block;
search, from the video data memory, for neighboring luma samples to carry out the downward sampling of the neighboring luma samples, in which the searched neighboring luma samples comprise a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of an upper left luma sample of the luma block;
determining a plurality of luma samples subjected to descending sampling based on the neighboring luma samples sought, in which one of the luma samples subjected to descending sampling corresponds to a luma sample subjected to descending sampling immediately above the upper left luma sample;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and decode by chromatic model prediction (LM) the chroma block based on the predictive block.
[10]
A device according to claim 9, in which a coordinate of the upper left luma sample of the luma block is (xO, yO), and in which the luma samples
Petition 870190053680, of 12/06/2019, p. 128/171 neighbors searched exclude luma samples that have an x coordinate less than xO and a y coordinate less than yO.
[11]
11. Device according to claim 9, in which to determine the plurality of sub-sampled luma samples, the video decoder is configured to:
apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsampled from the plurality of subsampled luma samples, when no luma sample that is above and to the left of the upper left luma sample is needed for subsampling according to the first filter; and applying a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper lumen sample left of the luma block is necessary for subsampling according to the first filter.
[12]
Device according to claim 11, wherein the first filter uses three luma samples from a first line that is above the luma block and three luma samples from a second line that is above the first line.
[13]
Device according to claim 11, wherein the second filter uses less than three luma samples from a first line that is above the luma block and less than three luma samples from a second line
Petition 870190053680, of 12/06/2019, p. 129/171
6/26 which is above the first line.
[14]
14. Device according to claim 9, in which to determine the plurality of sub-sampled luma samples, the video decoder is configured to:
generate, without searching, luma values that correspond to luma samples located above and to the left of the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[15]
15. Device according to claim 9, wherein the LM prediction decodes the chroma block based on the predictive block, the video decoder is configured to add the predictive block to a residual block to reconstruct the chroma block.
[16]
16. Device according to claim 9, wherein the video decoder is configured to:
decode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag decoding comprises decoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[17]
17. Method of encoding video data, the method comprising:
determine a block of luma that corresponds to a
Petition 870190053680, of 12/06/2019, p. 130/171
7/26 chroma block;
search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of a top left luma sample from the luma block;
determining a plurality of luma samples subjected to descending sampling based on the neighboring luma samples sought, in which one of the luma samples subjected to descending sampling corresponds to a luma sample subjected to descending sampling immediately above the upper left luma sample;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and prediction by linear model (LM) that encodes the chroma block based on the predictive block.
[18]
18. The method of claim 17, wherein a coordinate of the upper left luma sample of the luma block is (xO, yO), and the neighboring luma samples sought exclude luma samples that have an x coordinate. less than xO and a y coordinate less than yO.
[19]
19. The method of claim 17, wherein determining the plurality of subsampled luma samples comprises:
Petition 870190053680, of 12/06/2019, p. 131/171
8/26 applying a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsampled from the plurality of luma samples subsampled, when no luma sample is above and to the left of the upper left luma sample it is necessary for subsampling according to the first filter; and applying a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper lumen sample left of the luma block could be necessary for subsampling according to the first filter.
[20]
20. The method of claim 19, wherein the first filter uses three luma samples from a first line that is above the luma block and three luma samples from a second line that is above the first line.
[21]
21. The method of claim 19, wherein the second filter uses less than three luma samples from a first line that is above the luma block and less than three luma samples from a second line that is above the first line.
[22]
22. The method of claim 17, wherein determining the plurality of subsampled luma samples comprises:
generate, without searching, luma values that
Petition 870190053680, of 12/06/2019, p. 132/171
9/26 correspond to luma samples located above and to the left of the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[23]
23. The method of claim 17, wherein the LM prediction that encodes the chroma block based on the predictive block comprises subtracting the predictive block from the chroma block to generate a residual block that will be used by a decoder of video, to reconstruct the chroma block.
[24]
24. The method of claim 17, which further comprises:
encode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag encoding comprises encoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[25]
25. Device for encoding video data, the device comprising:
a video data memory; and
A video encoder comprising at least one fixed-function circuitry or programmable circuitry, where the video encoder is configured to:
Petition 870190053680, of 12/06/2019, p. 133/171
10/26 determining a luma block that corresponds to a chroma block;
search, from the video data memory, for neighboring luma samples to carry out the downward sampling of the neighboring luma samples, in which the searched neighboring luma samples comprise a plurality of luma samples that are above the luma block and exclude luma samples that are above and to the left of an upper left luma sample of the luma block;
determining a plurality of luma samples subjected to descending sampling based on the determined neighboring luma samples, wherein one of the luma samples subjected to descending sampling corresponds to a luma sample subjected to descending sampling immediately above the upper left luma sample;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and code by prediction by linear model (LM) the chroma block based on the predictive block.
[26]
26. The device of claim 25, wherein a coordinate of the upper left luma sample of the luma block is (xO, yO), and in which the neighboring luma samples sought exclude luma samples that have an x coordinate less than xO and a y coordinate less than yO.
Petition 870190053680, of 12/06/2019, p. 134/171
11/26
[27]
27. Device according to claim 25, in which to determine the plurality of sub-sampled luma samples, the video encoder is configured to:
apply a first filter to a first set of neighboring luma samples sought to determine a first luma sample subsampled from the plurality of luma samples subsampled, when no luma sample that is above and to the left of the upper left luma sample is required for subsampling according to the first filter; and applying a different second filter to a second set of neighboring lumen samples sought to determine a second lumen sample from the plurality of lumen samples, when at least one lumen sample is above and to the left of the upper lumen sample left of the luma block is necessary for subsampling according to the first filter.
[28]
28. The device of claim 27, wherein the first filter uses three luma samples from a first line that is above the luma block and three luma samples from a second line that is above the first line.
[29]
29. The device of claim 27, wherein the second filter uses less than three luma samples from a first line that is above the luma block and less than three luma samples from a second line that is above the first line.
Petition 870190053680, of 12/06/2019, p. 135/171
12/26
[30]
30. Device according to claim 25, in which to determine the plurality of sub-sampled luma samples, the video encoder is configured to:
generate, without searching, luma values that correspond to luma samples located above and to the left of the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[31]
31. The device according to claim 25, wherein the LM prediction encodes the chroma block based on the predictive block, the video encoder is configured to subtract the predictive block from the chroma block to generate a residual block that will be used by a video decoder to reconstruct the chroma block.
[32]
32. Device according to claim
25, where the video encoder is configured to:
encode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag encoding comprises encoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[33]
33. Method of decoding video data, the method comprising:
Petition 870190053680, of 12/06/2019, p. 136/171
13/26 determining a luma block that corresponds to a chroma block;
search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are on the left and that they are more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block;
determining a plurality of subsampled luma samples based on the searched neighboring luma samples, where one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and prediction by linear model (LM) that decodes the chroma block based on the predictive block.
[34]
34. The method of claim 33, wherein a coordinate of the upper left luma sample of the luma block is (xO, yO), and the neighboring luma samples sought exclude luma samples that have an x coordinate. less than (χθ-k) and a y coordinate greater than yO, and where k is the limit number and k is an integer greater than 0.
Petition 870190053680, of 12/06/2019, p. 137/171
14/26
35. Method, in a deal with claim 34, in what k is equal to 4. 36. Method, in a deal with claim 33, in what one of samples ; in luma subsamples corresponds The an sample of luma subsample more than two columns The left of the upper luma sample left.37. Method, in a deal with claim 33, in what the determination gives plurality of luma samples subsampled comprises: apply a first filter to a first set
of the neighboring luma samples sought, to determine a first luma sample subsampled from the plurality of subsampled luma samples, when no luma sample that is more than the limit number of samples to the left of the luma block and below the luma sample upper left is necessary for subsampling according to the first filter, and where the first sample of a subsampled luma is in a column which is a first number of columns to the left of the luma block; and applying a second, different filter to a second set of neighboring luma samples searched, to determine a second luma sample subsampled from the plurality of luma samples subsampled, when at least one luma sample which is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block is required for subsampling according to the first filter, where the second subsampled luma sample is in a column which is a second number of columns to the left block
Petition 870190053680, of 12/06/2019, p. 138/171
15/26 luma, and where the second number is greater than the first number.
[35]
38. The method of claim 37, wherein the first filter uses three luma samples from a first line which is to the left of the luma block and three luma samples from a second line which is below the first line.
[36]
39. The method of claim 37, wherein the second filter uses less than three luma samples from a first line that is to the left of the luma block and less than three luma samples from a second line that is below the first line.
[37]
40. The method of claim 33, wherein determining the plurality of subsampled luma samples comprises:
generate, without searching, luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[38]
41. The method of claim 33, wherein the LM prediction that decodes the chroma block based on the predictive block comprises adding the predictive block to a residual block to reconstruct the chroma block.
[39]
42. The method of claim 33, which
Petition 870190053680, of 12/06/2019, p. 139/171
16/26 additionally comprises:
decode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag decoding comprises decoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[40]
43. Device for decoding video data, the device comprising:
a video data memory; and a video decoder comprising at least one fixed-function circuitry or programmable circuitry, wherein the video decoder is configured to:
determining a luma block that corresponds to a chroma block; search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are on the left and that they are more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block;
determine a plurality of subsampled luma samples based on the neighboring luma samples sought, in which one of the subsampled luma samples corresponds to a subsampled luma sample which is the
Petition 870190053680, of 12/06/2019, p. 140/171
17/26 limit number of samples to the left of the luma block;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and prediction by linear model (LM) that decodes the chroma block based on the predictive block.
44 Device, in a deal with the claim 43 in what a coordinate gives sample of upper luma left of luma block is (xO , yO), and in that samples
neighboring luma's excluded luma samples that have an x coordinate less than (xO-k) and a y coordinate greater than yO, and where k is the limit number and k is an integer greater than 0.
[41]
45. The device of claim 44, wherein k is equal to 4.
[42]
46. The device of claim 43, wherein one of the subsampled luma samples corresponds to a subsampled luma sample more than two columns to the left of the upper left luma sample.
[43]
47. Device according to claim
43, in which to determine the plurality of subsampled luma samples, the video decoder is configured to:
apply a first filter to a first set of neighboring luma samples, to determine a first subsampled luma sample from the plurality of subsampled luma samples, when no luma sample being
Petition 870190053680, of 12/06/2019, p. 141/171
18/26 more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter, and in which the first subsampled luma sample is in a column that it is a first number of columns to the left of the luma block; and applying a different second filter to a second set of neighboring luma samples, to determine a second luma sample subsampled from the plurality of subsampled luma samples, when at least one luma sample which is more than the limit number of samples to left of the luma block and below the upper left luma sample of the luma block is required for subsampling according to the first filter, where the second subsampled luma sample is in a column which is a second number of columns to the left of the luma block, and where the second number is greater than the first number.
[44]
48. The device of claim 47, wherein the first filter uses three luma samples from a first line which is to the left of the luma block and three luma samples from a second line which is below the first line.
[45]
49. The device of claim 47, wherein the second filter uses less than three luma samples from a first line that is to the left of the luma block and less than three luma samples from a second line that is below the first line.
[46]
50. Device according to claim 43, in which to determine the plurality of samples of
Petition 870190053680, of 12/06/2019, p. 142/171
19/26 luma subsamples, the video decoder is configured to:
generate, without searching, luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[47]
51. The device of claim 43, wherein the LM prediction decodes the chroma block, the video decoder is configured to add the predictive block to a residual block to reconstruct the chroma block.
[48]
52. Device according to claim
43, where the video decoder is configured to:
decode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and in which to decode the flag, the video decoder it is configured to decode the flag based on a context comprising one or more flags that indicate whether LM predictive encoding is enabled for neighboring blocks.
[49]
53. Method of encoding video data, the method comprising:
determine a block of luma that corresponds to a
Petition 870190053680, of 12/06/2019, p. 143/171
20/26 chroma block;
search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are on the left and that they are more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block;
determining a plurality of subsampled luma samples based on the searched neighboring luma samples, where one of the subsampled luma samples corresponds to a subsampled luma sample which is the limit number of samples to the left of the luma block;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and code by prediction by linear model (LM) the chroma block based on the predictive block.
[50]
54. The method of claim 53, wherein a coordinate of the upper left luma sample of the luma block is (xO, yO), and the neighboring luma samples sought exclude luma samples that have an x coordinate. less than (χθ-k) and a y coordinate greater than yO, and where k is the limit number and k is an integer greater than 0.
[51]
55. Method, according to claim 54, in
Petition 870190053680, of 12/06/2019, p. 144/171
21/26 that k is equal to 4.
[52]
56. The method of claim 53, wherein one of the subsampled luma samples corresponds to a subsampled luma sample more than two columns to the left of the upper left luma sample.
[53]
57. The method of claim 53, wherein determining the plurality of subsampled luma samples comprises:
apply a first filter to a first set of neighboring luma samples searched, to determine a first luma sample subsampled from the plurality of luma samples subsampled, when no luma sample that is more than the limit number of samples to the left of the block of luma and below the upper left luma sample is required for subsampling according to the first filter, and in which the first subsampled luma sample is in a column which is a first number of columns to the left of the luma block; and applying a second, different filter to a second set of neighboring luma samples searched, to determine a second luma sample subsampled from the plurality of luma samples subsampled, when at least one luma sample which is more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block is required for subsampling according to the first filter, where the second subsampled luma sample is in a column which is a second number of columns to the left of the luma block, and where the second number is greater than the first
Petition 870190053680, of 12/06/2019, p. 145/171
22/26 number.
[54]
58. The method of claim 57, wherein the first filter uses three luma samples from a first line which is to the left of the luma block and three luma samples from a second line which is below the first line.
[55]
59. The method of claim 57, wherein the second filter uses less than three luma samples from a first line that is to the left of the luma block and less than three luma samples from a second line that is below the first line.
[56]
60. The method of claim 53, wherein determining the plurality of subsampled luma samples comprises:
generate, without searching, luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[57]
61. The method of claim 53, wherein the LM prediction that encodes the chroma block based on the predictive block comprises subtracting the predictive block from the chroma block to generate a residual block that will be used by a decoder of video, to reconstruct the chroma block.
[58]
62. The method of claim 53, which
Petition 870190053680, of 12/06/2019, p. 146/171
23/26 additionally comprises:
encode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag encoding comprises encoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.
[59]
63. Device for encoding video data, the device comprising:
a video data memory; and a video encoder comprising at least one fixed function circuitry or programmable circuitry, wherein the video encoder is configured to:
determining a luma block that corresponds to a chroma block; search for neighboring luma samples to carry out the downward sampling of neighboring luma samples, in which the neighboring luma samples sought comprise a plurality of luma samples that are to the left of the luma block and exclude luma samples that are on the left and that they are more than a limit number of samples to the left of the luma block and below an upper left sample of the luma block;
determine a plurality of subsampled luma samples based on the neighboring luma samples sought, in which one of the subsampled luma samples corresponds to a subsampled luma sample which is the
Petition 870190053680, of 12/06/2019, p. 147/171
24/26 limit number of samples to the left of the luma block;
determine one or more scheduling parameters based on luma samples submitted to descending sampling;
determine a predictive block based on one or more scheduling parameters; and code by prediction by linear model (LM) the chroma block based on the predictive block.
64 Device, in a deal with the claim 63, in what a coordinate gives sample of upper luma left of luma block is (xO , yO), and in that samples
of neighboring luma searched exclude luma samples that have an x coordinate less than (χθ-k) and a y coordinate greater than yO, where k is the limit number and k is an integer greater than 0.
[60]
65. The device of claim 64, wherein k is equal to 4.
[61]
66. Device according to claim
63, in which one of the subsampled luma samples corresponds to a subsampled luma sample more than two columns to the left of the upper left luma sample.
[62]
67. Device according to claim
63, in which to determine the plurality of subsampled luma samples, the video encoder is configured to:
apply a first filter to a first set of neighboring luma samples, to determine a first subsampled luma sample from the plurality of subsampled luma samples, when no luma sample being
Petition 870190053680, of 12/06/2019, p. 148/171
25/26 more than the limit number of samples to the left of the luma block and below the upper left luma sample is required for subsampling according to the first filter, and where the first subsampled luma sample is in a column that it is a first number of columns to the left of the luma block; and applying a different second filter to a second set of neighboring luma samples, to determine a second luma sample subsampled from the plurality of subsampled luma samples, when at least one luma sample which is more than the limit number of samples to left of the luma block and below the upper left luma sample of the luma block is required for subsampling according to the first filter, where the second subsampled luma sample is in a column which is a second number of columns to the left of the luma block, and where the second number is greater than the first number.
[63]
68. The device of claim 67, wherein the first filter uses three luma samples from a first line which is to the left of the luma block and three luma samples from a second line which is below the first line.
[64]
69. The device of claim 67, wherein the second filter uses less than three luma samples from a first line that is to the left of the luma block and less than three luma samples from a second line that is below the first line.
[65]
70. The device of claim 63, wherein for determining the plurality of samples of
Petition 870190053680, of 12/06/2019, p. 149/171
26/26 luma subsamples, the video encoder is configured to:
generate, without searching, luma values that correspond to luma samples located more than the limit number of samples to the left of the luma block and below the upper left luma sample of the luma block; and applying a filter to the generated luma values to determine at least one subsampled luma sample from the plurality of subsampled luma samples.
[66]
71. The device according to claim 63, in which it codes the chroma block by LM prediction based on the predictive block, the video encoder is configured to subtract the predictive block from the chroma block to generate a residual block that will be used by a video decoder to reconstruct the chroma block.
[67]
72. Device according to claim
63, where the video encoder is configured to:
encode a flag for a current block that includes the luma block and the chroma block, where the flag indicates that LM predictive encoding is enabled for the chroma block, and where the flag encoding comprises encoding the flag with based on a context comprising one or more flags that indicate whether predictive encoding by LM is enabled for neighboring blocks.

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR112019011883A2|2019-10-22|sample model linear access prediction mode for video coding

---

JP2020150561A|2020-09-17|Overlapped motion compensation for video coding

---

JP6827944B2|2021-02-10|Downsampling process for linear model prediction mode

---

TWI533679B|2016-05-11|Parameter sets in video coding

---

KR20180102569A|2018-09-17|Multi-type tree framework for video coding

---

ES2874552T3|2021-11-05|Palette mode encoding and decoding layout

---

ES2727814T3|2019-10-18|Syntactic structure of hypothetical reference decoder parameters

---

US9521393B2|2016-12-13|Non-nested SEI messages in video coding

---

KR101825262B1|2018-02-02|Restriction of prediction units in b slices to uni-directional inter prediction

---

TWI558178B|2016-11-11|Method and device of coding video data, and computer-readable storage medium

---

US20150016503A1|2015-01-15|Tiles and wavefront processing in multi-layer context

---

BR112019013705A2|2020-04-28|temporal prediction of modified adaptive loop filter to support time scalability

---

KR20180113515A|2018-10-16|Efficient conversion coding with optimized compact multipath transforms

---

ES2780686T3|2020-08-26|Type of dependency between views in MV-HEVC

---

JP2016526334A|2016-09-01|Video coding using sample prediction between color components

---

KR20200139163A|2020-12-11|Combined position-dependent intra prediction extended with angular modes

---

EP3198873A1|2017-08-02|Parsing dependency reduction for palette index coding

---

US11172229B2|2021-11-09|Affine motion compensation with low bandwidth

---

JP2018514985A|2018-06-07|Device and method for processing video data

---

BR112019014090A2|2020-02-04|intraprevision techniques for video encoding

---

JP2018524906A|2018-08-30|Reference picture list configuration in intra block copy mode

---

JP2017507539A|2017-03-16|Method for coding recovery point supplemental enhancement information | messages and region refresh information SEI messages in multi-layer coding

---

BR112019027071A2|2020-07-07|Improved intra-prediction in video coding

---

BR112021004492A2|2021-05-25|adaptive multiple transform coding

---

BR122021010905A2|2021-07-20|IMAGE DECODING METHOD PERFORMED BY A DECODING APPARATUS, IMAGE ENCODING METHOD PERFORMED BY AN ENCODING APPARATUS, COMPUTER-READABLE NON TRANSIENT STORAGE MEDIA, DECODING APPARATUS FOR IMAGE DECODING AND IMAGE ENCODING APPLIANCE FOR ENCODING

同族专利:

---

公开号 | 公开日

---

US20180176594A1|2018-06-21|

---

KR20190097036A|2019-08-20|

---

JP2020502925A|2020-01-23|

---

CN110024406A|2019-07-16|

---

EP3556099A1|2019-10-23|

---

WO2018118940A1|2018-06-28|

---

US10477240B2|2019-11-12|

---

TW201830963A|2018-08-16|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

US9693070B2|2011-06-24|2017-06-27|Texas Instruments Incorporated|Luma-based chroma intra-prediction for video coding|

---

CN103096055B|2011-11-04|2016-03-30|华为技术有限公司|The method and apparatus of a kind of image signal intra-frame prediction and decoding|

---

US9438904B2|2012-01-19|2016-09-06|Futurewei Technologies, Inc.|Reduced look-up table for LM mode calculation|

---

CN109218730A|2012-01-19|2019-01-15|华为技术有限公司|Reference pixel reduction for LM intra prediction|

---

EP2805501A1|2012-01-24|2014-11-26|Futurewei Technologies, Inc.|Simplification of lm mode|

---

US9438905B2|2012-04-12|2016-09-06|Futurewei Technologies, Inc.|LM mode with uniform bit-width multipliers|

---

CN105474639B|2013-07-10|2018-08-07|凯迪迪爱通信技术有限公司|Video coding apparatus, video decoder, video system, method for video coding, video encoding/decoding method and program|

---

US10455249B2|2015-03-20|2019-10-22|Qualcomm Incorporated|Downsampling process for linear model prediction mode|

---

WO2016154963A1|2015-04-01|2016-10-06|Mediatek Inc.|Methods for chroma coding in video codec|

---

US10484712B2|2016-06-08|2019-11-19|Qualcomm Incorporated|Implicit coding of reference line index used in intra prediction|US10873746B2|2016-12-21|2020-12-22|Sharp Kabushiki Kaisha|Intra prediction image generation device using cross-component liner model, image decoding apparatus, and image coding apparatus using same|

---

GB2567249A|2017-10-09|2019-04-10|Canon Kk|New sample sets and new down-sampling schemes for linear component sample prediction|

---

WO2020015433A1|2018-07-15|2020-01-23|Huawei Technologies Co., Ltd.|Method and apparatus for intra prediction using cross-component linear model|

---

KR20210030444A|2018-07-16|2021-03-17|후아웨이 테크놀러지 컴퍼니 리미티드|Video encoder, video decoder and corresponding encoding and decoding method|

---

TW202017376A|2018-08-17|2020-05-01|大陸商北京字節跳動網絡技術有限公司|Simplified cross component prediction|

---

GB2591379A|2018-09-12|2021-07-28|Beijing Bytedance Network Tech Co Ltd|Single-line cross component linear model prediction mode|

---

JP2022500967A|2018-10-08|2022-01-04|ベイジン ダージャー インターネット インフォメーション テクノロジー カンパニー リミテッドBeijing Dajia Internet Information Technology Co., Ltd.|Simplification of cross-component linear model|

---

US10939118B2|2018-10-26|2021-03-02|Mediatek Inc.|Luma-based chroma intra-prediction method that utilizes down-sampled luma samples derived from weighting and associated luma-based chroma intra-prediction apparatus|

---

KR20210089132A|2018-11-06|2021-07-15|베이징 바이트댄스 네트워크 테크놀로지 컴퍼니, 리미티드|Intra prediction based on location|

---

AU2019391197A1|2018-12-07|2021-06-17|Beijing Bytedance Network Technology Co., Ltd.|Context-based intra prediction|

---

WO2020130745A1|2018-12-21|2020-06-25|삼성전자 주식회사|Encoding method and device thereof, and decoding method and device thereof|

---

CN109672895A|2018-12-27|2019-04-23|北京佳讯飞鸿电气股份有限公司|A kind of HEVC intra-frame prediction method and system|

---

EP3903482A1|2019-02-22|2021-11-03|Beijing Bytedance Network Technology Co. Ltd.|Neighbouring sample selection for intra prediction|

---

EP3903493A1|2019-02-24|2021-11-03|Beijing Bytedance Network Technology Co. Ltd.|Parameter derivation for intra prediction|

---

KR20210130153A|2019-02-28|2021-10-29|인터디지털 브이씨 홀딩스 인코포레이티드|Method and device for picture encoding and decoding|

---

EP3703366A1|2019-02-28|2020-09-02|InterDigital VC Holdings, Inc.|Method and device for picture encoding and decoding|

---

US20200288159A1|2019-03-08|2020-09-10|Qualcomm Incorporated|Combined residual coding in video coding|

---

EP3939259A1|2019-03-12|2022-01-19|Beijing Dajia Internet Information Technology Co., Ltd.|Video coding using multi-model linear model|

---

US10742992B1|2019-03-26|2020-08-11|Electronic Arts Inc.|Video compression for video games|

---

CN113711590A|2019-04-18|2021-11-26|北京字节跳动网络技术有限公司|Reducing delay in cross component mode|

---

WO2021040479A1|2019-08-29|2021-03-04|엘지전자 주식회사|Device and method for coding video on basis of filtering|

法律状态:
2021-10-13| B350| Update of information on the portal [chapter 15.35 patent gazette]|

优先权:

---

申请号 | 申请日 | 专利标题

---

US201662436319P| true| 2016-12-19|2016-12-19|

---

US15/845,484|US10477240B2|2016-12-19|2017-12-18|Linear model prediction mode with sample accessing for video coding|

---

PCT/US2017/067344|WO2018118940A1|2016-12-19|2017-12-19|Linear model prediction mode with sample accessing for video coding|

---