GB2312348A - Chromakeying with transition colour volumes - Google Patents

Abstract

Pixel values are specified by three color components (RGB) of colour space. A set of pixels may be identified as belonging to a defined colour volume, ie a sub-set of the world volume. This colour volume is defined by planes of constant red, vector 503, constant green, vector 505 and constant blue, vector 506. The colour volume may also be modified with respect to an additional plane defined by a plurality of these axis. Thus, the additional plane may represent values of constant luminance, vector 502, 501, constant yellow, vector 511, constant magenta, vector 509, constant cyan, vector 507 etc. When used in chromakeying, a first defined volume is enclosed within a second defined volume and space between these volumes represents the grey-scale of blending edge. By including additional planes for modifying the volumes, it is possible to optimise the keying volumes with reference to intuitive colour space parameters (figure 8 not shown).

Description

Title: PROCESSING IMAGE DATA The present invention relates to processing image data, in which pixel values are specified by three color components of color space and a set of pixels are identified as belonging to a defined color volume.

INTRODUCTION Systems for processing image data in which pixel values are specified in three color components of color space are known. A set of pixels may be identified as belonging to a defined color volume when measures are being undertaken to separate constituent parts of an image, usually involving the removal of a foreground object from a background object within a pixel plane. Thus, Chroma Keying procedures have become well developed in digital environments where, usually, a foreground image will be recorded against a highly saturated blue background whereafter the foreground object can be removed from this background by a process of chroma keying. A key signal is generated at regions identified as belonging to the background color which is then used to remove the foreground object from its background while, similarly, a corresponding hole can be cut in a new background image. Thus, video material may be recorded under studio conditions, whereafter specific objects within the recording may be composited with a new background, thereby creating special effects and reducing production costs.

Often, video or film material will have been recorded, for keying purposes, under less than favourable conditions. Under these circumstances, distinguishing a first set of colors from a second set of colors can be particularly difficult. Furthermore, blending edges are required which represent the interface between the foreground object and the new background, where a degree of blending must occur so as to enhance the realism of the effect. If blending of this type does not occur and hard transitions occur on pixel boundaries, visible artifacts will be present within the image and it will be clear to anyone viewing the resulting clip that the two image parts originated from separate sources.

A problem with known systems is that it may be difficult to adjust color volumes so as to ensure that all key colors are within an internal volume, all non-key colors are outside an external volume, with the required blending regions being outside the internal volume but inside the external volume.

In a preferred embodiment, the additional plane represents values of constant luminance. Alternatively, or in addition thereto, the additional plane may represent constant values of a secondary color, such as yellow, cyam or magenta.

According to a second aspect of the present invention, apparatus for processing image data are provided, including means for specifying pixel values representing three color components of color space such that a set of pixels may be identified as belonging to a defined color volume, comprising means for defining planes for bounding color volumes7 wherein a color volume is bound by planes defined by a single color axis; and said defined color volume is modified with respect to an additional plane defined by a plurality of said axes.

SUMMARY OF THE INVENTION According to a first aspect of the present invention, there is provided a method of processing image data, wherein pixel values are specified by three color components of color space and a set of pixels are identified as belonging to a defined color volume, comprising steps of bounding a color volume by defining planes offset from respective axes of said color volumes; and modifying said defined color volume with respect to an additional plane defined by a plurality of said axes.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows a compositing station arranged to key foreground images against new background images; Figure 2 shows an example of a foreground image to be composited against a background, using the equipment identified in Figure 1, in which colors are identified within a color volume; Figure 3 illustrates an identification of color volumes within a red green blue color space, in which an internal volume represents a key color and an external volume represents blending colors; Figure 4 illustrates a modified color volume of the type shown in Figure 3, in which an additional plane has been introduced in accordance with the present invention; Figure 5 details an image displayed on the monitor identified in Figure 1, including a menu; Figure 6 details the menu identified in Figure 5, including a control area for adjusting key color space; and Figure 7 details the control area identified in Figure 6, for making modifications to key color space.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The invention will now be described by way of example only with reference to the previously identified drawings.

A compositing station is shown in Figure 1, in which digital representations of image frames are processed within a processing device 101. The processing device 101 includes permanent storage devices and a processing system controlled in accordance with the UNIX operating system.

A processing device 101 responds to manual operation of a stylus 102 against a touch table 103. Image data is supplied to a monitor 104, in the form of control menus and image clips. An operator may also supply data to the processing system 101 via a manually operable keyboard 105.

An example of an image frame displayed on monitor 104 is shown in Figure 2. The scene has been recorded onto video tape in a television studio, shot against a blue screen background. The blue saturated background is identifiable by the processing device 101, which is arranged to produce a key or matte signal in response to detecting this particular color of blue. The procedure works extremely well if all background images within the shot maintain the background blue color. In addition, this background blue color, or any color close to it, should not be present in the foreground images. In this way, it is possible to cleanly cut-out the required foreground image and thereby allow this foreground material to be composited with a different background image. This procedure is very effective in creative video environments, such as promotional productions created for association with popular music recordings.

Within the processing device 101, each pixel within each image frame defines a particular color for that pixel position. In broadcast television systems, colors are often specified in terms of a luminance plus two color different signals, usually referred to YUV. Alternatively, in many computer graphics based systems, images will be recorded as three primary color components red green blue. In a television monitor, arranged to receive YUV signals, possibly in accordance with the popular D1 standard, conversion is ultimately required in order to convert the YUV signals into RGB for the three color guns. Furthermore, it is highly undesirable to effect many conversions between different types of color standards. It is therefore preferable to maintain color components within a particular type and perform all processing function upon the one type of color components.

In the present example, color components are recorded as three components representing the primary colours red green and blue (RGB) with eight or ten bits being assigned to each of these components.

When performing a chroma keying process (chroma matting or chroma video suppress) each RGB color may be identified as belonging to one of three sets. The first set represents the full key color of the background which will often be a relatively small volume of colors having a relatively high blue component with relatively low red and green components. Surrounding this volume of colors will be the set of pixels where a transition is to occur. This set will have blue values that are relatively lower than the hard key set, with red and green values that are relatively higher. This set of color values represent pixels where a key will have an intermediate value, usually between 0 and 225 in an eight bit system, so as to provide a degree of blending between the foreground object and the new background. Thirdly, a set of color values will exist, usually the largest set, representing allowed foreground values where the key will have a hard value of 0. Thus, in this region, all of the foreground image will be composited with no blending through to the background.

This arrangement of color volumes within an RGB color space is illustrated in Figure 3. Three dimensional space is illustrated in which any point within this space is defined by orthogonal coordinates representing red, green and blue components. Thus, axis 301 represents the intensity of the red component, axis 302 represents the intensity of the green component and axis 303 represents the intensity of the blue component. Thus, every pixel within the image may be mapped onto a region within the RGB color space.

Keying volumes have been defined by an internal color volume 304, surrounded by an external color volume 305. A key signal will be generated from all pixel values which lie within the internal color volume 304. Each key pixel derived from an image pixel lying within this internal volume 304 will be set to its maximum extent, usually 255. Thus, colors falling within this volume present within the foreground image, will be removed or suppressed completely. Similarly, the corresponding area in the new background will receive no keying or suppressing whatsoever.

The opposite is true of all colors lying outside the external volume 305. These colors represent the values of pixels lying within the required part of the foreground image therefore, for all color values lying within this volume, the key signal will be set to zero such that the foreground image is retained and the corresponding area of the background image will be totally removed or suppressed.

Between the outside of the internal volume 304 and the inside of the external volume 305, an intermediate volume exists which is required to provide blending between the foreground image and the background image. For blending of this type to be available at all portions of the image, where appropriate, it is essential for space to exist between the external volume and the internal volume. Thus, under circumstances where the internal volume abuts or is very close to the external volume, sharp transitions will occur between the key being full on and the key being full off. Thus, if a transition between the foreground object and the background object exists within this area of the color space, a visible artifact will result, thereby giving an unacceptably degraded composite.

Any particular point within the color volume shown in Figure 3 is uniquely definable by specifying its red, green and blue components.

Thus, any particular color may be enclosed within planes similar to the planes shown in Figure 3. It is therefore an established practice to define color volumes for keying purposes using parallel planes defining cuboids, as shown in Figure 3. However, using this approach, it is possible, in some keying situations, for planes of an internal volume to be uncomfortably close to parallel planes of the external volume 305.

Under these conditions, it may not be possible to satisfy the three constraints of ensuring that undesirable colors are removed, while required colors are retained and while a smooth blending edge is established between the two extremes. In particular, it is likely that the blending edge will be too sharp or not present at all. The present invention allows an internal volume and an external volume to be defined more precisely while still retaining an intuitive color volume model. In particular, color volumes of the type identified as 304 and 305 in Figure 3, are bounded by planes offset from respective axis of the color volume. Thus, a volume includes planes such as planes 306 and 307, which are planes of constant red. Similarly, front facing plane 308 and its opposite plane 309 are planes of constant green, with the top plane 310 and bottom plane 311 being planes of constant blue.

However, in addition to these planes, the color volume is modified so as to include additional planes defined by a plurality of the axes.

An example of such a volume is illustrated in Figure 4. Planes are defined by specifying red components, as illustrated by arrow 401.

Similarly, blue components may be specified, as illustrated by arrow 402. Thirdly, green values may be specified by adjustments in the direction perpendicular to arrows 401 and 402 arrows. Thus, in this way, it possible to construct volumes of the type shown in Figure 3. These planes have a specified value for one of the axes, with the plane being defined by all other possible combinations of the other color components. Thus, the planes equate to planes of constant red, planes of constant blue and planes of constant green. This results in color volumes of the type illustrated in Figure 3 being generated.

In addition, it is possible to specify planes with respect to a plurality of the axes, as distinct from planes having a constant value of one of the three color components. Thus, perpendicular planes 403 and 404 are illustrated, which effectively cut through opposing corners of the color space defined by a cuboid. The actual position of these planes may be adjusted by specifying their position along axis 405.

Axis 405 intersects the origin 406, at which position the red green and blue components are all set to zero, a pixel with this color is therefore black. Axis 405 extends from the origin until it reaches the top far right vertex where the red green and blue components are set to their maxim extent, ie white. Control planes such as planes 403 and 404 are perpendicular to axis 405 and represent planes where the sum of the red green and blue components are constant. Consequently, these planes represent pixels of constant luminance which, in turn, may be intuitively injusted by video artists.

A cube representing the total color volume for RGB space is shown in Figure 5. At an origin 501 all of the color values are set to zero and red green and blue component values extend from this origin.

Vertex 502 represents a point at which each of the red green and blue values are set to their maximum extent. As previously stated, points 501 and 502 may be connected by a vector and all planes perpendicular to this vector represent values of constant luminance.

Axis 503 represents increasing values of red and point 504 represents the point at which the red component is set to its maximum value, with the blue and green components remaining at zero. Similarly, vertex 505 represent the point at which the green value has been set to its maximum extent, with the red and blue values remaining at zero.

Point 506 is that at which the blue component has been set to its maximum extent with the green and red values being set to zero. A vector 507 extends from point 504 to its opposing point 508. Planes may be created which are perpendicular to this vector, following a similar geometry to planes 403, 404 illustrated in Figure 4. Similarly, point 505 may be connected by a vector 509 to its opposite point 510.

Again, planes may be constructed which lie perpendicular to this vector, having a similar geometry to planes 403 and 404. Finally, a vector 511 may be extended from point 506 to its opposing point 512 and again planes may be defined which are perpendicular to this vector, again with a similar geometry to planes 403 and 404 shown in Figure 4. Thus, a total of 4 plane pairs may be defined, each of which effectively cutsoff one of the color space corners of the cuboid shown in Figure 4.

Colors generated by combining two primary colors are well known as the secondary colors and are identified as cyan, yellow and magenta (CYM). Yellow is produced by combining red and green, with magenta being created by combining red and blue and cyan being created by combining green and blue.

Point 508 represents a color where the green value is set to its maximum and the blue value is set to its maximum, with the red value being set to its minimum. Thus, point 508 represent bright cyan where the red component is zero and the blue and green components are set to their maximum value. Descending back along vector 507 towards point 404 results in the red component being increased with the cyan component being reduced. Thus, any plane defined which is perpendicular to vector 507 may be considered as defining points of constant cyan.

Point 505 represents a point at which the green component is set to its maximum extent and vector 509 moves to the opposite point 510 at which the green component is set to its minimum extent. However, at point 510 both the blue and red components have been set to their maximum extent therefore point 510 represents the point of maximum magenta. Similarly, planes perpendicular to vector 509 may be considered as being planes of constant magenta.

Finally, point 506 represents the point where the blue component is at its maximum extent, with the red and green components being zero. At the opposite corner 512, the opposite is true in that the blue component is set to zero with the red and green components being set to their maximum extent. Thus, planes perpendicular to vector 511 may be considered as defining the set of points at which the yellow component is constant.

Returning to Figure 4, it can be seen that in addition to planes 403 and 404, representing values of constant luminance, three other sets of planes may be defined, representing levels of constant cyan, levels of constant yellow and levels of constant magenta. A video artist will be familiar with all of these color components and therefore it is possible to adjust the internal and external color volumes by controlling color component parameters representing the position of specific bounding planes, in addition to the basic RGB planes.

Monitor 104 is detailed in Figure 6 and includes manually adjustable control devices 601. An image is displayed on the monitor 104, including a canvas region 602 for displaying video images and clips, in combination with a control region 603, arranged to display a control menu. Regions within the canvas 602 and operational controls within the control menu 603 are selectable in response to manual operation of the stylus 102. The position of the stylus 102 over the touch tablet 103 is illustrated by a cursor 604 and particular selections may be made by placing the cursor over part of the control region 603 and thereafter placing the stylus 102 into pressure.

Control region 603 is detailed in Figure 7 and includes a plurality of selectable controls similarly to those found on conventional video tape recorders. These allow clips of video to be controlled and include a fast-rewind button 701, a reverse button 702, a stop button 703, a play button 704 and a fast-forward button 705.

Modes of operation may be selected by controls 706 and parameters for controlling chromakeys are displayed within region 707.

Chromakey-control region 707 is detailed in Figure 8. Soft control buttons 801, 802, 803, 804, 805, 806 and 807 allow planes for red, green, blue, cyan, yellow1 magenta a luminance to be selected respectively. Each color includes a legend identifying its respective color and, in addition, the buttons may be colored in a color similar to that to which they select.

Alongside each of the selection buttons 801 to 807 is a respective control bar. Control bar 808 is arranged to control the red control planes. Planes having values of constant red are specified by plane indicators 809 and 810. These indicators are moveable in response to manual operation by placing the cursor 604 above the appropriate indicator, forcing the stylus into pressure and moving the stylus to the left or to the right resulting in bars, such as bars 809 or 810 being dragged within region 808. Thus, similar bars within similar regions are provided for the other control planes as shown in Figure 8.

Thus, by adjusting the intuitive control panel shown in Figure 8, it is possible for a video artist to accurately position bounding planes for an inner volume and an outer volume, thereby ensuring that an intermediate volume is present to provide blending between the two key extremes. The control mechanism illustrated in Figure 8 allows planes to be defined which are offset from respective axes of the color volume which, in this example, represents planes of constant red, constant green or constant blue. In addition, the control mechanism shown in Figure 8 allows intuitive adjustments to be made for planes defined by a plurality of the red green and blue axes. In particular, adjustment may be made to planes defining constant luminance, constant cyan, constant yellow and constant magenta. This approach could also be extended to environments which color components are defined by other sets of color axes. Thus, for example, color components could be stored within CYM color space with additional planes being defined for density, red, green and blue. Similarly, the axes need not be orthogonal axes and polar coordinates or combinations may be used, as required, for example, in YUV color space. Thus, primary axes may represent contributions for luminance plus color difference, with additional axes being provided for specifying related characteristics like hugh and saturation.

Claims (22)

1. A method of processing image data, wherein pixel values are specified by three color components of color space and a set of pixels are identified as belonging to a defined color volume, comprising steps of bounding a color volume by defining planes off-set from respective axes of said color volume; and modifying said defined color volume with respect to an additional plane defined by a plurality of said axes.

2. A method according to Claim 1, wherein said three color components represent components for red, green and blue.

3. A method according to Claim 1, wherein bounding planes are defined as planes having constant values of red, constant values of blue or constant values of green.

4. A method according to Claim 1, wherein said additional plane represents constant luminance values.

5. A method according to Claim 1, wherein said additional plane represents constant values of a secondary color, (yellow, cyan or magenta).

6. A method according to Claim 1, wherein the defined color volume is modified with respect to a plurality of additional planes, each defined by a plurality of the primary axes.

7. A method according to Claim 6, wherein said additional planes are defined as values having constant luminance, constant cyan, constant magenta and constant yellow.

8. A method according to Claim 1, wherein a first bounding volume is specified within a second bounding volume.

9. A method according to Claim 8, wherein said first bounding volume represents colors from which a key will be generated.

10. A method according to Claim 8, wherein the volume between said first bounding volume and said second bounding volume identifies colors for which varying edge intensities will be generated for said key.

11. Apparatus for processing image data, including means for specifying pixel values representing three color components of color space such that a set of pixels may be identified as belonging to a defined color volume, comprising means for defining planes for bounding color volumes, wherein a color volume is bound by planes defined by a single color axis; and said defined color volume is modified with respect to an additional plane defined by a plurality of said axes.

12. Apparatus according to Claim 1, wherein storage means are provided for storing color components representing red, green and blue.

13. Apparatus according to Claim 11, wherein processing means are provided for defining bounding planes having values of constant red, constant blue or constant green.

14. Apparatus according Claim 11, wherein processing means are provided for defining an additional plane representing constant luminance values.

15. Apparatus according to Claim 11, including processing means for defining additional planes of a constant secondary color, such as yellow, cyam or magenta.

16. Apparatus according to Claim 11, wherein processing means are provided for defining a color volume with respect to a plurality of additional planes, each defined by a plurality of the primary axes.

17. Apparatus according to Claim 16, wherein said additional planes are defined as values having constant luminance, constant cyam, constant magenta and constant yellow.

18. Apparatus according to Claim 11, wherein processing means are provided for defining a first bounding volume within a second bounding volume.

19. Apparatus according to Claim 18, including key generating means arranged to generate a key signal on colors defined within said first bounding volume.

20. Apparatus according to Claim 18, including key generating means for generating a key signal with soft edges, wherein said soft edged data is derived from colors lying between an identified first volume and an identified second volume.

21. A method of processing image data substantially as herein described with reference to the accompanying drawings.

22. Apparatus for processing image data substantially as herein described with reference to the accompanying drawings.