CN1682217B - Media article composition - Google Patents

Abstract

A method and apparatus for the automatic composition of media articles is disclosed. Previous attempts to do this have involved the manual production of a detailed framework listing the types of media elements that might be inserted into the slots specified in that framework and the subsequent selection of media elements for those slots. The present invention improves upon this by associating metadata ( 51 ) with the media elements in a content store, which metadata includes relationship data indicating how what is portrayed by that media element relates to what is portrayed by one or more other media elements. Because relationship metadata is included media articles can be generated automatically with the need for the production of a detailed framework describing that media article.

Description

Synthesis of media materials

technical field

The present invention relates to methods and apparatus for synthesizing media articles.

Background technique

Media materials portray content (real, fake or computer-generated) for people's senses. The media material can be presented to people through various media (including text, voice, sound, picture or animation).

As recording technology advances, the amount of recorded media material available to consumers is growing rapidly. Media material is typically recorded in multiple media files (note that although the plural "media" is used here, "media files" should be understood to include files intended to be delivered to users via only one medium, such as text or voice , and "multimedia" files whose content is delivered by multiple media). The Internet is the latest communication network to emerge and provides worldwide transmission of recorded digital media files representing text, sound, pictures, animation or combinations thereof. Due to the considerable number of media files that can be accessed over the Internet, there is a need to tag these media files with some description of what the media files contain. Thus, for example, HTML (Hypertext Markup Language) files contain "meta" tags that include keywords that indicate what topics are included in the web page presented to the user.

Marking media files with metadata is more beneficial when a group of users agree on how the metadata should be structured and what elements that metadata should contain. Typically, XML (Extensible Markup Language) is used to define such a structure and the elements contained in the structure. In fact, XML can be used to define a metadata "language". An example of such a metadata "language" is StoryML, as described in "StoryML: An XMLExtension for Woven Stories" by P. Gerdt et al., pages 893-902 of "proceeding of the IntelligentTutoring Systems conference 2002". StoryML is metadata designed to describe the contribution of a co-authored novel. Likewise, it includes elements giving the author of the assignment and the assignment relationship to other assignments.

The Moving Picture Experts Group is discussing a proposal to add metadata to video files ("Multimedia Content Description Interface", commonly known as MPEG-7).

International patent application WO 02/057959 discloses computer software that provides users with tools for organizing media files. Various metadata can be associated with these files. These files and metadata are stored in "relational" databases, note that "relational" as used in the expression relational database has little to do with database entries or what is expressed in those entries, instead it refers to the mathematical set theory used in meaning of the word "relevant".

One method of compositing media material involves bringing together multiple components. For example, a movie consists of multiple scenes described in a script. There have been some attempts to automatically generate video material in this way. For example, a Waltz composition based on the musical dice game can be found at http://sunsite.univie.ac.at/Mozart/dice/. Similarly, an automatic story generator (Romance Writer) is available from http://familygames.com/features/humor/romance.html.

Contents of the invention

According to a first aspect of the present invention, a method for automatically synthesizing media materials is provided, including:

Analyzing digital metadata associated with the first set of stored media data, the digital metadata includes:

related set identification data for identifying the second set of storage media data; and

relationship data for representing the relationship between the content represented by the first set of stored media data and the content represented by the second set of stored media data; and

The first and second sets of stored media data are disposed in a media profile based on the analysis.

By analyzing the digital metadata associated with the first storage media data set (the digital metadata includes the identifier of the second storage media data set, and the content represented by the first storage media data set is related to the content represented by the first storage media data set) relationship indication between the contents represented by the second set of stored media data), and setting the first and second set of stored media data in the media material according to the analysis, to provide a method for synthesizing the media material , which eliminates the need for another source of sequencing information when synthesizing media material.

The expression "stored collection of media data" includes collections of media data files, streams, or pointers to files or databases.

Digital collections of stored media data are stored in a variety of formats. The representation of a collection of storage media data is not intended to be limited to any particular format, but may include, for example, a file containing only data relating to the location of components seen by the user while playing a computer game—then, The data in this file is processed by rendering software to generate an image for display to the user.

In some embodiments, the method further includes generating the set of identification data and the relationship data.

Preferably, the metadata further includes content data, the content data represents the content represented by the set of stored media data, and the method further includes the following steps: according to the content data, selecting one from a plurality of sets of stored media data or more sets of stored media data, said one or more sets comprising said first and second sets of stored media data.

The combination of the following processes makes automatic media composition for a particular subject easier than existing compositions: a) conduct a search to provide multiple possible components for the media, or provide a portion of the media; and b) then A plurality of selected components are set according to metadata associated with the components.

Preferably, the method further comprises the steps of: making a plurality of such selections; and concatenating the results of said selections.

This enables a method of media material composition that fully follows established patterns, but which enables a degree of flexibility within patterns not seen in traditional systems. Thus, the type of generalized schema known for narrative and film can still be used (e.g., the StoryCraft program from StoryCraft Inc., 560 Roland Drive, Norfolk, VA 23509, USA by requiring authors to introduce heroes and antagonists before conflict occurs, and then is the hero's triumph, to help the author write the novel), but variation can easily be introduced within the mod by changing the nature of the choices.

This would eg enable the low-cost creation of different versions of a movie or computer game, while meeting different artistic or quality requirements, while still retaining subjectively high standards for narrative, movement and audio continuity.

According to a second aspect of the present invention, a media data synthesis device is provided, which includes:

one or more memory devices for storing, for each of the plurality of sets of stored media data, metadata including relational data representing the relationship between the contents of said set of stored media data and one or more Relationships among contents in other storage media data sets; and

one or more processors in communication with said one or more memory devices and operable to set The stored media data sets or their identifiers are used to synthesize media materials.

In a preferred embodiment, an object-oriented database is used to store objects containing metadata associated with sets of stored media data identified by said metadata. Relational metadata can then be represented by relationships between multiple objects in the object-oriented database.

Alternatively, the set of media data may be stored in a file system.

Description of drawings

Specific embodiments of the present invention will now be described by way of example only with reference to the accompanying drawings, in which:

1 is a schematic diagram of components of a personal computer operable to provide a media assembly system according to a first embodiment of the present invention;

FIG. 2 is an architectural diagram of a computer program used when controlling the personal computer of FIG. 1 according to the first embodiment of the present invention;

FIG. 3 represents metadata associated with media elements following use of the media tagging tool of FIG. 2;

Figure 3B represents a cause object stored in the object-oriented database of Figure 2;

Figures 4A to 4G illustrate how editors create relational data contained in the metadata of Figure 3;

Figure 5 is a hierarchical representation of relational data entered by an editor as shown in Figures 4A to 4G;

Fig. 6 represents the template data generated using the template generating tool of Fig. 2;

Figure 7 shows user profile data generated using the user profile generation tool of Figure 2;

FIG. 8 is a flowchart representing the operation of the template populator component of the first embodiment of the present invention;

Figure 9 shows the operation of the template combiner in relation to the first part of the template shown in Figure 7;

Figures 10A to 10D represent the operation of the template combiner associated with the second part of the template shown in Figure 7;

Fig. 11 shows the edit determination list that may be generated by the template combiner module;

Figure 12 represents a display presented to a user upon execution of a computer program providing a preferred embodiment of the present invention;

Figure 13 represents the same display at a later stage of the program run;

Figure 14 shows additional sub-windows that open when the user selects one of the media cabinets;

FIG. 15 shows an additional sub-window presented to the user when the user selects one of the media elements contained in the selected media cabinet;

Figure 16 shows the application window after the user has indicated that he wishes to see the relationship data associated with selected (or all) media elements contained in the media cabinet;

Figure 17 shows a further sub-window enabling the user to define vocabulary;

Figure 18 represents the display when the user selects the template used when generating the media material and edits the template;

Figure 19 represents the display when the template is expanded after the user selects the template used when generating the media material;

Figure 20 shows a subwindow used to display the results of the user running the Template Composer component of the program;

Figure 21 shows the additional functionality provided to the user through the Query Builder window (as shown in Figures 18 and 19) in a modification of the preferred embodiment of the present invention; and

Figure 22 illustrates additional controls provided to the user when previewing media material using the improvements of the preferred embodiment of the present invention.

Detailed ways

Figure 1 shows a personal computer comprising known hardware components connected together in a conventional manner. These well-known hardware components include central processing unit 10 , random access memory 12 , read-only memory 14 , hard disk 16 , and input/output devices 18 , 20 , 22 , 24 and 26 . These hardware components are interconnected by one or more data and address buses 28 . Input/output devices include a monitor 18, a keyboard 20, a mouse 22, a CD ROM drive 24, and a network card 26. The network card is connected to the server computer 30 through the public Internet 32 .

Figure 2 shows the software and data components of a system according to a first embodiment of the invention. The software includes: three input program modules, namely, media tagging tool program 40, template generation tool program 42, user profile generation tool program 44; object-oriented database management software 45; and two output program modules, namely, template A combiner program module 46 and a content composition program module 48 .

。该数据库存储三种不同分类的对象，即，媒体对象51(每一个媒体对象都包括对存储在内容存储器50中的多个媒体文件之一进行描述的元数据)、模板对象52和用户简档对象53。该数据库中的这些对象还存储在计算机的硬盘16中。可以通过诸如DVD、CD、CD-ROM的可移动数据载体而不是硬盘，来提供用于面向对象的数据库和/或内容存储的永久存储，或者在可以通过例如由网卡26提供的网络连接访问的不同计算机(例如通过URI进行访问)上提供用于面向对象的数据库和/或内容存储的永久存储。The system also includes two memories for permanent data storage. The first of these two memories is the content memory 50, which includes a plurality of media files stored in the hard disk 16 using the file system provided by the computer's operating system program. A second memory for permanent storage of data includes an object-oriented database 54 known as the Excelon Corporation, Burlington, Massachusetts.

. The database stores three different categories of objects, namely, media objects 51 (each media object includes metadata describing one of a plurality of media files stored in the content store 50), template objects 52, and user profiles Object 53. The objects in the database are also stored on the hard disk 16 of the computer. Permanent storage for object-oriented databases and/or content storage may be provided by removable data carriers such as DVD, CD, CD-ROM instead of a hard disk, or on a network accessible via a network connection such as provided by network card 26. Persistent storage for object-oriented databases and/or content storage is provided on different computers (eg accessed via URI).

The three input program modules control the computer (FIG. 1) to provide a user interface for generating objects comprising data structures structured according to a predetermined structured data model. This allows different program modules to exchange data more easily. In cases where the constructed data modules are common, different parties may write different components of the program, which nevertheless are able to exchange data with each other. Additionally, objects created by one editor can be used by another editor working at a different location or later.

In this embodiment, these three input program modules provide the following functions:

The media tagging tool provides an interface for editors to update the content store 50 and object-oriented database 54 . In fact, it is contemplated that an editor using this embodiment may have access to media elements generated by other editors, rushes from various sources, parts of programmed programs, still photos, and various other media pieces of his own settings (both in electronic form). These media elements are all stored in the appropriate directory structure in the content store. Each catalog is referred to in the art as a "cabinet"—a reference to a labeled cabinet in which multiple rolls of film associated with a particular project are stored.

Media elements are typically recorded digitally in a file, which can contain multiple elements in a linear stream. Accordingly, such a file may contain one or more media elements associated therewith. The media marking tool enables an editor to preview a file and, in the case of multiple media elements in the file, to set a start point and an end point for defining the range of each media element within the file. If there is only one media element associated with the file, the start point is simply set to zero and the end point is set to the length (duration) of the media. In this way, editors can generate multiple files, each of which constitutes a media element. The editor provides a name for each file while defining the start and end points of the media element.

However, for the purposes of this description, assume that the editor starts with just one file that includes an electronic representation of an unedited film recorded at a football game, introduction sequences for a football program, etc. Unedited movie segments are known in the art as "dailies." Using the media markup tool 40, the editor can select various parts of the dailies and store each part as a media element in a shorter file in a directory in the content store 50.

The media tagging tool also provides tools that enable editors to generate or edit metadata for media elements stored in the content store 50 . The tool stores this metadata as objects in an object-oriented database 54 .

Upon selection of a directory in the content store, the editor is presented with a graphical user interface that displays a set of icons (FIG. 4A), each icon representing one of the plurality of media elements stored in the directory. For example, the icons may be still pictures from the video sequence contained in the file. In the examples below, they are represented as boxes containing the editor-entered media element identifier in the upper right corner and a description of what happened in the media element in the center of the box. The user interface enables the editor to select media elements using the mouse 22 and to move them around the screen using the mouse.

After selecting one of these media elements, the editor enters the metadata to be associated with that media element in two stages. In a first stage, the editor can double-click on one of these pictures to bring up a form on which the values of the parameters contained in the schema can be entered.

The second to twelfth lines of Fig. 3 show examples of metadata generated in the first stage (the information in the first line has been generated when the editor provided the file with the media element identifier).

It is ideal to set metadata according to a structured data model. In each row, the entries in the rightmost column represent attribute values entered by the user. A structured data model may specify that a number of attributes should be marked as members of a first-level aggregation unit, referred to herein as an attribute set (second column from the left in a row with four columns). The structured data model may also specify that multiple sets should be marked as members of the second level of clustering units, referred to herein as attribute supersets (the leftmost column in a row with three or four columns). Those skilled in the art will appreciate that further levels of clustering may be provided.

This hierarchical structure is influenced by the multimedia content description interface described above. The intention is not to enforce the use of a complete data model in all possible applications, but to enable re-use of content within the themes of a studio or specific project group (eg wildlife documentaries). The provided data model is intended to provide the largest set of elements and an interface to facilitate the use of these elements, as well as a vocabulary that can be applied to them.

This metadata contains a variable number of parameters (but must still conform to a predetermined structured data model). In the example shown in Figure 3, the editor entered values for 18 attributes. They include:

i) Media element ID - used to identify the media element, in this example, the editor provides it with a value of 0.xx, where xx indicates the position of the media element in the original sample;

Immediately following the media element ID is the "media" superset, which includes the two attributes and the "position" set of attributes. These two properties are:

ii) URI - the Universal Resource Identifier of the file containing the media element;

iii) format - a representation of the format used to provide the data making up the file;

The "Locations" collection contains the following two properties:

iv) In - represents the time elapsed from the start of the dailies at the beginning of the media element;

v) Out - represents the time elapsed from the start of the dailies at the beginning of the media element;

Immediately following the "media" superset is a superset of the four "structural" attributes. This superset starts with:

vi) Description - a description of the content of the file;

Immediately following this description is another collection (called "events") that contains the following three properties:

vii) Nature - the type of event seen in the video sequence recorded in this file;

viii) performer - the person who performs the main act of the event;

ix) Recipient - the person who bears the main action of the event;

These attributes are followed by a superset of scoped attributes that apply only to media elements related to material derived from two-team sports competitions;

The first two attributes belong to a collection (called "team") of the following two attributes:

x) Home team - the name of the participating team playing at its home field in the football game featured in the original sample;

xi) Away team - the name of the other team participating in the football match shown in this raw sample;

Immediately following this collection are the following two properties:

xii) Enforcer Allegiance - the party supported by the Enforcer (if any);

xiii) Recipient Allegiance - party supported by the recipient (if any);

Immediately following these two properties is a collection (called "concept content") containing the following two properties:

xiv) Interest value - this value is 0 and 1, indicating how important the editor thinks the media element is; and

xv) Rating - This value indicates the appropriateness of the media element for presentation to people according to age criteria - similar to ratings for movies.

Once the editor enters this data, the picture is replaced with the description of the media element given as the value of the "description" attribute above. The form is then removed from the display to return a display of pictures representing the media elements selected by the editor from the original dailies (FIG. 4A).

The media element metadata is then stored in the object-oriented database 54 as media objects.

The second phase of metadata generation, in which one or more "relationship" attributes are generated, will now be described with reference to Figures 4A to 4H. A graphical user interface is provided to the user to enable him to indicate the relationship properties between these media elements by moving and clicking the icons representing these media elements on the screen 18 of the PC (FIG. 1).

One type of relationship an editor can indicate is causation. For this, the editor clicks a button element displayed on a screen (not shown), which changes the shape of the cursor. The user thereafter moves the cursor over the media element which he determines is the cause of another media element. He then clicks on the media element to identify it as the cause media element, then moves the cursor over the media element representing the effect of the cause, and clicks again. After this process is complete, an arrow is drawn from the first media element to the diamond representing the cause object, and then a second arrow is drawn from the diamond to the second media element. An editor can make this type of causal association when he believes that viewers viewing the media element representing the effect also expect to see the cause. In the example shown in Figure 4B, the editor indicates:

i) media element 0.13 is caused by media element 0.12, and media elements 0.14 and 0.15 are both caused by media element 0.13; and

ii) Media element 0.53 is caused by media element 0.52 which in turn is caused by media element 0.51.

A causal object ( FIG. 3B ) with its own media element identifier is added to the object-oriented database 54 in response to the input of the causal association. Metadata associated with the media element representing the effect is updated to include a parameter representing the media element identifier of the cause object. In more detail, the media object has a linked list of pointers to any cause objects that lead to or are caused by the media object. The cause object also maintains two linked lists of pointers back to the media objects it is associated with.

The last parameter shown in Figure 3 shows an example. The cause object also includes metadata representing the identifier of the media element that caused the effect.

Multiple different media objects can cause the same effect (e.g. a hero will die from being poisoned, or be crushed to death), and a single cause can cause many different effects (e.g. an undefeated demon The queen would live to be 103 years old, while the heartbroken princess took a vow of celibacy and became a nun). Thus, cause-effect relationships are represented using cause objects.

The second type of relationship an editor can indicate is an order relationship. To indicate this relationship, the editor arranges the media elements he wishes to group in the same rectangular area of the screen, sorts them from left to right according to the order they should follow, clicks on another button (not shown), Causes the cursor to change shape and moves the cursor to a position on one of the corners of the rectangular area. Thereafter, the editor clicks the button with the mouse 22, keeps the button pressed, and simultaneously moves to the opposite corner of the rectangular area, and releases the button at the opposite corner. The result is a thick solid rectangular line drawn around the media elements contained within the rectangular area, while a thick short arrow is drawn in the upper left corner of the area bounded by the rectangular line. In the example in FIG. 4C , the editor indicates that the media element representing Team B's fifteenth pass is immediately followed by the media element representing Team B's sixteenth pass, while the media element representing Team B's sixteenth pass The media element representing Team B's first goal is immediately followed by the media element for the second pass.

Where the editor feels that these media elements should be displayed in the indicated order, the editor may wish to indicate an order relationship of this nature (if more than one of these media elements is selected by the template combiner module) . For example, media elements representing gardening at different times of the year may be arranged in a sequence such that a media element representing a spring garden comes before a media element representing a summer garden, etc.

When a sequence is generated in this manner, a sequence object is generated in the object-oriented database as a container object that holds the media objects associated with the media elements contained within the sequence. As will be seen hereinafter, it is possible to generate sequences which themselves contain multiple sequences. This hierarchical property is reflected in the first number in the identifier belonging to the sequence. In the case where the sequence consists only of individual media elements, the sequence identifier is of the form 1.x, where x Simply add 1. Therefore, the sequence shown in Figure 4C was given the identifier 1.1.

The media objects (ie, metadata) associated with each media element in the sequence have the location of the media element in which the sequence is added. The object-oriented database also records the fact that each media object is a child of (ie contained within) the newly created sequence object. An example of this sequence location metadata can be seen in the penultimate row of Figure 3.

In this example, there is no metadata to give a description of the group or sequence, but there may be metadata that can be accessed, for example, by right-clicking, selecting properties, and in the description field of the resulting dialog Enter text to enter.

A third relationship between media elements that an editor may wish to indicate is group affiliation. In the event the editor wishes to indicate that multiple media elements in the group were selected and then displayed together, the editor may do so. Groups are formed in the same manner as sequences, except that the order of the media elements on the screen does not matter and the editor clicks a third button (not shown) that represents the group before defining a rectangular area that Contains the media elements to be included in the group.

This operation creates a group object, a container object that holds the media objects associated with the media elements in the group. Group objects are also stored in the object-oriented database 54 .

Figure 4D represents the display the editor sees after completing the following operations: forming media elements 0.13, 0.14, and 0.15 into a first-level group 1.2; forming media elements 0.39, 0.40, and 0.41 into a second-level group sequence 0.3; forming media elements 0.52 and 0.53 form the second level group 1.4.

Fig. 4E shows that the editor combined the media element 0.51 and said third primary sequence 1.4 into a secondary group 2.1. This creates another set of objects in the database, which is a container object for another set of objects. In this way, a hierarchy of container objects can be built in the object-oriented database 54 .

Figure 4F shows that the editor set two primary sequences 1.1 and 1.3, primary group 1.2 and secondary group 2.1 as tertiary group 3.1.

Figure 4G represents the editor-defined four-level sequence.

FIG. 5 shows a plurality of relationships entered by an editor in hierarchical form, and thus reflects the object hierarchy that the editor has created within the object-oriented database 54 on the hard disk 16 .

Returning to FIG. 2, template generation tool 42 provides an interface for editors and/or producers to specify desired characteristics of media material, thereby generating template objects (FIG. 6). The tool stores the metadata making up the template objects in an object-oriented database 54 .

Similar to media objects, template objects used in this embodiment conform to a comprehensive predefined data model. As seen in Figure 6, the predefined data model includes a header field and a number of sections. Each part is a set including name field, query field, and constraint field (optional). The Template Composer tool prompts the user for a name for the template and indicates the section structure (the top-level section can itself contain multiple sections). The editor indicates the structure of the part, for example using a graphical user interface similar to the folder listing provided in Microsoft Windows Explorer. In the example given in Figure 6, the template has a three-part planar structure.

Since the module encodes media profile properties using complex queries and potentially deep structures, the editor interface splits the task of template generation among multiple roles. Someone assigned the editor role defines the top-level structure of that template, while someone assigned the producer role (producers typically have closer control over the actual product being produced) refines that structure and adds queries (request information from an object-oriented database). Specifically, as will be explained below, producers specify links to user profiles, thereby defining a "balance of power" between themselves and consumers.

The template generation tool provides an object browser that can be used to search for existing media objects and template objects. Modifications can be made to existing templates, and sections of templates can be copied into new templates.

Having defined the partial structure using the GUI described above, it is possible to use the Media Object Browser to provide the editor/producer with a GUI that facilitates the query formation process.

Editors use the graphical user interface to enter query strings for each section. The query string used for the first part in Figure 6 indicates that the URI of the candidate media object used to populate this slot in the template must have the string "Football Intro" in it.

Query strings for sections can be quite complex, as shown in the "Body" section of the template in Figure 6. Here, the method of writing the query in the XPath language and evaluating the query will be described after describing the user profile metadata.

In cases where the editor wishes to impose some constraints on the media elements represented by the media objects retrieved from the database in response to the query, the editor also enters constraints on these parts. The purpose of constraints is to limit how the template composer composes media objects. Possible examples of constraints include time (e.g. the section must be 5 minutes long), space (e.g. the show must be viewed on a 640*480 pixel monitor), or quantity (in the "title line" section of a news program There must be 5 news).

The user profile generation tool 44 provides a user interface for generating a user profile similar to that shown in FIG. 7 .

The user profile represents the preferences of a particular user. As with media objects and template objects, this data must conform to a predetermined data structure or schema. The schema specifies that individual users are identified by an identifier (the number "1" in this example). The "structure" element of the user profile represents the user's favorite things - in this example football team B, especially Paulo Di Canio and Trevor Sinclair, Essex cricket team, especially Nasser Hussain, actress Milla Jovovich and actor Jimmy Nail .

The Template Composer program module (FIG. 8) provides the process for automatically composing the edit-determined list prepared for compositing a collection of media objects into a user's personalized media profile. At the beginning (step 60), the Template Composer takes as its input a specific template, a specific user profile, and a pointer to a store of media objects (in this example, an indication of the location of the object-oriented database 54).

After specifying specific templates, user profiles, and storage of media objects, the Template Composer checks against any query template (FIG. 6) written in the XPath language (step 61). If such a query was found, it was decided to evaluate the XPath using the Apache project's Xalan processor.

For example, when considering the user profile in Figure 7 (specified when starting the Template Composer program module), the query

(Team is "!profile(Profile/Sports//Team)") AND

(Action is "Goal"))

resolved to

((Team is “Team B” OR “Essex”) AND (Action is “Goal”))

The template combiner then identifies the first part of the template (FIG. 8) and iterates through the hierarchy of the template (steps 62 to 75).

Each iteration (steps 62 to 75) includes the next part of the template found, and any queries in that part are performed in the object-oriented database 54 (step 62) to return a selection of relevant media objects.

The first iteration is associated with the section titled "Introduction" in Figure 6. The iteration begins by executing a query that contains this part (ie, the URI contains "Football Intro") (step 62). Figure 9A shows the selections that resulted in this example.

Then, in step 64, a tree is constructed that includes the selected media object as its "leaf node". This structure is generated by retrieving the parent of the first selected media object, followed by retrieving the parent of that parent, etc., until reaching a media object that has no parent associated with it (the "introduction" object in this example has no parent object, so it is the only object contained in the tree). Here, a singly-linked list from leaf objects to top-level containers is reconstructed. Check another selected leaf object (if more than one object was selected as a result of the query), and then check the leaf object's ancestors until encountering one that already exists in the linked list representing the first object's ancestors , or until another top-level container is encountered. This process is repeated for all other objects in the selection to reconstruct the minimal tree containing these objects.

As mentioned above, in the first iteration, the resulting tree contains only the "introduction" media object 0.1.

Subsequent steps during the loop (steps 66 to 72) of the instructions in the template assembler that alter the tree data structure stored and used in generating the edit determination list have no effect on the first part of the template, so These subsequent steps are described below in terms of the second iteration of the instruction loop executed in the second portion of the template.

Throughout the iterations, this tree structure is stored in the volatile memory 12 of the PC.

At the end of each iteration of the instruction set, it is determined (step 74) whether the last part in the template was considered. If not, the next part is identified (step 75) and the next iteration is performed.

A second iteration is performed based on the center portion of the soccer score highlight template (FIG. 6). In this case, the second query is performed using the user profile ( FIG. 7 ) in order to resolve the query taking into account the user's preferences as described above.

This query (step 62) results in the selection of media elements 0.12 and 0.53 (FIG. 10A). A new tree data structure is created as described above, and stores data representing the tree structure shown in FIG. 10B.

Thereafter, the selection of media objects is expanded to take into account the cause/effect relationships specified by the user (step 66). In particular, this step includes checking the metadata of each selected media object to find out how many cause objects are associated with that media object. If no cause object is found, the media object is moved to the list of resolved media objects. If only one cause object is found, select to transfer the cause object to the list of cause objects and transfer the media object to the list of resolved media objects. If more than one cause object is found, each possible cause object is added to the list of possible cause objects (if it is not already present in the list), and the media object is added to the unresolved media object in the list.

In this example, only one cause object was found (shown in FIG. 3B ), so the media object 0.53 is transferred to the list of resolved media objects (and added to the tree data structure associated with this iteration of the instruction loop 10C), and transfer the cause object to the cause object list.

In cases where the list of possible cause objects is generated, the cause object that caused most of the unparsed media objects can be found. This cause object is transferred to the list of cause objects, and the media object it causes is added to the above-mentioned list of resolved media objects. This process is repeated until the list of all unresolved media objects is empty. In this example, this step is not performed.

Each cause object in the cause object list is then checked to see how many media objects it was caused by. If only one media object caused the cause object, then the cause object is transferred to the list of resolved cause objects and the media object is added to the list of unresolved media objects. If more than one media object caused the cause object, these media objects are added to the list of possible media objects (if they do not already exist in the list).

In this example, since the only one cause object (CO1) in the list of cause objects is caused by only one media object (0.52), that cause object is transferred to the list of resolved cause objects, and the media object object (0.52) into the list of unparsed media objects.

With a list of possible cause objects obtained, the media objects that lead to most cause objects can be found. The cause object is then transferred to the list of resolved cause objects and the media object is added to the list of unresolved media objects. This operation is repeated until the list of cause objects is empty.

The above process is then repeated for any unresolved media objects (to trace the causality chain back to the original cause). Thus, in this example, the above process is repeated for media element 0.52 and results in the addition of media object 0.51 to the tree associated with that iteration (FIG. 10D).

Immediately after the tree is created (steps 62 to 66), the objects in the tree are sorted (steps 68 to 70).

The first stage of sorting (step 68) considers the ordering information entered by the user. The sorting is done using a known "quick sort" algorithm to set the nodes of the tree in the correct order as identified by the sequential position metadata associated with the object. The operation starts at the top of the tree and then moves towards the leaf nodes (ie, media objects) of the tree.

The second stage of sorting considers causal/effect links between members of a group or between descendants of group members (ie, objects further down the tree). In cases where groups do not have such a causal/effect link, this stage of sorting does not need to be performed on these groups.

The second phase of the sort is started by marking each member of the group in the tree with all causes and effects attached (if it is a media object) or any of its descendants with all causes and effects attached (if it is a container).

Furthermore, these tags are subsequently added to the object metadata to reflect the following logical association: if a leads to b and b leads to c, then a leads to c. Do the same to reflect the following logical association: If f is caused by e and e is caused by d, then f is caused by d.

Then, a known quicksort algorithm is used to ensure that causes appear before effects. As is known to those skilled in the art, the quicksort implementation enables the user to define a function that gives the order of two objects passed to it. In this example, a function is provided that says: a precedes b for a leading to b, and if c leads to d, then d follows c.

Thus, at the end of the sorting step (steps 68 to 70) in the second iteration, the media elements 0.12, 0.51, 0.52 and 0.53 form the leaf nodes of a tree corresponding to the The central part is linked.

The template combiner then evaluates any constraints and updates the tree accordingly (step 72). To evaluate time constraints, the durations of the individual media objects included in the tree are calculated by subtracting the "out" attribute from the "in" attribute, and then added together to obtain the actual duration. If the duration is found to be longer than the target duration, the media object is removed from the tree. If the duration is shorter than the target duration, the media object is added to the tree.

In this embodiment, tree pruning or growing is performed based on the interest value metadata associated with these media objects.

In a case where the actual duration is longer than the target duration, the following processing is performed:

1) If the difference between the target duration and the actual duration is less than the duration of the shortest media element in the tree, then terminate the process;

2) Otherwise, generate a list of media objects in the tree, and sort the list according to the order of the "value of interest" attribute.

3) Remove the media object with the lowest value from the tree (assuming it is not the cause of one or more other media objects in the list) and recalculate the actual duration. Recalculate the difference between the target duration and the actual duration, and repeat the above steps until the difference is less than the duration of the shortest remaining media element in the tree.

In the case where the actual duration is shorter than the target duration, the following processing is performed:

A) By removing the last AND operator of the query in that section and the condition after it, or if there is no AND operator, by adding an OR operator (followed by a statement such as "value of interest > 0.6") to modify the query. A new tree for the current section is then generated in the same manner as the original tree. This process is repeated until the actual duration is greater than the target duration. Thereafter, steps 1) to 3) above are performed.

After performing this pruning or growing, the second iteration ends.

It should be appreciated that the third iteration only produces a tree including media object 0.99.

When all parts are combined by media object metadata and sorted according to the provided query, constraints, and user preferences, the template combiner evaluates the leaves of the tree generated in three iterations of the loop. The media elements found at the nodes are concatenated to output (step 78) the edit-determined list (FIG. 11).

The edit determination list (FIG. 11) generated by the template combiner program module (46) is passed to the content combiner module (48). In this example, the content compositor module outputs scenes one by one as a streaming video display, or concatenates scenes together to generate a show file.

The content synthesizer provides a process for automatically composing a collection of media elements into a user's personalized media profile. The combiner is similar to a traditional non-linear editing system in that the combiner uses an edit determination list (FIG. 11) to locate and combine a collection of media elements into a complete media material. However, while traditional editing tools are only used in post-production and exist on the editor's premise, the content synthesizer can be at the edge of the network, or indeed on the user's device (such as a PC or Portable lightweight application used in personal digital recorders). This means that media material can be synthesized at the point of delivery, which has considerable benefits in terms of the level of personalization that can be achieved. Additionally, changes to any metadata including templates, media objects or user profiles can be dynamically reflected in the resulting media material.

When invoked by a user, the content synthesizer passes the user profile and templates to the template composer, which processes them as described above, and returns a list of edit confirmations to the synthesizer. The editorial determination list is then parsed to find out which media elements are required and how these media elements interact with each other. A timeline is then assembled from this information, which timeline includes references to the various media elements in the content store 50 . Apply transition effects (examples of which are defined in the ANSI/SMPTE258M/1993 standard) to the media where required. Finally, any title text and graphical overlays are parsed and rendered as a static image, which is then added to the timeline with high priority, ensuring they are visible on top of any other graphical elements.

After assembling the timeline, use appropriate media compositor techniques such as Apple Quicktime or Microsoft DirectShow to render the personalized display in real time. The modular nature of the synthesizer means that content synthesizers can be more easily implemented in embedded architectures such as set-top boxes.

The graphical user interface provided by the preferred embodiment of the present invention will now be described with reference to FIGS. 11 to 20 .

FIG. 12 shows the application window that appears when the media material application is running on the computer of FIG. 1. FIG. As can be seen, many well-known graphical user interface components are provided.

FIG. 13 shows the display after the user has selected a project to open by selecting "Project" using the File menu or the "Open" button. In the preferred embodiment, the directory structure in the content store 50 is organized into top-level project directories, each of which contains one or more "media cabinets," i.e., directories that hold collections of media elements, and a or more "template cabinets", ie, directories that hold collections of templates. As can be seen from Figure 13, when the user opens the "PaperMill 03.db" project, the project sub-window is displayed, and the project sub-window shows: the "Paper Mill03.db" project contains files called "Paper Media" and "juliamark ” and a template cabinet called “Paper Templates”.

Figure 14 shows an additional sub-window that opens when the user double-clicks the "Paper Media" media drawer. This subwindow displays multiple rows, each row containing multiple column entries. It can be seen that by moving the mouse pointer over the title bar of the sub-window, a list of all attributes included in the structured data model for describing media elements can be displayed to the user. By using a mouse click, the user is able to select which elements of the structured data model are displayed in the "Paper Media" media cabinet window. It should be noted that this structured data model corresponds strictly to the structured data model that underlies the metadata shown in FIG. 3 .

When one of the thumbnails provided in the first column of each row is double-clicked, the metadata associated with the media element represented by that thumbnail is displayed in another sub-window (FIG. 15). The value assigned to each property is editable. Additionally, certain attributes have vocabularies associated with them (ie, a limited number of predetermined values for the attribute). In this example, right-clicking in the value field will display a drop-down list containing these predetermined values. The user can select one of these predetermined values from the list in the usual manner.

When the user selects multiple (and possibly all) media elements contained in the sub-window shown in Figure 14, right-clicks and selects the "Relationships" option, another sub-window (Figure 16) is displayed, which The relationships between the various media elements included in the selection are shown. In a manner similar to that shown in Figures 4A-4G, groups of media elements are displayed by enclosing the corresponding thumbnails in a solid line border, similarly showing the order - except that the border includes an arrow symbol near the upper left corner. Also shown are two diamonds representing the causative object as described in the first embodiment. The cause object on the left indicates that there is a cause/effect relationship between media elements JKY08a and JKY08b.

In a modification of the first embodiment described above, a user may right-click on the diamond shape representing the cause object, and the user is then presented with a list of options, namely "backward", "forward", "both ways", "detach all ” and “Delete”. The first three represent the manner in which the tree expansion process described in the first embodiment above expands the tree when a media element connected with a cause object is selected. If "Forward" is selected, selecting JKY08a will cause JKY08b to be added to the tree, but not vice versa. "Backward" has the opposite effect, ie if JKY08b is selected, then JKY08a is added to the tree, not vice versa. If Bidirectional is selected, selecting either will cause the other to be added to the tree, expanding the tree. The selections of "backward", "forward" and "bidirectional" are stored as another row of metadata in FIG. 3B.

Figure 17 shows the "Vocabularies" sub-window that the user can open by selecting the "Vocabularies" option from the Edit menu. The structured data model underlying the media objects can again be seen in the upper left box of the sub-window. A user may define a vocabulary - ie, a list of acceptable values for any attribute the user selects in the data model. In Figure 17, the user has selected the attribute "Place" and has defined the values "By the canal", "PaperMill" and "Sainsbury" as possible values for this attribute - this vocabulary is associated with the currently selected item and is stored in Object-oriented database (54 among Fig. 2).

Figure 17 also shows the extensibility of the structured data model. This extensibility is achieved by allowing users to add domain specific attributes (remember, Figure 3 also includes examples of these attributes). This domain specifying attribute is shown in the box at the lower left of the sub-window shown in FIG. 17, and the buttons "Add" and "Del" can be used to add or delete the domain specifying attribute. These domain-specific parameters are added to the data model associated with the currently open project.

FIG. 18 shows the steps a user performs to generate template metadata similar to that shown in FIG. 6 . To generate media object metadata, the user first opens the project subwindow. The user then double-clicks on one of the displayed template cabinets (in the example shown, the user double-clicked the "Papertemplates" template cabinet) to open the "Query Builder" subwindow. It can be seen that the tree on the left side of the sub-window is similar to the table in Figure 6, and each "<new scene>" corresponds to a section.

Circles in the tree structure represent filters for selecting media objects from the media cabinet. The following three filters exist:

i) A circle containing "=, or >, or <, or ≠"

For a user-selected attribute, these filters select all media objects in that media cabinet that have a value for that attribute that is equal to, greater than, etc., that is shown next to the circle.

ii) A circle containing (*) represents a filter for selecting all media objects in this media cabinet that contain the string shown next to the circle.

The logical operators used to combine the results of this filtering operation are called combinators and are shown as triangles. There are three combinations called "Random", "Sequential" and "Either/Or".

Order combinators display the results of filter operations directly below the order combinator in the tree only in the order in which they are set in the tree. Each sequence combinator has a straight arrow through the sequence combinator - an example of which is shown in the first and third parts of "Machine (860)" shown in FIG. 18 .

A random combinator displays the results of one or more filters below the random combinator in a random order in the tree. Random combinators have a zigzag arrow through the random combinator. An example thereof is shown in the second part of "Machine (860)" shown in FIG. 18 .

Or a combinator (not shown) selects the result of the two filter branches below it. The OR combinator has a separating arrow through the OR combinator.

FIG. 19 shows how the user modifies the tree and thereby generates template data similar to that shown in FIG. 7 . When adding a filter, the user can select an attribute, and then can enter the desired value.

Another element that can be added to the tree is a "funnel". It acts like a filter, allowing only a user-defined number of randomly selected media objects to be promoted to higher levels of the tree.

Where multiple filters are nested, each filter is applied in turn to the results of the filters throughout the tree.

To generate the edit confirmation list, the user clicks the button marked "!" in the toolbar while selecting a template. This edit OK list is then added to the edit OK list history subwindow, which is displayed when the corresponding button on the toolbar is pressed. Right-clicking on the editorial confirmation list in this window provides the user with the following options: play the video according to the editorial confirmation list; preview the storyboard (the result is shown in Figure 20); import or export the editorial confirmation list ; Edit the edit confirmation list in the form of the original text or delete the edit confirmation list.

Another feature is provided in a modification of the preferred embodiment of the invention. A graphical user interface providing this feature to the user is shown in FIG. 21 . Display a checkbox to the user when the text "Jacky, Problem" is selected and a filter is inserted. When the user ticks the checkbox, the user is presented with a list of dropdown controllers as described. By selecting one of these controls, the user can apply a "controller" to that filter. This controller can be provided multiple times (it can be seen that the first controller is applied twice in the template shown in Figure 21).

Figure 22 shows a pair of slider controls that can be moved by the user. By, for example, dragging the topmost slider further to the left, the "Coneptual PlotValue" associated with controller #1 can be decreased (and vice versa). By changing the position of the slider, the user is able to change the template generated by the query builder directly from the media material preview window (FIG. 22) without redefining the filter. Not only does this provide an easier way of updating filters in templates, but it forms the basis for a simplified user interface that does not require access to the query builder, etc. In other words, non-technical users are provided with the means to generate custom video displays.

The invention can be implemented in many different ways. For example, the above-described embodiments may be altered in one or more of the following ways to provide alternative embodiments of the invention (this list is not exhaustive):

i) In the above embodiments, the input program module generates a data structure with a predetermined structure. In some embodiments, the predetermined structure is provided by a document conforming to XML Schema guidelines;

ii) The user can change the constraints in the template - for example, the user can be provided with a graphical user interface in which he can select the duration of the media material he wishes to see. The corresponding value can then be added to the template object by the template combiner program;

iii) One constraint that can be added to each selection or combination of selections can be a constraint on the number of times a given media element appears in the completed media material. For example, the constraint may specify that any media element can only appear once in a given media profile.

Claims (4)

1. automatic method of synthetic media article, described method comprises:

Specify storage, the template of media object and the user profiles of representing user's preference, this template is used to specify the desired characteristic of media article and has a plurality of parts, each part in these a plurality of parts all comprises inquiry, each media object all is associated with corresponding media elements and comprises the digital metadata relevant with its respective media element, and this digital metadata comprises:

The related media object identification data is used to discern the related media object; And

Relation data, this relation data represent by and the content of the corresponding media elements of current media object representative and by and the content of the corresponding media elements representative of described related media object between relation;

Find the various piece in the described template iteratively and each part carried out following the processing:

Use described user profiles to resolve and carry out any inquiry in this part, returning selection to a plurality of media object,

Structure comprises the tree of selected all media object as leaf node,

By add by appointment because of the media object of/fruit relation recognition cause as selected media object, come described tree is expanded,

According to named order information described tree is sorted,

According to described because of/fruit relation sorts to described tree,

Constraint according to appointment in the described template is added/is removed the media object in the described tree;

Determine tabulation to being connected with the corresponding media elements of media object in the described tree to generate editor; And

Use described editor to determine to tabulate complete media article is located and be combined as to media elements.

2. method according to claim 1 also comprises generating described related media object identification data and described relation data.

3. method according to claim 1, wherein the described metadata of each media object also comprises content-data, this content-data is represented the content by the corresponding media elements representative of described media object, and the described step of wherein carrying out inquiry comprises: according to described content-data, select one or more media elements from a plurality of media elements.

4. according to any one the described method in the aforementioned claim, wherein said media elements comprises video data.

Images