JP7420585B2 - AR display control device, its program, and AR display system - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) “The Institute of Electronics, Information Engineers, General Incorporated Association” announced the “Web Proceedings” on December 4, 2019

Application of Article 30, Paragraph 2 of the Patent Act (2) Announced on December 13, 2019 at the “HCG Symposium 2019” held by the “Institute of Electronics and Information Engineers” from December 11 to 13, 2019

The present invention relates to an AR display control device that causes an AR display device to display a partner, a program thereof, and an AR display system.

In recent years, research on virtual reality (VR) and augmented reality (AR) has been conducted. For example, a technology has been proposed in which a user's alter ego (avatar) is displayed in a three-dimensional virtual space and the three-dimensional virtual space is shared among the users (Patent Documents 1 and 2). The methods described in these Patent Documents 1 and 2 align the lines of sight of each other's alter egos in order to promote dialogue within a three-dimensional virtual space.

Japanese Patent Application Publication No. 9-6985 Japanese Patent Application Publication No. 2003-150978

The techniques described in Patent Documents 1 and 2 cannot handle live-action models of users, and it is also difficult for users to view television programs while sharing a three-dimensional virtual space. For this reason, it is practical to use AR display devices such as AR glasses for space sharing when viewing television programs. In other words, by displaying the other party on the user's AR display device when viewing a television program, it is possible to realize space sharing in viewing the television program.

The inventors of the present application are also conducting research on space sharing. For example, when two people are watching a TV program, when the other person is in front of the user (for example, within the user's field of vision), the impression that ``two people are watching the TV program'' becomes stronger. It is considered to be.

Here, consider a case where the AR display device displays the other party in front of the user so as to be within the user's field of vision. In this case, even if the other party looks back at the user in the real world, the other party is located in front of the user on the user's AR display device, so the other party does not look back at the user at all. In this manner, in the AR display device, the other party is displayed in a state where the other party only turns halfway, which gives the user a sense of discomfort.

An object of the present invention is to provide an AR display control device, a program therefor, and an AR display system that can reduce discomfort when sharing a space.

In order to solve the above problems, an AR display control device according to the present invention connects a second person who views another video display device to an AR display device worn by a first person who views the video display device. An AR display control device that displays images on the side of a video display device viewed by a person, the AR display control device comprising: a line-of-sight information receiving section, a mode determining section, a redirection mode display control section, a real mode display control section, and a return mode display control section. The motion mode display control section is configured to include a motion mode display control section.

According to this configuration, the line-of-sight information receiving unit receives line-of-sight information indicating whether the second person is looking at the first person or whether the second person has turned away from the first person. .
The mode determination unit determines that the second person is in redirection mode when the second person continues to look at the first person for a predetermined time or more based on the line of sight information, and the mode determination unit determines that the second person is in redirection mode when the second person looks away from the first person. If the second person continues to look away from the first person for a predetermined period of time after the redirection mode is determined, the mode is determined to be the return motion mode.

When the redirection mode is determined, the redirection mode display control unit causes the AR display device to display the second person looking back at the first person.
When the real mode is determined, the real mode display control unit displays the second person in an actual posture on the AR display device.
When the return motion mode is determined, the return motion mode display control unit causes the AR display device to display a state in which the second person, who is turning toward the first person, returns to its actual posture.
In this way, in the AR display device, even when the second person is displayed in front of the first person, the second person is in a state where the second person is looking back at the first person.

Further, in order to solve the above-mentioned problems, an AR display system according to the present invention uses a photographing camera that photographs a second person viewing another video display device, and a photographed image of the photographing camera to display the second person. The configuration includes a three-dimensional model/posture information generation device that generates a three-dimensional model and posture information, and the above-mentioned AR display control device.

According to this configuration, the AR display control device connects the second person to the AR display device worn by the first person viewing the video display device so that the second person is on the side of the video display device viewed by the first person. to be displayed.
In this manner, in the AR display system, even when the second person is displayed in front of the first person, the second person is in a state looking back at the first person.

Note that the present invention can also be realized by a program for causing a computer to function as the above-mentioned AR display control device.

According to the present invention, even when the second person is displayed in front of the first person, the second person is in a state where he or she is looking back at the first person, which reduces discomfort when sharing space. can do.

1 is a schematic diagram of an AR display system according to a first embodiment. FIG. 4 is an explanatory diagram illustrating space sharing in viewing a television program, in which (a) shows how to display one's own AR glasses, and (b) shows how to display the other party's AR glasses. FIG. 3 is an explanatory diagram illustrating a state in which the other party is looking back at oneself, in which (a) shows how to display one's own AR glasses, and (b) shows how to display one's AR glasses. FIG. 4 is an explanatory diagram illustrating redirection, in which (a) shows how to display your own AR glasses, and (b) shows how to display the other party's AR glasses. (a) is an explanatory diagram illustrating space sharing in viewing a TV program, (a) is an explanatory diagram illustrating a state in which the other party turns toward you, and (c) is an explanatory diagram illustrating redirection. . FIG. 1 is an overall configuration diagram of an AR display system according to a first embodiment. FIG. 1 is a block diagram showing the configuration of a capture device according to a first embodiment. FIG. 1 is a block diagram showing the configuration of an AR display control device according to a first embodiment. (a) to (c) are explanatory diagrams illustrating changes in the position of the head and the posture of the body when a person turns around. 9 is a state transition diagram showing the operation of the AR display control device of FIG. 8. FIG. FIG. 2 is an explanatory diagram illustrating an example of motion path information. (a) and (b) are explanatory diagrams explaining motion paths in a redirection mode display control section and a return motion mode display control section. FIG. 2 is an overall configuration diagram of an AR display system according to a second embodiment. FIG. 3 is an overall configuration diagram of an AR display system according to a third embodiment.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, the embodiments described below are for embodying the technical idea of the present invention, and unless there is a specific description, the present invention is not limited to the following embodiments. Further, in each embodiment, the same means may be denoted by the same reference numerals, and a description thereof may be omitted.

(First embodiment)
[Overview of AR display system]
An overview of the AR display system 1 will be explained with reference to FIGS. 1 to 5.
As shown in FIG. 1, an AR display system 1 uses AR glasses (AR display device) 4 to realize space sharing in viewing television programs. Here, two people H ( _HA , _HB ) wearing AR glasses 4 (4 _A , 4 _B ) are watching a TV program on video display devices 6 (6 _A , 6 _B ) placed at different locations. are watching.

At this time, as shown in FIG. 2(a), the AR display system 1 displays the person (second person) _HB on the AR glasses _4A worn by the person (first person) _HA , and the person _HA The CG is synthesized and displayed on the side of the video display device _6A when viewed from above. Therefore, as shown in FIG. 5(a), the person _HB is displayed in front of the AR glasses _4A worn by the person _HA . _Similarly , the AR display system 1 _, as _shown in _FIG . Apparatus) 6 Composite and display CG on _{the B} side.
In addition, in FIGS. 2 to 4, the person H wearing the AR glasses 4 is shown with a solid line, and the person H displayed by CG composition is shown with a broken line.

Now, consider a case where person _HB turns to person _HA . As shown in FIG. 3(b), the person _HA is displayed in front of the person _HB 's AR glasses _4B . Therefore, the movement (amount of head rotation) when the person _HB turns towards the person _HA becomes smaller, but the head of the person _HB is still facing the direction of the person _HA . On the other hand, as shown in FIG. 3(a), the person HB is also displayed in front of the AR glasses _4A of the person _HA , so the person _HB is displayed in front of the person _H as shown in FIG. It will be in a state where you are not looking at _A completely. In other words, the amount of head rotation when the person _HB turns toward the person _HA is insufficient, and the head of the person _HB is not facing the direction of the person _HA . Therefore, in the AR display system 1, as shown in FIGS. 4(a) and 5(c), redirection processing is performed so that the person _HB completely turns around to face the person _HA .

In space sharing for viewing a television program, the display position of the person _HB can be set arbitrarily as long as it is in front of the person _HA (the same applies to the display position of the person _HA ). For example, the person _HB is displayed one meter in front of the person _HA . At this time, the person _HB may be displayed diagonally in front of the person _HA so as not to interfere with viewing the television program.
Hereinafter, one of the persons _HA and _HB may be referred to as "self," and the other person _HA and _HB may be referred to as the "other party."

[Configuration of AR display system]
The configuration of the AR display system 1 will be described with reference to FIGS. 1 and 6.
As shown in FIGS. 1 and 6, the AR display system 1 includes a camera 2, a capture device 3, AR glasses 4, and an AR display control device 5. In this AR display system 1, a camera 2 and a capture device 3 are connected via a cable. Further, in the AR display system 1, an AR display control device 5 is built into the AR glasses 4, and the capture device 3 and the AR display control device 5 are connected via a network NW such as the Internet.

Here, the AR display system 1 includes two cameras 2 ( _2A , _2B ) and two capture devices 3 ( _3A , _3B ) to correspond to the two people _HA , _HB . ), two AR glasses 4 (4 _A , 4 _B ), and two AR display control devices 5 ( 5 _A , 5 _B ). These cameras 2, capture devices 3, AR glasses 4, and AR display control devices 5 have the same configuration. Furthermore, the video display device 6 viewed by person _HA is referred to as video display device _6A , and the video display device 6 viewed by person _HB is referred to as video display device _6B .

The camera 2 is a general camera that photographs the person H, and outputs a photographed image of the person H to the capture device 3. This photographed video is necessary for the capture device 3 to generate a 3DCG model and bone information of the person H. Here, the camera _2A outputs the captured image of the person _HA to the capture device _3A , and the camera _2B outputs the captured image of the person _HB to the capture device _3B . For example, the camera 2 is placed at the top center of the video display device 6 so as to directly face the person H. Further, although the camera 2 is configured with one general RGB camera, it may be a stereo camera or an RGB camera with a depth camera.

The capture device 3 generates a 3DCG model of the person H and bone information of the person H using the captured image of the person H input from the camera 2. This 3DCG model is a three-dimensional model of person H. Further, the bone information is posture information representing the position of each joint of the person H, that is, the posture of the person H. Then, the capture device 3 transmits the generated 3DCG model and bone information of the person H to the AR display control device 5 via the network NW.

Here, your own 3DCG model and bone information are necessary to display yourself on the other party's AR glasses 4. Therefore, the capture device _3A continues to transmit the 3DCG model and bone information of the person _HA to the AR display control device _5B of the person _HB . Similarly, the capture device _3B continues to transmit the 3DCG model and bone information of the person _HB to the AR display control device _5A of the person _HA .
Note that the configuration of the capture device 3 will be described later.

The AR glasses 4 are equipped with an AR display 41 like general AR glasses, and also have an AR display control device 5 built in (FIG. 8). Further, the AR glasses 4 include a line-of-sight sensor 40 that detects the line-of-sight of the person H, and outputs the line-of-sight detection result by the line-of-sight sensor 40 to the AR display control device 5.

The AR display control device 5 displays, on the AR glasses 4 worn by the person viewing the video on the video display device 6, the person viewing the video on another video display device 6 on the side of the video display device 6. Further, the AR display control device 5 uses the 3DCG model and bone information of the other party received from the capture device 3 to perform a redirection process so that the other party completely turns toward the other party.

Here, the AR display control device 5 requires line-of-sight person information (line-of-sight information) in order to determine whether redirection processing is necessary. This gaze person information is information indicating whether the other party is looking at oneself or whether the other party has turned away from oneself. Therefore, the AR display control device 5 continues to receive the other party's line-of-sight person information from the other party's AR display control device 5 via the network NW.

To summarize the above, the AR display control device _5A of the person _HA displays the person _HB on the AR glasses _4A of the person HA using the 3DCG model and bone information of the person _HB received from the capture device _{3B .} Display. At this time, the AR display control device _5A determines whether redirection processing is necessary based on the line-of-sight person information of the person _HB received from the AR display control device _5B , and the person _HB completely turns around to face the person _HA . Perform redirection processing as follows. Further, the AR display control device _5A generates line-of-sight person information of the person _HA and transmits it to the AR display control device _5B .

Similarly, the AR display control device _{5B of} the person _HB displays the person HA on the AR glasses _4B of the person _HB using the 3DCG model and bone information of the person _HA received from the capture device _3A . let At this time, the AR display control device _5B determines whether redirection processing is necessary based on the line-of-sight information of the person _HA received from the AR display control device _5A , and the person _HA completely turns around to face the person _HB . Perform redirection processing as follows. Furthermore, the AR display control device _5B generates line-of-sight person information of the person _HB and transmits it to the AR display control device _5A .
Note that the configuration of the AR display control device 5 will be described later.

The video display device 6 is, for example, a general broadcast receiver that displays television programs. Furthermore, the video display device 6 may display video content other than television programs (for example, video content distributed over the network NW). Although the video display device 6 is not included in the AR display system 1, it is shown because it is closely related to the AR display system 1.

[Capture device configuration]
The configuration of the capture device 3 will be described with reference to FIG. 7. Note that in FIG. 7, illustration of the network NW is omitted.
The capture device 3 includes a captured video input section 30, a 3DCG model generation section 31, a bone information generation section 32, and a 3DCG/bone information transmission section 33.

The photographed video input unit 30 is for inputting the user's own photographed video from the camera 2. Then, the shot video input section 30 outputs the input shot video of the user to the 3DCG model generation section 31.

The 3DCG model generating section 31 generates a 3DCG model of the user using his or her photographed video input from the photographed video input section 30.
Here, the 3DCG model generation unit 31 can generate the 3DCG model using any method. When the camera 2 is configured with one RGB camera, the 3DCG model generation unit 31 can generate a 3DCG model by deep learning. For example, the 3DCG model generation unit 31 may generate a 3DCG model from each frame included in the video shot by the user using a neural network trained on photographs of people. Further, when the camera 2 is a stereo camera, the 3DCG model generation unit 31 can obtain depth information from the stereo image and generate a 3DCG model from this depth information. Further, when the camera 2 is an RGB camera with a depth camera, the 3DCG model generation unit 31 can generate a 3DCG model in the same way as a stereo camera, since the image of the depth camera represents depth information.
Then, the 3DCG model generation section 31 outputs its own generated 3DCG model to the bone information generation section 32 and the 3DCG/bone information transmission section 33.

The bone information generation unit 32 uses the 3DCG model of the user input from the 3DCG model generation unit 31 to generate bone information of the user. That is, the bone information generation unit 32 generates bone information corresponding to the 3DCG model in order to operate the 3DCG model. Here, the bone information generation unit 32 can generate bone information using any method (for example, QuickRig described in Reference 1). Then, the bone information generation section 32 outputs the generated own bone information to the 3DCG/bone information transmission section 33.

Reference 1: Maya (registered trademark), [online], Autodesk Corporation, [searched on February 13, 2020], Internet <URL: https://www.autodesk.co.jp/products/maya/ overview〉

The 3DCG/bone information transmitter 33 transmits the own 3DCG model input from the 3DCG model generator 31 and the own bone information input from the bone information generator 32 to the other party's AR display control device 5. It is something.
Here, the 3DCG/bone information transmitter 33 can transmit the 3DCG model and bone information at an arbitrary frequency. For example, the 3DCG/bone information transmitter 33 can save the bandwidth of the network NW by reducing the frequency of transmitting 3DCG models. On the other hand, when the 3DCG/bone information transmitter 33 increases the frequency of transmitting bone information, the posture change of the 3DCG model becomes smoother.

[Configuration of AR display control device]
The configuration of the AR display control device 5 will be described with reference to FIG. 8. Note that in FIG. 8, illustration of the network NW is omitted.

As shown in FIG. 8, the AR display control device 5 includes a line-of-sight input section 50, a line-of-sight determination processing section 51, a line-of-sight person information transmission section (line-of-sight information transmission section) 52, and a 3DCG/bone information reception section (three-dimensional model posture information receiving section) 53, line-of-sight person information receiving section (line-of-sight information receiving section) 54, mode determining section 55, motion path information recording section 56, redirection mode display control section 57A, and real mode display control section 57B, a return motion mode display control section 57C, and a virtual display section 58.

The line-of-sight input unit 50 is for inputting the detection result of one's own line-of-sight from the line-of-sight sensor 40. Then, the line-of-sight input section 50 outputs the inputted line-of-sight detection result to the line-of-sight determination processing section 51 .

The line of sight determination processing unit 51 determines whether or not the user is looking at the other person based on the detection result of the user's line of sight input from the line of sight input unit 50 .
Here, the line-of-sight determination processing unit 51 receives information about the other party's area displayed on the AR glasses 4 from a virtual display unit 58 (to be described later), and receives the detection result of its own line-of-sight from the line-of-sight input unit 50. Ru. In this case, the line of sight determination processing unit 51 determines whether or not the user is looking at the other person based on whether or not the user's line of sight while wearing the AR glasses 4 overlaps the area of the other person displayed on the AR glasses 4. . In other words, the line-of-sight determination processing unit 51 determines that the user is looking at the other person when the user's line of sight overlaps with the other person's area. On the other hand, when the user's line of sight does not overlap with the other party's area, the line of sight determination processing unit 51 determines that the user has turned away from the other party.
Thereafter, the line-of-sight determination processing section 51 outputs the determination result of whether or not the user is looking at the other person to the line-of-sight person information transmitting section 52 as line-of-sight person information.

The line-of-sight person information transmitting unit 52 transmits the line-of-sight person information inputted from the line-of-sight determination processing unit 51 to the other party's AR display control device 5 via the network NW.

The 3DCG/bone information receiving unit 53 receives the 3DCG model and bone information of the other party from the capture device 3 of the other party via the network NW. Then, the 3DCG/bone information receiving section 53 outputs the received 3DCG/bone information of the other party to the mode determining section 55.

The line-of-sight person information receiving unit 54 receives the other party's line-of-sight person information from the other party's AR display control device 5 via the network NW. Then, the line-of-sight person information receiving section 54 outputs the received line-of-sight person information of the other party to the mode determining section 55 .

The mode determining unit 55 determines whether the mode is a redirection mode, a real mode, or a return motion mode based on the gaze person information of the other party inputted from the gaze person information receiving unit 54 .
Specifically, the mode determining unit 55 determines that the mode is redirection mode when the other party continues to look at the user for a predetermined period of time or more. In this case, the mode determination unit 55 instructs a redirection mode display control unit 57A, which will be described later, to select a redirection mode, and outputs 3DCG/bone information.
Further, the mode determining unit 55 determines that the mode is real mode when the other party looks away from the user. In this case, the mode determination unit 55 instructs a real mode display control unit 57B (described later) to select the real mode, and outputs 3DCG/bone information.
Further, the mode determining unit 55 determines that the mode is the return motion mode if the other party continues to look away from the user for a predetermined period of time or more after the mode is determined to be the redirection mode. In this case, the mode determination unit 55 instructs a return motion mode display control unit 57C, which will be described later, to select a return motion mode, and outputs 3DCG/bone information.

The motion path information recording unit 56 is a storage device such as a memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive) that records motion path information in advance. This motion path information is referred to by the redirection mode display control section 57A and the return motion mode display control section 57C. Note that the details of the motion path will be described later.

The redirection mode display control unit 57A controls the virtual display unit 58 to display the other party in the redirection mode when the mode determination unit 55 determines that the mode is the redirection mode. This redirection mode is a mode in which the AR glasses 4 are displayed so that the other party is completely facing towards you. At this time, the redirection mode display control unit 57A displays the other party's 3DCG model using a predetermined motion path so that it naturally turns toward itself.

Here, as shown in FIGS. 9(a) to (c), when the opponent turns around, the angle of the head does not simply change, but the center position of the head (center of gravity) P _C and the posture of the body change. also changes. In addition, in FIG. 9, the direction of the head is illustrated with a dashed line. Therefore, it is preferable that the redirection mode display control unit 57A causes the body of the 3DCG model (the upper body such as shoulders and arms) to follow the head so that the opponent's turning motion becomes natural. For example, the redirection mode display control unit 57A can make the body of the 3DCG model follow the head by using IK (Inverse Kinematics).

The real mode display control unit 57B controls the virtual display unit 58 to display the other party in the real mode when the mode determining unit 55 determines that the mode is the real mode. This real mode is a mode in which the opponent is displayed on the AR glasses 4 in the actual posture. In other words, the real mode display control unit 57B displays the opponent's 3DCG model in the posture of the bone information.

The return motion mode display control section 57C controls the virtual display section 58 to display the opponent in the return motion mode when the mode determination section 55 determines that the mode is the return motion mode. This return motion mode is a mode in which the AR glasses 4 are displayed so that the opponent who is facing the user returns to their actual posture. At this time, the return motion mode display control section 57C displays the opponent's 3DCG model returning to its actual posture using a motion path that is opposite to the motion path of the redirection mode display control section 57A.

The virtual display unit 58 synthesizes CG and displays the other party on the AR glasses 4 (AR display 41) according to the control from the redirection mode display control unit 57A, the real mode display control unit 57B, and the return motion mode display control unit 57C. It is something to do. That is, the virtual display unit 58 virtually displays the opponent on the AR glasses 4 in any of the redirection mode, real mode, or return motion mode. Furthermore, the virtual display unit 58 outputs information about the opponent's area displayed on the AR glasses 4 to the line-of-sight determination processing unit 51.

[Operation of AR display control device]
The operation of the AR display control device 5 will be explained with reference to FIG.
Hereinafter, the operation of the AR display control device _5A when displaying the person _HB on the AR glasses _4A of the person _HA will be described as an example. Further, the person _HA may be written as "self" and the person _HB may be written as "other party."

First, the overall operation of the AR display system 1 will be summarized.
The other party's camera _2B photographs the person _HB , and the other party's capture device _3B generates a 3DCG model and bone information of _{the person HB using} the captured image of the camera 2B. Then, the other party's capture device _3B transmits the 3DCG model and bone information of the person _HB to its own AR display control device _5A . Furthermore, the other party's AR display control device _5B transmits the line-of-sight person information of the person _HB to its own AR display control device _5A .

Then, the own AR display control device _5A displays the person _HB on the AR glasses _4A of the person _HA using the 3DCG model, bone information, and line-of-sight person information of the person _HB . At this time, the AR display control device _5A performs the operation described below.

As shown in FIG. 10, the mode determination unit 55 of the AR display control device _5A transitions from the initial state S0 to the real mode state S1. For example, when the AR glasses _4A are powered on or reset, the AR display control device _5A is put into an initial state.

In the real mode state S1, the mode determination section 55 instructs the real mode display control section 57B to select the real mode. Then, the real mode display control unit 57B displays the person _HB in the real mode on the AR glasses _4A of the person _HA .

Furthermore, in the real mode state S1, the mode determination unit 55 determines whether or not the person HB has looked at the person _HB based on the person _HB 's gaze information.
When the person _HB looks at himself, the mode determination unit 55 transitions to the real mode (determination processing) state S2.

In the real mode (determination process) state S2, the mode determination unit 55 determines that the time _t1 during which the person _HB continues to look at the person _HB is equal to or greater than a preset threshold _T1 , based on the person HB's line of sight information. Determine whether or not. This threshold T ₁ can be set to any value (for example, threshold T ₁ =1 second).
If the time t ₁ is equal to or greater than the threshold T ₁ , the mode determination unit 55 transitions to the redirection mode state S3.

Furthermore, in the real mode (determination processing) state S2, the mode determination unit 55 determines whether or not the person HB has turned away from the person _HB based on the person _HB 's line of sight information.
When the person _HB looks away from himself, the mode determination unit 55 transitions to the real mode state S1.
If the person H _B is not looking away from himself and the time t ₁ is less than the threshold T ₁ , the mode determination unit 55 maintains the real mode (determination processing) state S2.

In the redirection mode state S3, the mode determination section 55 instructs the redirection mode display control section 57A to select the redirection mode. Then, the redirection mode display control unit 57A displays the person _HB on the AR glasses _4A of the person _HA in the redirection mode.

Furthermore, in the redirection mode state S3, the mode determination unit 55 determines whether or not the person _HB has turned away from himself or not, based on the person _HB 's gaze information.
When the person _HB looks away from himself, the mode determination unit 55 transitions to a redirection mode (determination process) state S4.
When the person _HB is looking at himself, the mode determination unit 55 maintains the redirection mode state S3.

In the redirection mode (determination process) state S4, the mode determination unit 55 determines that the time _t2 during which the person _HB continues to look away from himself or herself is a preset threshold value T, based on the person _HB 's line of sight information. Determine whether it is ₂ or more. This threshold T ₂ can be set to any value (for example, threshold T ₂ =1 second).
If the time t ₂ is equal to or greater than the threshold T ₂ , the mode determination unit 55 transitions to the return motion mode state S5.

Furthermore, in the redirection mode (determination processing) state S4, the mode determination unit 55 determines whether or not the person HB has looked at the person _HB , based on the line-of-sight person information of the person _HB .
When the person _HB looks at himself, the mode determination unit 55 transitions to the redirection mode state S3.
If the person H _B is not looking at the user and the time t ₂ is less than the threshold T ₂ , the mode determination unit 55 maintains the redirection mode (determination processing) state S4.

In the return motion mode state S5, the mode determination section 55 instructs the return motion mode display control section 57C to select the return motion mode. Then, the return motion mode display control unit 57C displays the person _HB on the AR glasses _4A of the person _HA in the return motion mode.
When the return motion ends, the mode determination unit 55 transitions to the real mode state S1.

Note that the operation of the AR display control device _5B when displaying the person _HA on the AR _{glasses 4B} of the person HB is also the same, so the explanation will be omitted.

[Motion path]
Motion paths in the redirection mode display control section 57A and the return motion mode display control section 57C will be described with reference to FIGS. 11 and 12.

The redirection mode display control unit 57A changes the 3DCG model of the person _HB from the actual posture of the head of the person _HB (head angle θ ₁ ) to the angle at which the head of the person _HB turns toward the person HA (head angle θ ₂ ). Continuously display _A looking back. At this time, the redirection mode display control unit 57A uses the motion path information in the motion path information recording unit 56 to determine a motion path from head angle θ ₁ to head angle θ ₂ .

As shown in _FIG . 11, the motion path information MP includes head position (x _θ , y _θ , z _θ ) and head rotation amount (rx _θ , ry _θ , rz _θ ). In other words, the motion path information MP includes the head position (x _θ , y _θ , z _θ ) and the amount of head rotation (rx _θ , ry θ , rz _{θ ) from the front (angle θ = 0°) to the head angle θ .} ) shows the amount of change. Note that in the motion path information MP of FIG. 11, the head angles θ are spaced at intervals of 30°, but the intervals of the head angles θ are not particularly limited.

As shown in FIG. 12(a ₎ , the AR display control device _5A determines the head position (x, y, _z ) and head rotation amount (rx , ry, rz). Therefore, the redirection mode display control unit 57A determines the head position (x, y, z) and head rotation amount (rx, ry, rz) of the person H _B in the actual posture (head angle θ ₁ ). By converting the coordinates to a coordinate system based on H _A , the angle at which the person H _A is turned (head angle θ ₂ ) can be calculated, as shown in FIG. 12(b). This coordinate transformation is similar to that used when displaying the person _HB on the AR glasses 4A of the person _HA . The direction of the head of the person _HB is expressed by the direction ry with the Y-axis being vertical (the direction of the video display device _6B (upward direction in FIG. 12(b)) is 0 as the front of the person _HB . °).

Then, the redirection mode display control unit 57A causes the AR glasses _4A to display a motion path in accordance with the calculated head angle _θ2 such that the person _HB turns around to face the person _HA . Determination of a motion path will be described below using two specific examples.

<Determination of motion path: 1st example>
In the method of this first example, the time t=0 when the redirection mode is determined is set, and the motion path at each time t is determined so that the person _HB at time t= _T3 completely turns around the person _HA . (However, 0≦t≦T ₃ ).

Here, a function for calculating the head angle θ _t at each time t is expressed by the following equation (1). Furthermore, in order to redirect the head of the person _HB more naturally, a quadratic interpolation function shown in equation (2) below may be used.

First, the redirection mode display control unit 57A calculates the head angle θ _t at each time t from the head angles θ ₁ and θ ₂ using the function of equation (1) or equation (2). Note that this head angle _θt is calculated using a coordinate system based on the person _HB before coordinate transformation. Then, the redirection mode display control unit 57A determines the head position (x _t , y _t , z _t ) and the head position of the person _HB corresponding to the head angle θ _t at each time t from the motion path information MP in FIG. The partial rotation amount (rx _t , ry _t , rz _t ) is calculated. Thereafter, the redirection mode display control unit 57A determines the head position (x _t , y _t , z _t ) and head rotation amount (rx _t , ry _t , rz _t ) of the person H _B at each time t. Coordinates are converted to a coordinate system based on _A.

Furthermore, the person _HA may move during the redirection mode, and the angle at which the person _HA is turned (head angle θ ₂ ) may change. In this case, the redirection mode display control unit 57A periodically recalculates the head angles θ ₁ and θ ₂ to obtain the head angle θ _t and determines the _head position (x _t , y _t , z _t ) and the amount of head rotation (rx _t , ry _t , rz _t ).

<Determination of motion path: 2nd example>
In the method of the first example described above, the head position (x, y, z) and head rotation amount (rx, ry, rz) of the person _HB are immediately updated to the motion path information MP when the redirection mode is entered. If the position is set at a value of

Therefore, in the method of the second example, the head position (x, y, z) and the amount of head rotation (rx, ry, rz) of the person H _B in the actual posture are used to gradually adjust the head position of the motion path information MP. The position (x _θ , y _θ , z _θ ) and head rotation amount (rx _θ , ry _θ , rz _θ ) are approached.

To simplify the explanation, we will focus on the x-coordinate of the head position. The redirection mode display control unit 57A calculates the x' coordinate of the head position in the next frame from the x coordinate of the head position in the current frame using the following equation (3). This equation (3) is a recurrence equation using a preset constant k. As the value of this constant k increases, the head position (x _θ , y _θ , z _θ ) and head rotation amount (rx _θ , ry _θ , rz _θ ) of the motion path information MP are slowly approached (however, k≧1). Furthermore, if the constant k is set to 1, the result will be similar to the first example.
Note that the y-coordinate and z-coordinate of the head position and the amount of head rotation (rx, ry, rz) can also be calculated using the same recurrence formula as for the x-coordinate of the head position.

The return motion mode display control unit 57C displays on the AR glasses _4A such that the person _HB who is turning towards the person _HA returns to the actual posture using a motion path that is opposite to the motion path in the redirection mode display control unit 57A. Just let it happen. Here, the return motion mode display control section 57C can determine a motion path using the same method as the redirection mode display control section 57A, so a description thereof will be omitted.
Further, since the motion path when displaying the person _HA on the AR glasses _4B of the person _HB is also similar, the description thereof will be omitted.

[Action/Effect]
As described above, in the AR display system 1 according to the first embodiment, even when displaying the other party in front of you on your own AR glasses 4, the other party will be in a state where they are completely looking back at you, so the spatial It is possible to reduce the discomfort of sharing.
Furthermore, since the AR display system 1 has the same device configuration on its own side and on the other side, the system configuration can be simplified.

(Second embodiment)
[AR display system configuration]
With reference to FIG. 13, differences in the configuration of an AR display system 1B according to the second embodiment from the first embodiment will be described.

As shown in FIG. 13, the AR display system 1B according to the second embodiment differs from the first embodiment in that the capture device 3B is made into a server. In other words, the AR display system 1B includes two cameras 2B (2B _A , 2B _B ), one capture device 3B, two AR glasses 4 (4 _A , 4 _B ), and two AR displays. and a control device 5 ( _5A , _5B ).

The camera 2B is a general network camera that transmits a captured image of the person H to the capture device 3B via the network NW. At this time, the camera 2B may add identification information of the photographed person H to the photographed video. By referring to this identification information, it is possible to specify whether the captured image of the person _HA or the person _HB is captured by the capture device 3B, which will be described later.

The capture device 3B generates a 3DCG model of the person H and bone information of the person H using the photographed video of the person H input from the camera 2B. That is, the capture device 3B generates 3DCG models and bone information of the persons H _{A and} H _B. For example, the capture device 3B can be configured with a general server or a cloud server placed on the network NW. Then, the capture device 3B refers to the identification information, transmits the 3DCG model and bone information of the person _HA to the AR display control device _5B , and sends the 3DCG model and bone information of the person _HB to the AR display control device _5A. Send.
In other respects, the capture device 3B is similar to the first embodiment, so further explanation will be omitted.

[Action/Effect]
As described above, the AR display system 1B according to the first embodiment can reduce the sense of discomfort in space sharing, similarly to the first embodiment.
Furthermore, the AR display system 1B can be realized with one capture device 3B by making the capture device 3B a server, and the system configuration can be further simplified.

(Third embodiment)
[Configuration of AR display system]
With reference to FIG. 14, differences in the configuration of an AR display system 1C according to the third embodiment from the first embodiment will be described.

As shown in FIG. 14, an AR display system 1C according to the third embodiment differs from the first embodiment in that a capture device 3C is built into AR glasses 4C and that captured images are transmitted and received. In other words, the AR display system 1C includes two cameras 2C (2C _A , 2C _B ), two capture devices 3C (3C _A , 3C _B ), and two AR glasses 4C (4C _A , 4C _B ). and two AR display control devices 5 ( _5A , _5B ).

The camera 2C is a general network camera that transmits a captured image of the person H to the capture device 3C built in the AR glasses 4C of the other party via the network NW. That is, the camera _2A transmits the photographed image of the person _HA to the capture device _3B of the AR glasses _4CB worn by the person _HB . Further, the camera _2B transmits the captured image of the person _HB to the capture device _3A of the AR glasses _4CA worn by the person _HA .

The capture device 3C receives a captured image of the person H from the camera 2C via the network NW, and generates a 3DCG model and bone information of the person H using the received captured image. Further, the capture device 3C is built into the AR glasses 4C.
In other respects, the capture device 3C and the AR glasses 4C are the same as in the first embodiment, so further explanation will be omitted.

[Action/Effect]
As described above, the AR display system 1C according to the third embodiment can reduce the sense of discomfort in space sharing, similar to the first embodiment.
Furthermore, since the AR display system 1C transmits and receives captured images that have a smaller amount of data than the 3DCG model or bone information, it is possible to save the communication band of the network NW.

(Modified example)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes within a range that does not depart from the gist of the present invention.
In each of the embodiments described above, the AR display device is described as being an AR glass, but the present invention is not limited to this. For example, the AR display device may be a smartphone having an AR display function.
In each of the embodiments described above, the AR display control device is described as being built into the AR glasses, but the AR display control device may be provided outside the AR glasses.

In each of the embodiments described above, the AR display device is described as hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program that causes hardware resources such as a CPU, memory, and hard disk included in a computer to operate as the above-mentioned AR display device. These programs may be distributed via communication lines, or may be written and distributed on recording media such as CD-ROMs and flash memories.

1-1C AR display system 2-2C Camera 3-3C Capture device 4, 4C AR glasses 5 AR display control device 6 Video display device 30 Photographed video input section 31 3DCG model generation section 32 Bone information generation section 33 3DCG/bone information transmission Section 40 Gaze sensor 41 AR display 50 Gaze input section 51 Gaze determination processing section 52 Gaze person information transmission section (Gaze information transmission section)
53 3DCG/bone information receiving unit (3D model posture information receiving unit)
54 Line-of-sight person information receiving unit (line-of-sight information receiving unit)
55 Mode determination section 56 Motion path information recording section 57A Redirection mode display control section 57B Real mode display control section 57C Return motion mode display control section 58 Virtual display section

Claims (7)

A second person viewing another video display device is displayed on an AR display device worn by a first person viewing the video display device so as to be on the side of the video display device being viewed by the first person. An AR display control device,
a line-of-sight information receiving unit that receives line-of-sight information indicating whether the second person is looking at the first person or whether the second person has turned away from the first person;
Based on the line of sight information, if the second person continues to look at the first person for a predetermined period of time or more, the redirection mode is determined, and the second person looks away from the first person. If the second person continues to look away from the first person for a predetermined time or more after the redirection mode is determined, the mode is determined to be the return motion mode. A determination section;
When it is determined that the redirection mode is selected, a redirection mode display control unit causes the AR display device to display the second person looking back at the first person;
a real mode display control unit that displays the second person in an actual posture on the AR display device when the real mode is determined;
a return motion mode display control unit that causes the AR display device to display such that the second person who is turning towards the first person returns to an actual posture when the return motion mode is determined;
An AR display control device comprising: further comprising a motion path information recording unit that prerecords motion path information indicating a head position and a head rotation amount for each head angle of the second person,
The redirection mode display control unit coordinately transforms the head position and head rotation amount of the second person in the actual posture to display the head of the second person when looking back at the first person. 2. An angle is calculated, and the second person is displayed on the AR display device so as to look back at the first person using a motion path according to the calculated head angle. AR display control device. The return motion mode display control section displays the AR display so that the second person who is turning toward the first person returns to his or her actual posture with a motion path that is opposite to the motion path in the redirection mode display control section. The AR display control device according to claim 2, wherein the AR display control device is configured to cause the device to display the information. further comprising a 3D model posture information receiving unit that receives the 3D model and posture information of the second person,
The redirection mode display control unit, the real mode display control unit, and the return motion mode display control unit display the second person on the AR display device using the three-dimensional model and posture information of the second person. The AR display control device according to any one of claims 1 to 3, wherein the AR display control device displays an image. Based on the first person's line of sight detection result input from the AR display device, whether or not the first person's line of sight overlaps the area of the second person displayed on the AR display device. a line of sight determination processing unit that determines whether the first person is looking at the second person;
a line-of-sight information transmitting unit that transmits the determination result of the line-of-sight determination processing unit as the line-of-sight information to another AR display device worn by the second person;
The AR display control device according to any one of claims 1 to 4, further comprising the following. A program for causing a computer to function as the AR display control device according to any one of claims 1 to 5. a camera that photographs a second person viewing another video display device;
a three-dimensional model/posture information generation device that generates a three-dimensional model and posture information of the second person using a captured image of the photographing camera;
An AR display control device according to any one of claims 1 to 5,
The AR display control device displays the second person on the AR display device worn by the first person viewing the video display device, using the three-dimensional model and posture information of the second person. An AR display system characterized by displaying images on the side of a video display device that is viewed by a person.

Images