We analyzed changes in human gait (way of walking) that corresponded to changes in human gaze direction. As preliminary analysis of gaze-gait relations, we focused on arm and leg swing amplitudes as a measure of gait and analyzed the relationship between gaze and arm/leg swings. We observed a tendency for decreased swing amplitude in the arm further from the gazing direction, and for increased swing amplitude in the arm closer to the gaze direction. Our results suggest that it may be possible to estimate gaze from human gait.

We propose a tactile expression mechanism that can make physical contact and provide direction indications. We previously proposed a wearable robot that can provide physical contact for elderly support in outdoor situations. In our current scheme, wearable message robots, which we mounted on the user's upper arm, give such messages to users as navigational information, for example. Using physical contact can improve relationships between users and robots. However, our previous prototypes have a problem because the types of tactile expressions (that the robots can make) are limited. Thus, we propose a tactile expression mechanism using a pneumatic actuator array for wearable robots. Our proposed system consists of four pneumatic actuators and creates such haptic stimuli as direction indications as well as stroking a user's arm. Our wearable robots were originally designed as appropriate support and communication for two types of physical contact: notification and affection. Our proposed mechanism for physical contact and direction indications naturally extends not only notification but also the affection abilities of the robot. Our robots and our proposed mechanism are expected to support the mobility of senior citizens by reducing their anxiety on outings.

This research proposed a depth measurement error model of consumer depth cameras such as Microsoft KINECT, and its calibration method. These devices are originally designed for video game interface, thus, the obtained depth map are not enough accurate for 3D measurement. To decrease these depth errors, several models have been proposed, however, these models consider only camera-related parameters. Since the depth sensors are based on projector-camera systems, we should consider projector-related parameters. Therefore, we propose the error model of the consumer depth cameras especially the KINECT, considering both intrinsic parameters of the camera and the projector. To calibrate the error model, we also propose the parameter estimation method by only showing a planar board to the depth sensors. Our error model and its calibration are necessary step for using the KINECT as a 3D measuring device. Experimental results show the validity and effectiveness of the error model and its calibration.

We proposed a method to achieve positions and poses of multiple cameras and temporal synchronization among them by using blinking calibration patterns. In the proposed method, calibration patterns are shown on tablet PCs or monitors, and are observed by multiple cameras. By observing several frames from the cameras, we can obtain the camera positions, poses and frame correspondences among cameras. The proposed calibration patterns are based on pseudo random volumes (PRV), a 3D extension of pseudo random sequences. Using PRV, we can achieve the proposed method. We believe our method is useful not only for multiple camera systems but also for AR applications for multiple users.

We proposed a wearable partner agent, that makes physical contacts corresponding to the user’s clothing, posture, and detected contexts. Physical contacts are generated by combining haptic stimuli and anthropomorphic motions of the agent. The agent performs two types of the behaviors: a) it notifies the user of a message by patting the user’s arm and b) it generates emotional expression by strongly enfolding the user’s arm. Our experimental results demonstrated that haptic communication from the agent increases the intelligibility of the agent’s messages and familiar impressions of the agent.

We analyzed the gait changes by simulating left knee disorders in subjects. Our aim is a method that estimates the presence or the absence of leg disorders and the disordered parts from the image sequences of a subject's walking. However, gait can be changed not only by physical disorders but also such factors as neural disorders or aging. Thus we simulate the physical disorders using a knee brace. Healthy subjects wore knee braces. We compared the normal and simulated disordered walking (while wearing the knee brace) to analyze what changes occur in their gait due to physical disorders. We also analyzed whether the changes are common to all subjects. Analysis results show that the changes common to all subjects are caused by the simulated left knee disorders.

We proposed "Voisticky" and "Mixticky", schemes for browsing and recording memorandum-like sticky notes in a three-dimensional virtual space around a user with a smartphone. For intuitive browsing and recall of such memos and information, it is important to achieve browsability and contemporaneousness of the memos/information. Mixticky allows the user to place and peel off memos on a virtual balloon around her/him corresponding to each relative direction (e.g., front, left, or 45 [deg] from front to right). The user can record and browse voice, movie, image, handwriting, and text memos as virtual sticky notes in each direction by pointing the phone. The system employs snapping gestures with the smartphone as metaphors for the “put” and “peel off” motions corresponding to a physical sticky note. The virtual balloon can be carried in various scenarios so that the user can easily resume her or his thought processes anywhere.

We proposed a real-time gaze estimation method based on facial-feature tracking using a single video camera that does not require any special user action for calibration. Many gaze estimation methods have been already proposed; however, most conventional gaze tracking algorithms can only be applied to experimental environments due to their complex calibration procedures and lacking of usability. In this paper, we propose a gaze estimation method that can apply to daily-life situations. Gaze directions are determined as 3D vectors connecting both the eyeball and the iris centers. Since the eyeball center and radius cannot be directly observed from images, the geometrical relationship between the eyeball centers and the facial features and eyeball radius (face/eye model) are calculated in advance. Then, the 2D positions of the eyeball centers can be determined by tracking the facial features. While conventional methods require instructing users to perform such special actions as looking at several reference points in the calibration process, the proposed method does not require such special calibration action of users and can be realized by combining 3D eye-model-based gaze estimation and circle-based algorithms for eye-model calibration. Experimental results show that the gaze estimation accuracy of the proposed method is 5 [deg] horizontally and 7 [deg] vertically. With our proposed method, various application such as gaze-communication robots, gaze-based interactive signboards, etc. that require gaze information in daily-life situations are possible.

This paper proposes a video communication assist system using a companion robot in coordination with the user's conversational attitude toward the communication. In order to maintain a conversation and to achieve comfortable communication, it is necessary to provide the user's attitude-aware assistance. First, the system estimates the user's conversational state by a machine learning method. Next, the robot appropriately expresses its active listening behaviors, such as nodding and gaze turns, to compensate for the listener's attitude when she/he is not really listening to another user's speech, the robot shows communication-evoking behaviors (topic provision) to compensate for the lack of a topic, and the system switches the camera images to create an illusion of eye-contact, corresponding to the current context of the user's attitude. From empirical studies and a demonstration experiment, i) both the robot's active listening behaviors and the switching of the camera image compensate for the other person's attitude, ii) elderly people prefer long intervals between the robot's behaviors, and iii) the topic provision function is effective for awkward silences.

We introduced an interactive guide plate system by adopting a gaze-communicative stuffed-toy robot and a gaze-interactive display board. An attached stuffed-toy robot on the system naturally show anthropomorphic guidance corresponding to the user’s gaze orientation. The guidance is presented through gaze-communicative behaviors of the stuffed-toy robot using joint attention and eye-contact reactions to virtually express its own mind in conjunction with b) vocal guidance and c) projection on the guide plate. We adopted our image-based remote gaze-tracking method to detect the user’s gazing orientation. The results from both empirical studies by subjective / objective evaluations and observations of our demonstration experiments in a semipublic space show i) the total operation of the system, ii) the elicitation of user’s interest by gaze behaviors of the robot, and iii) the effectiveness of the gaze-communicative guide adopting the anthropomorphic robot.

We proposed a body-mounted system to capture user experience as audio/visual information. The proposed system consists of two cameras (head-detection and wide angle) and a microphone. The head-detection camera captures user head motions, while the wide angle color camera captures user frontal view images. An image region approximately corresponding to user view is then synthesized from the wide angle image based on estimated human head motions. The synthesized image and head-motion data are stored in a storage device with audio data. This system overcomes the disadvantages of head-mounted cameras in terms of ease of putting on/taking off the device. It also has less obtrusive visual impact on third persons. Using the proposed system, we can simultaneously record audio data, images in the user field of view, and head gestures (nodding, shaking, etc.) simultaneously. These data contain significant information for recording/analyzing human activities and can be used in wider application domains such as a digital diary or interaction analysis. Experimental results demonstrate the effectiveness of the proposed system.

We proposed a method to estimate human head positions and poses by using head-mounted cameras and fixed cameras.

We proposed a distributed automatic method of calibrating cameras for multiple-camera-based vision systems. because manual calibration is a difficult and time-consuming task. However, the data size and computational costs of automatic calibration increase when the number of cameras is increased. We solved these problems by employing a distributed algorithm. With our method, each camera estimates its position and orientation through local computation using observations shared with neighboring cameras. We formulated our method with two kinds of geometrical constraints, an essential matrix and homography. We applied the latter formulation to a human tracking system to demonstrate the effectiveness of our method.