Interactive Sculpting of Digital Faces Using an Anatomical Modeling Paradigm

Digitally sculpting 3D human faces is a very challenging task. It typically requires either 1) highly-skilled artists using complex software packages for high quality results, or 2) highly-constrained simple interfaces for consumer-level avatar creation, such as in game engines. We propose a novel interactive method for the creation of digital faces that is simple and intuitive to use, even for novice users, while consistently producing plausible 3D face geometry, and allowing editing freedom beyond traditional video game avatar creation. At the core of our system lies a specialized anatomical local face model (ALM), which is constructed from a dataset of several hundred 3D face scans. User edits are propagated to constraints for an optimization of our data-driven ALM model, ensuring the resulting face remains plausible even for simple edits like clicking and dragging surface points. We show how several natural interaction methods can be implemented in our framework, including direct control of the surface, indirect control of semantic features like age, ethnicity, gender, and BMI, as well as indirect control through manipulating the underlying bony structures. The result is a simple new method for creating digital human faces, for artists and novice users alike. Our method is attractive for low-budget VFX and animation productions, and our anatomical modeling paradigm can complement traditional game engine avatar design packages.     

基于直方图统计学习的人脸检测方法

提出一种基于直方图统计学习的人脸检测方法,对人脸样本和非人脸样本进行小波变换,运用一组小波系数来表征各种人脸特征信息。统计每个训练样本的直方图分布,用于描述人脸和非人脸外观特征的概率分布,每个直方图表示一组小波系数与它们在人脸中位置的联合概率密度。该方法可以准确检测自然场景中的多幅人脸,对侧面人脸有很好的检测效果。     

Interactive Generation of Realistic Facial Wrinkles from Sketchy Drawings

Synthesizing facial wrinkles has been tackled either by a long process of manual sculpting on 3D models, or using automatic methods that do not allow for user interaction or artistic expression. In this paper, we propose a method that accepts interactive sketchy drawings depicting wrinkle patterns, and synthesizes realistic looking wrinkles on faces. The method inherits the simplicity of sketching, making it possible for artists as well as novice users to generate realistic facial detail very efficiently, allowing fast preview for physical makeup, or aging simulations for fun and professional applications. All strokes are used to infer the wrinkles, retaining the expressiveness of the sketches and realism of the final result at the same time. This is achieved by designing novel multi‐scale statistics tailored to the wrinkle geometry and coupled to the sketch interpretation method. The statistics capture the cross‐sectional profiles of wrinkles at different scales and parts of a face. The strokes are augmented with the statistics extracted from given example face models, and applied to an input face model interactively. The interface gives the user control over the shapes and scales of wrinkles via sketching while adding extra details required for realism automatically.     

Detection, Localization, and Sex Classification of Faces from Arbitrary Viewpoints and under Occlusion

This paper presents a novel framework for detecting, localizing, and classifying faces in terms of visual traits, e.g., sex or age, from arbitrary viewpoints and in the presence of occlusion. All three tasks are embedded in a general viewpoint-invariant model of object class appearance derived from local scale-invariant features, where features are probabilistically quantified in terms of their occurrence, appearance, geometry, and association with visual traits of interest. An appearance model is first learned for the object class, after which a Bayesian classifier is trained to identify the model features indicative of visual traits. The framework can be applied in realistic scenarios in the presence of viewpoint changes and partial occlusion, unlike other techniques assuming data that are single viewpoint, upright, prealigned, and cropped from background distraction. Experimentation establishes the first result for sex classification from arbitrary viewpoints, an equal error rate of 16.3 percent, based on the color FERET database. The method is also shown to work robustly on faces in cluttered imagery from the CMU profile database. A comparison with the geometry-free bag-of-words model shows that geometrical information provided by our framework improves classification. A comparison with support vector machines demonstrates that Bayesian classification results in superior performance.     

漫画风格的人脸肖像生成算法

为模拟艺术家画出的具有夸张效果的肖像漫画,由用户输入正面人脸照片,通过交互获取关键特征点,根据特征点计算面部特征值来判断需要夸张的部分和各自的夸张方式;然后根据特征点自动生成不同层级的网格并分层实施夸张变形;最后进行图像处理以获得不同艺术风格的图像.该算法从研究漫画家的作画风格入手,总结了肖像漫画特定的夸张规律,所生成的漫画风格人脸肖像效果较好,能够应用于非真实感图形学和数字娱乐等领域.     

Local statistics of retinal optic flow for self-motion through natural sceneries

Image analysis in the visual system is well adapted to the statistics of natural scenes. Investigations of natural image statistics have so far mainly focused on static features. The present study is dedicated to the measurement and the analysis of the statistics of optic flow generated on the retina during locomotion through natural environments. Natural locomotion includes bouncing and swaying of the head and eye movement reflexes that stabilize gaze onto interesting objects in the scene while walking. We investigate the dependencies of the local statistics of optic flow on the depth structure of the natural environment and on the ego-motion parameters. To measure these dependencies we estimate the mutual information between correlated data sets. We analyze the results with respect to the variation of the dependencies over the visual field, since the visual motions in the optic flow vary depending on visual field position. We find that retinal flow direction and retinal speed show only minor statistical interdependencies. Retinal speed is statistically tightly connected to the depth structure of the scene. Retinal flow direction is statistically mostly driven by the relation between the direction of gaze and the direction of ego-motion. These dependencies differ at different visual field positions such that certain areas of the visual field provide more information about ego-motion and other areas provide more information about depth. The statistical properties of natural optic flow may be used to tune the performance of artificial vision systems based on human imitating behavior, and may be useful for analyzing properties of natural vision systems.     

Automatic Portrait Segmentation for Image Stylization

Portraiture is a major art form in both photography and painting. In most instances, artists seek to make the subject stand out from its surrounding, for instance, by making it brighter or sharper. In the digital world, similar effects can be achieved by processing a portrait image with photographic or painterly filters that adapt to the semantics of the image. While many successful user‐guided methods exist to delineate the subject, fully automatic techniques are lacking and yield unsatisfactory results. Our paper first addresses this problem by introducing a new automatic segmentation algorithm dedicated to portraits. We then build upon this result and describe several portrait filters that exploit our automatic segmentation algorithm to generate high‐quality portraits.     

Prediction of human eye movements in facial discrimination tasks

Under natural viewing conditions, human observers selectively allocate their attention to subsets of the visual input. Since overt allocation of attention appears as eye movements, the mechanism of selective attention can be uncovered through computational studies of eyemovement predictions. Since top-down attentional control in a task is expected to modulate eye movements significantly, the models that take a bottom-up approach based on low-level local properties are not expected to suffice for prediction. In this study, we introduce two representative models, apply them to a facial discrimination task with morphed face images, and evaluate their performance by comparing them with the human eye-movement data. The result shows that they are not good at predicting eye movements in this task.     

Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability

The ability to recognize facial emotions is target behaviour when treating people with social impairment. When assessing this ability, the most widely used facial stimuli are photographs. Although their use has been shown to be valid, photographs are unable to capture the dynamic aspects of human expressions. This limitation can be overcome by creating virtual agents with feasible expressed emotions. The main objective of the present study was to create a new set of dynamic virtual faces with high realism that could be integrated into a virtual reality (VR) cyberintervention to train people with schizophrenia in the full repertoire of social skills. A set of highly realistic virtual faces was created based on the Facial Action Coding System. Facial movement animation was also included so as to mimic the dynamism of human facial expressions. Consecutive healthy participants (n = 98) completed a facial emotion recognition task using both natural faces (photographs) and virtual agents expressing five basic emotions plus a neutral one. Repeated-measures ANOVA revealed no significant difference in participants’ accuracy of recognition between the two presentation conditions. However, anger was better recognized in the VR images, and disgust was better recognized in photographs. Age, the participant’s gender and reaction times were also explored. Implications of the use of virtual agents with realistic human expressions in cyberinterventions are discussed.     

Supporting the creative drive: investigating paralinguistic voice as a mode of interaction for artists with upper limb disabilities

For people with upper limb disabilities visual art is an important activity that allows for expression of individuality and independence. They show remarkable endurance, patience and determination to adapt their remaining capabilities to create visual art. There are significant advantages of digital technologies in assisting artists with upper limb disabilities. Paralinguistic voice recognition technologies have proven to be a particularly promising mode of interaction. Despite these benefits, technological support for people with upper limb disabilities to create visual art is scarce. This paper reports on a number of case studies of several artists with upper limb disabilities. These case studies illustrate the struggles they face to be creative and also show the significant advantages of digital technologies in assisting such artists. An investigation into people’s ability to use the volume of their voice to control cursor movement to create drawings on the screen is also reported. With motivation, training and practise, use of volume to control drawing tasks shows great promise. It is believed that paralinguistic voice has wider implications beyond assisting artists with upper limb disabilities, such as: an alternative mode of interaction for disabled people to perform tasks other than creating visual art, alternative mode of interaction for hands busy environments and as a voice training system for people with speech impairments.     

Computer‐Suggested Facial Makeup

Finding the best makeup for a given human face is an art in its own right. Experienced makeup artists train for years to be skilled enough to propose a best‐fit makeup for an individual. In this work we propose a system that automates this task. We acquired the appearance of 56 human faces, both without and with professional makeup. To this end, we use a controlled‐light setup, which allows to capture detailed facial appearance information, such as diffuse reflectance, normals, subsurface‐scattering, specularity, or glossiness. A 3D morphable face model is used to obtain 3D positional information and to register all faces into a common parameterization. We then define makeup to be the change of facial appearance and use the acquired database to find a mapping from the space of human facial appearance to makeup. Our main application is to use this mapping to suggest the best‐fit makeup for novel faces that are not in the database. Further applications are makeup transfer, automatic rating of makeup, makeup‐training, or makeup‐exaggeration. As our makeup representation captures a change in reflectance and scattering, it allows us to synthesize faces with makeup in novel 3D views and novel lighting with high realism. The effectiveness of our approach is further validated in a user‐study.     

Learning to sketch human facial portraits using personal styles by case-based reasoning

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists’ personal illustration styles to the peer methods.     

Locating human faces in photographs

The human face is an object that is easily located in complex scenes by infants and adults alike. Yet the development of an automated system to perform this task is extremely challenging. An attempt to solve this problem raises two important issues in object location. First, natural objects such as human faces tend to have boundaries which are not exactly described by analytical functions. Second, the object of interest (face) could occur in a scene in various sizes, thus requiring the use of scale independent techniques which can detect instances of the object at all scales.Although, the task of identifying a well-framed face (as one of a set of labeled faces) has been well researched, the task of locating a face in a natural scene is relatively unexplored. We present a computational theory for locating human faces in scenes with certain constraints. The theory will be validated by experiments confined to instances where people's faces are the primary subject of the scene, occlusion is minimal, and the faces contrast well against the background.This work was supported in part by the NSF and The Eastman Kodak Company.     

正立和倒立面孔的混合识别

目的 改变正立和倒立面孔只是一种简单倒置关系的观点,研究基于视觉神经整体和局部信息流的正立和倒立面孔混合识别。方法 模拟视觉信息流在视通路中的传递和处理过程,首先构建底层神经网络,建立敏感纹理特征以及对称卷积核的机制,实现正立和倒立面孔图像的去除冗余和预处理;接着提出一种基于局部区域提取的池化神经网络层的概念,构建多局部特征融合的网络结构,实现局部信息的压缩提取和融合;最后根据高级视觉皮层中左右半脑协作的特点,提出一种融合整体和局部信息的预测函数。结果 以AT&T数据库为例,本文方法在经典卷积神经网络模型上增加了多局部特征融合的网络结构,识别准确率从98%提高到100%,表明局部信息能够提高对正立面孔识别的能力;同时采用合适的训练数据集,调节融合时整体与局部信息的关系比,结合使用合适模型训练方式,该模型对正立和倒立面孔的识别率分别为100%和93%,表明对正立和倒立面孔识别具有良好的特性。结论 本文方法说明了整体和局部特征的两条视觉通路虽然分别在正立和倒立面孔识别上起了决定性的作用,但它们并不是孤立存在的,两条通路所刻画的面孔信息应该是一种互补式的关系。不仅为面孔识别提供一种新思路,而且将有助于对视觉神经机制的进一步理解。     

Artistic Low Poly rendering for images

This paper presents an automatic approach for generating low poly rendering of images, which is particularly popular in the recent art design community. Distinguishing from the traditional image triangulation methods for the sake of compression or vectorization, we propose some critical principles of such Low Poly rendering problem, and simulate the artists creation procedures straightforwardly. To produce the visual effects with clear boundaries, we constrain the vertices along the feature edges extracted from the input image. By employing the Voronoi diagram iteration guided by a feature flow field, the vertices in the result image well reflect the feature structure of the local shape. Moreover, with the salient region detection, we can achieve different mesh densities between the front object and the background. Some special color processing techniques are employed to make our result more artistic. Our method works well on a wide variety of images, no matter raster photographs or artificial images. Experiments show that our approach is able to generate satisfying results similar to the artwork created by professional artists.     

Audio-visual human recognition using semi-supervised spectral learning and hidden Markov models

This paper presents a multimodal system for reliable human identity recognition under variant conditions. Our system fuses the recognition of face and speech with a general probabilistic framework. For face recognition, we propose a new spectral learning algorithm, which considers not only the discriminative relations among the training data but also the generative models for each class. Due to the tedious cost of face labeling in practice, our spectral face learning utilizes a semi-supervised strategy. That is, only a small number of labeled faces are used in our training step, and the labels are optimally propagated to other unlabeled training faces. Besides requiring much less labeled data, our algorithm also enables a natural way to explicitly train an outlier model that approximately represents unauthorized faces. To boost the robustness of our system for human recognition under various environments, our face recognition is further complemented by a speaker identification agent. Specifically, this agent models the statistical variations of fixed-phrase speech using speaker-dependent word hidden Markov models. Experiments on benchmark databases validate the effectiveness of our face recognition and speaker identification agents, and demonstrate that the recognition accuracy can be apparently improved by integrating these two independent biometric sources together.     

Modeling and Animating Realistic Faces from Images

We present a new set of techniques for modeling and animating realistic faces from photographs and videos. Given a set of face photographs taken simultaneously, our modeling technique allows the interactive recovery of a textured 3D face model. By repeating this process for several facial expressions, we acquire a set of face models that can be linearly combined to express a wide range of expressions. Given a video sequence, this linear face model can be used to estimate the face position, orientation, and facial expression at each frame. We illustrate these techniques on several datasets and demonstrate robust estimations of detailed face geometry and motion.     

A statistic approach to the detection of human faces in color nature scene

In this paper, a novel algorithm for oriental face detection is presented to locate multiple faces in color scenery images. A binary skin color map is first obtained by applying the skin/non-skin color classification algorithm. Then, color regions corresponding to the facial and non-facial areas in the color map are separated with a clustering-based splitting algorithm. Thereafter, an elliptic face model is devised to crop the real human faces through the shape location procedure. Last, local thresholding technique and a statistic-based verification procedure are utilized to confirm the human faces. The proposed detection algorithm combines both the color and shape properties of faces. In this work, the color span of human face can be expanded as wilder as possible to cover different faces by using the clustering-based splitting algorithm. Experimental results reveal the feasibility of our proposed approach in solving face detection problem.