Human face reconstruction

Project 1 (ESR1):  Distributed optimization of biokinetic models based on large 4D sequences (IST Lisbon & 3Lateral)

Companies working in production of videogames or virtual reality devices use biokinetic models of human form in computer graphics field. In particular 3Lateral is posing a strong focus on the realistic reconstruction of human face expressions. This is done based on a database of biokinetic models through 3D and 4D (spatial or space-time) scanning of real human faces and accurate registration of this data for purpose of statistical modeling and machine learning. Biokinetic model or “Rig Logic”, as it is called in the industry, is composed by a series of vectors representing muscle contractions, appearance of which is obtained through scanning, and layers of corrective vectors. In order to be compatible with real time rendering engines, the model can be seen as hand crafted non linear principal component analysis (PCA) and it models the deformations of the human face from parameters to mesh. Given the function curve for each parameter, animation of the model can be created in order to render the appearance of the human face. Evaluation of Rig Logic parameters based on 4D data is done through a series of both linear and non-linear optimizations. 4D data sets can be extremely large (millions of frames) and often Rig Logic has to be calculated in real time (max 200ms delay from event to render). For this reason the student’s main goal will be to provide and implement innovative techniques of distributed optimization on a GPU cluster in an efficient way, in order to enhance human face and body representation. The optimization techniques will be developed also for the new geometric models developed in Project 2 to enhance face representation.

Project 2 (ESR2) : Mathematical morphology for the prediction of face expression transition (Univ. of Milan & 3Lateral)

A second challenge connected with the problem described in Project1 concerns a realistic description of human face expression transition. Human face expressions can be classified into about 60 different classes, but the transition from one class to another must be sufficiently “smooth” to avoid producing motion and perception artefacts. Actually a linear interpolation between the geometrical descriptions of the different classes in many cases is not sufficiently realistic. Thus the goal of the student enrolled in this project will be to develop and apply new stochastic geometric techniques to the 4D data scanned from real human face expression, to model correctly the transition of the space-time distributions of the landmarks or surfaces describing the expression. The developed models will take into account both the possible imbalance in the sampling of face expressions and the real-time computability requirements of the company, and will thus be reduced to fasten the computation, and optimized in a distributed manner in collaboration with the student of Project 1.

Project 3 (ESR3): Stochastic Geometric modelling and 3D image analysis for human face prostheses (Univ. of Milan & uROBOPTICS)

Extracting usable parametric geometry models from point data is an open challenge. Assuming a set of point clouds obtained from instances of an (unknown) parametric class of 3D objects, the challenge is to recover the underlying parametric surface model, knowing that point clouds are corrupted with noise, missing data, and are non-uniformly sampled with different densities. Prior work, describing human lower legs, was capable of achieving the required objectives with a dataset consisting of a hundred human scans. In this project, surface models with high intrinsic curvature will be considered, requiring both different modelling techniques and the creation of much larger real-world datasets. Registration of these datasets in a common reference frame, prior to model extraction, is a common pre-processing operation, consisting of identifying shared features, which can be pre-aligned. The industrial goal for this student will be to recover models of human face features (e.g. ears, noses) for the prosthetic industry, which require high quality colour models. In this setting, model instantiation is constrained by the border conditions of the existing face shape and texture to which the generated model will need to fit.

Financial Processes

Project 4 (ESR4): Credit scoring and statistical prediction of credit default (IST Lisbon & CIF, SDG)

The variety and volume of financial data is ever-expanding. In the past decade, information coming from traditional sources (e.g. exchanges, databases of financial institutions, or commercial data providers) became comparable in both size and velocity to the one available in social media, mobile interactions, server logs or customer service records. Companies are increasingly turning to data scientists to seek a meaningful relationship with these vast amounts of data. The large number of decisions involved in the consumer lending business makes it necessary to rely on models and algorithms rather than human discretion, and to base such algorithmic decisions on “hard” information, e.g., characteristics contained in consumer credit files collected by credit bureau agencies. Models are typically used to generate numerical “scores” that summarize the creditworthiness of consumers and may estimate the probability of credit default. This project is focused on the improvement of existing credit scoring models and of the related prediction of default  by combining the “traditional” databases of companies and individuals (financial reports, behavioural data, bank data, credit bureaus, etc.) with loan servicing data (days in arrears, collateral, loan-to-value, etc.) and with other available sources, in particular those available through open social media activity or transaction data. In the first phase, suitable measures of relevance of the considered variables will be introduced, through modern techniques of machine learning and data mining. In the second phase, new statistical models, more interpretable than the “black box” techniques cited above, will be introduced, and then optimized to come up with optimal risk-based financial funding procedures. Depending on the introduced measures and risk functions, the optimization problem to be solved can be non-convex.

Project 5 (ESR5): Scoring individual financial investment potential (TU/e & AcomeA)

Modern Financial Technology (fintech) companies offer, besides traditional systems of investment, also online applications and services where single investors may deposit any amount of money, which can be rapidly claimed at any time. Such online systems are often used by small investors like students, young people, etc. as “piggy banks” or by individuals with a larger capital, in parallel to other investments, to diversify their portfolio. The aim of this project is to identify customers who have a financial potential larger than their actual investments, to apply targeted marketing strategies. The financial behaviour of individuals is related to their attitude to risk and to save money, and to their way of living, etc. In this project, we will retrieve this information via socio demographic data, geolocalized data, analysis of social networks, by defining relevant variables and suitable measures and distances that may quantify the possible features of interest. Then ESR5 will develop a feature extraction procedure to reduce the problem dimension and identify the more relevant variables to describe the financial behaviour of the individuals. The identification of such variables is crucial and challenging, due to the heterogeneity of information to be considered. Based on the selected variables, the customers having an unexploited financial potential will then be identified via quantile regression techniques.


Production Systems

Project 6 (ESR6): Prediction of failure events in complex productive systems (TU/e & SDG) 

With the advent of Industry 4.0, many current industrial processes are subject to continuous monitoring of their efficiency by sensors, which provide numerical data of various types, with a high frequency. The data gathered by these systems have various uses. Two important but difficult objectives are (1) prediction of events that can impair (damage) the process output, and (2) prediction and optimization of the quality of the process end-product. However, the availability of vast amount of data leads business users to even more ambitious objectives: (3) understanding the causes of failures and varying quality, and (4) taking actions to improve the process, to avoid failures or to reduce the failure rates.These goals are very challenging. Good and reliable prediction usually requires both linear and nonlinear models and algorithms, which are often difficult to interpret, so that causal relationships remain unclear. On the contrary, interpretable models often are unable to represent complex, nonlinear and dynamic relationships. Therefore, the student will develop efficient models that are able to predict with a good reliability the occurrence of process impairment, and to tell which features mostly contribute to the damage in the given conditions. Three key mathematical ingredients that we will use are: (a) compositionality: building complex models from simpler components; (b) model reduction: simplification of the model, so that it can be simulated and the parameters identified, without losing much accuracy;(c) causality: understanding what is causing what and take improvement-oriented actions. Current trends for impairment prediction frequently use deep learning models, which are flexible and often provide good results, but usually very hard to interpret. In this project, the student will introduce innovative more interpretable models. Such models will support the development of innovative methods for process optimization and control by the student enrolled in Project 7.

Project 7 (ESR7): Big data optimization for logistic and supply chain management (Univ. of Novi Sad & Sioux LIME)

Quality control of high tech components is very stringent, with admissible out of specification production going towards below the 1 ppm (one in a million) to 1 ppb (one in a billion). This, in effect, means it is not enough to minimize process variations: faulty products will be caused by an unknown combination of events. Because of the low occurrence rate, this becomes impossible to study by experiment, and root cause finding has to be performed on detailed analysis of the propagation of and interaction between process variations. The number of parameters and interactions and the stochastic nature of the underlying models make both the root cause finding itself and the mitigation by (non-linear) optimal control very big. An approach is a sparse tensor representation of the high dimensional space, which in itself is a challenging optimisation problem. Although it is a challengingly big computational approach, recent advances in computational equipment have made it nearly feasible: a further reduction in complexity in e.g. sparsity, and performance of underlying optimisation strategies is required. An even more data driven problem is at the logistic end of the process. Logistic and supply chain planning from A to B in a realistic network, with many different intermediate actors, each one having their own incentives and objectives, results into a massively complex, heavily constrained and possibly non-convex optimisation problem. The aim of this project is to improve the control and adaptability of production processes and of supply chain management, under uncertainty conditions of a large scale set of (constrained) variables. To tackle the problem, the student will develop innovative optimization algorithms that can cope with huge amount of data in real time, to be applied to a mathematical model defined in such way that all important factors are well identified and able to model rare but significant events. The methods to identify causes of process impairment developed by the student enrolled in Project 6 will be of great value to set up the mathematical models studied in this project.