CS 194-26 Project 3

Overview

This project is centered around using user-defined facial keypoints to combine, morph, and warp faces in interesting ways!

Defining Correspondences

I stored my keypoints for each image as an array in text files, using the save_image_to_file function. To access these points, I used get_point_from_file -- this allowed me to only define and generate my points only once, which saved a lot of time when debugging an generating images!

I defined correspondences as such:

Example of keypoint mapping. This is how the Danes dataset was mapped

Here's a diagram of how I selected the keypoints in order to match this dataset!

Computing the "Mid-way" face

I first reshaped both images to be the same size. Then, I selected 70 keypoints total -- 66 around each face, and 4 for the corners of the image. At this point, I was ready to get started!

In order to calculate the midway face, I first calculated the average between the keypoints (image1_keypoints + image2_keypoints / 2) -- this is the "average" facial shape between the two images.This is important to use because morphing from one image to another, which may have a significantly different shape, would lead to some unrealistic warping. By warping both face shapes to the average shape, each image is 2x less warped while also properly aligning all keypoints and facial features. This alignment is crucial to being able to realistically blend images and find this desired "midway" image.

So, my next step was to warp these images to the average shape. After finding the desired shape using averages, I calculated the Delaunay triangulation of the average keypoints. By finding the triangulation, this allows us to warp the image by triangle (so by section of the image, defined by the keypoints), which allows us to faithfully warp the shape of the faces! I applied this triangulation schema to the keypoints of both images to generate the visuals below. Then, I calculated the affine transformation matrix between the average keypoints and each image in the set (me and Cole), then applying these transformations to every triangle in the image.

The final step is to find the final pixel values of the morphed image. In order to do this, I conducted inverse interpolation / warping using the polygon function -- this got the desired pixel values from the starting image in order to bring the correct pixels / averaged pixelsto the right location on the midway image. We inverse map in order to reduce computational power and to make sure all gaps are properly filled in and in the event that the pixel locations we are reading from are in between pixels!

After reshaping the two images, then I can average the two images to be the midway image, with a warp and cross dissolve fraction of 1/2.

Input image of me	Input image of Cole Sprouse
Delaunay triangulation on facial keypoints	Delaunay triangulation on facial keypoints
Midway image between me and Cole Sprouse

The Morph Sequence

The morph sequence is quite similar to the midpoint calcuation, but the cross dissolve and warping happens gradually.

The first step is to warp your image to the desired image. We have a "warp" parameter which dictates what percentage of the first image's shape will contibute to the shape at that stage, and how much the second image will contribute (for the midway image, this was simply a 50:50 average).

Then, I cross dissolved the two images, which is affecting the pixel colors / appearance. There is another dissolving parameter that dictates how much color each image contributes to the final image (the weight of each image in the morph, essentially).

I started both the warp and dissolve fractions off at 0, so it starts with a normal image of me. However, I then iterated through the warp and dissolve functions at increments of 0.1 from [0, 1], applying the morph function at every point, which finally resulted in a full morph into Cole. This resulted in the morph sequence shown below!

Morph between me and Cole Sprouse

The "Mean Face" of a Population

In order to find the mean face of the population, I found the average of all of the keypoints, morphed the shape of every face in the Danes dataset to that shape, and the averaged all of those images. I then used these average keypoints in conjunction with my own keypoints to warp my face to the Danes average, and warp the Danes average to my face.

Dane 1, morphed to average
Dane 2, morphed to average
Dane 3, morphed to average
Computed average face from Danes dataset	Average Danes looking left	Average Danes looking right	Average Danes smiling
	Input image of me	Delaunay triangles + keypoints
	My face warped to the shape of the average Danish person	The average Danish person's face warped into the shape of my face

Caricatures: Extrapolating from the Mean

A caricature is essentially just taking the features someone has that are not "average" (e.g. a large nose, small mouth, etc) and accentuating them. You can generate a facial caricature by extrapolating from the mean face. In order to do so, I applied the equation: mean_image_shape + alpha(input_image_shape - mean_image_shape) In order for this to be an extrapolation, alpha must be less than 0 or greater than 1.

While I wrote a separate function to calculate this caricature, I noticed that it's possible to calculate this by also pretty much just doing the exact same operation as I did when calcualting the mean face of the populating and morphing that, but switching up a few of the parameters and the alpha value.

By making the alpha positive, I am effectually accentuating the features in the input image that deviate from the average, which creates a caricature! I both subtracted 50% of the "average" features from the computed average image, but also accentuated my own features by 1.5x, leading to an image with large eyes, somewhat uneven smile, thicker eyebrows, etc. By making the alpha netagive, I am accentuating the "average" Danish features in the image, which is a wider face & nose, thinner eyebrows, and a smaller distance between the eyes and eyebrows, to name a few features.

alpha = 2

alpha = 1.5

alpha = -0.5

alpha = -1

Bells & Whistles

Change my ethnicity

In order to do this, I utilize the morphing functionality implemented earlier in the project. I simply warped my face into the average white female face. This can be edited along two axes: the shape warp and the appearance warp. For the shape warp, the dissolve_frac = 1 as the warp_frac increases (in this case i increased it in increments of 0.1). For the appearance warp, the warp_frac = 1 as the dissolve_frac increases (in this case i increased it in increments of 0.1). For the full warp, both the warp and dissolve_frac increase at the same time / increment (in this case I increased it in increments of 0.1).