Tinkering with SPADE-FACE

A sprained ankle on a rainy day means I’m actually making good on yesterday’s plan to try SPADE-FACE on RunwayML.

Here’s what that’s looking like:

A sketch of a face and its computer-generated result A sketch of a face and its computer-generated result A sketch of a face and its computer-generated result
Left: the 'segmentation' sketch input. Right: the computer-generated output.

There were a couple versions that didn’t work out so well. They were ones I sketched elsewhere and imported in via a PNG or JPG. Here’s what I mean:

A sketch of a face and its computer-generated result
Randall Flagg on a good day.

You can see the difference in black; the prior versions are pure #000000. The imported, demonic, one has gone through some unwanted colour profile changes. Same deal with all other colours. Similar results with PNG where there should be less colour differences.

A screenshot of all the colours that SPADE-FACE requires for facial features
Each of these hex codes must be matched exactly for the model to do its magic.

I had an idea to outsource the sketch creation to a whole bunch of people, ideally hand-rendered in crayon, marker, or acrylic. How would the output from a five year old kid in the U.S. differ from that of an adult in Sydney?

This colour matching snafu puts a damper on these plans as it means the segmentation sketch must be done in RunwayML or have been exported with the exact right colours and colour profile. That stuff is a black hole.

Anyway. There’s gotta be a way. I’ll figure something out.

Here’s a speed-run of live-editing a segmentation:

Drawing whilst the model is running. I'm doing this via the Sidecar feature that lets you use your iPad as a second Mac display, which means I can also use the Apple Pencil.

And here’s me learning how to chain models on RunwayML:

A screenshot of the RunwayML interface
My movements in in the camera are applied to the image and outputted below.