Newsletter #15

Published 2021-11-01

The coolest use of SnapML

SnapML lets anyone enhance their Snapchat lenses with machine learning. For most people this means things like style transfer or object detection, but GoSpooky managed to get 3D object tracking working. Check out their LEGO car demo. This is hands down the coolest use of SnapML I've seen.

Now this isn't something I could replicate tomorrow, but conceptually I know how you could do this yourself (for my 100% of readers that are smarter than me).

  • Create a 3D model of the object you want to track (complete with textures)
  • Render out a ton of different views of your model using different angles, positions, and lighting conditions. Use a transparent background
  • Overlay your renders over lots of different images to create your dataset
  • Train your machine learning model to recognize the object in the scene along with position/rotation information
  • Be the coolest lens developers on the planet

The trickest part would be to figure out which model architecture provides good predictions and which can run well in Snapchat. But creating the dataset is fairly easy provided you have a 3D model of the object you want to track. And who knows, maybe GoSpooky will share their training process or Snap will create and share their own? Regardless, this is a top-notch example of augmented reality and I hope we'll be seeing a lot more of this sort of thing in the future.

Spark AR update announcement

I guess an actual meaningful update to Spark AR is too much to ask for, but Facebook (or Meta) announced that they'll be adding 2D body tracking "soon." This isn't full 3D tracking, but 2D tracking is a lot better than nothing.

Spark AR certification

Facebook also announced a Spark AR certification program where you can complete a curriculum and then pay to take a certification exam. I could make a joke about how Spark doesn't have enough features yet to where it's worth getting certified to balance things out, but I won't do that. It sounds interesting, but most of the lens requests I get have been after people watch one of my tutorials or see one of the lenses I've made. They don't care at all that I'm an Official Lens Creator (or Snap Lens Network Member or whatever the heck it's called now) and I'm pretty sure most of them don't even know that's a thing. I'm not sure I'm going to try getting Spark AR certified, but it might be something you are interested in.