TensorFlow.js Monthly #3: Case studies, talks, and demos.

Welcome back Tensors to the 3rd ever TensorFlow.js monthly newsletter - a newsletter to keep you up to date with the latest and greatest in the world of "Web ML" (I've used the phrase in 2 newsletters now so yep, it's a thing), and my gosh we have a fair few updates to get through. So consider sharing this wonderfully written masterpiece with a friend so they can learn too, or if you are new hit that subscribe button to be sure to get the next edition, and with that, let's get started!

Case study: How LinkedIn use TensorFlow.js in Node to personalize performance for millions of users.

Yep, here you are reading a newsletter from LinkedIn that may very well have been optimized using the technology I am about to talk about - very meta (no not Facebook)! We just completed our latest blog post case study that has all the juicy details, but if you want the TLDR then I've got your back below.

Essentially some wonderful folk at LinkedIn (Nitin Pasumarthy, and Mark Pascual) explain how they have leveraged TensorFlow.js within the Node.js server side ecosystem along with their very bespoke custom backend stack over at LinkedIn.

Using TensorFlow.js they created a custom neural network that predicts if a request for any given URL would result in a fast page load. How? Well, based on the network conditions and device's characteristics they are then able to, in real-time, change the page's image quality before the resulting page is rendered client side in the browser.

What this means is that the images you see are the absolute best quality for each device that is requesting them, instead of just compressing all the images and sending low quality images to everyone who is on mobile. This real-time ML prediction ability resulted in billions of extra feed viral actions, and millions more engaged feed users, not to mention sponsored revenue also increased significantly for LinkedIn too due to the higher engagement. Great results.

So yes, there you have it, TensorFlow.js can be used for any scale and type of project if you put your mind to it as LinkedIn here have proven well. Furthermore the Node.js solution ran around 10% faster than the Python equivalent, which when you are dealing with millions or billions of requests is very significant - yet another reason to stick to JavaScript for production with that sweet JIT compiler. Huzzah! Anyhow, enough from me, go read the blog post yourself to learn more about why this matters.

Tech Talks: AI on the Web with TensorFlow.js with Laurence Moroney and Gant Laborde

This month was also a great month for talks around TensorFlow.js. Incase you missed it, two legends of machine learning, Laurence Moroney (author of too many books to list here and also my manager - so do let him know TensorFlow.js rocks if you get the chance) and Gant Laborde (GDE for TensorFlow.js and author of the TensorFlow.js O'Reilly book) put on a jolly good show at the Google Developers North America event "Behind the Brackets" that aired on the 24th March. Fear not, a recording is available to catch up if you missed it, so do check that out. I mean, look how happy they are, go check it out immediately on your next coffee break.

Speaking of talks - New ones are coming!

Google IO 2022

Do register for this year's Google IO 2022! For those of you who may have been living under a rock, Google IO is Google's biggest developer event of the year where we announce all our shiny new things and give opportunity for developers to connect with Googler's from all around the world. Needless to say TensorFlow.js has some updates at this year's IO so do save the date in May (next month) and look out for our TensorFlow.js team update by yours truly. See you there!

DevNexus: AI in the Web - TensorFlow.js

Which reminds me, Gant will be giving a new talk titled "AI in the Web - TensorFlow.js" at DevNexus live in person if anyone is near Atlanta, USA - I even sent him some TensorFlow.js sticker swag for folk who do make it in person. If you can't make it in person, the videos should be on the website shortly after too. Essentially this talk dives deeper into the the fact that AI is now everywhere, even on devices and websites, and how you do not need a Ph.D. in machine learning to take advantage of it. Great for folk new to the industry looking to get their hands on coding some cool demos in the browser.

It's demo time!

If you made it this far then I shall reward you with some awesome new TensorFlow.js demos - congratulations.

Hand mesh + Three.js = Photorealistic Wizardry

First up, on my travels around the interwebs, I bumped into an amazing Three.js expert, Daniel Esteban. I saw his initial creation and thought hey, this would be pretty neat if you combined it with our hand pose model. So I tweeted his way, and within just 2 days he posted this piece of wizzardry that integrated the TensorFlow.js hand pose model into the demo so you could literally interact and play with the photo realistic metallic liquid physics experiment he had made:

Go try it for yourself, because you know, it's powered by TensorFlow.js and runs in the browser so zero install required. Have some fun. You earnt it for getting this far down the newsletter ;-)

Now on this note, I would like to challenge all of you to consider helping people out as to how they could level up their work next time you see someone's social post for their web demos. Maybe you know of a TensorFlow.js model that could give them superpowers? Sometimes all it takes is a comment showing folk what exists and great things can happen! Share that knowledge you have after being a subscriber to this newsletter and let others play too :-) Together we can change the world and have smarter web apps of the future right now.

Hallway facial motion capture in the browser

Next up is a rather impressive demo by Matt Rossman using Hallway's web library. After reaching out to Hallway, it appears they are using a custom TFLite model to predict blend shapes from the face live in the browser thanks to TensorFlow.js support for digesting TFLite models in JavaScript. Reach out to Bryan Pratte or Jacob Muchow from Hallway if you want more details on how they use the TensorFlow.js library in their product.

Essentially though, this means they can take models that were designed for mobile and run them efficiently on the web too. In this demo you can see how the web (the top right head) compare's to iOS native results (bottom right). Given the speed of innovation in this space, and that the web is only just getting started, this is not half bad, and as more researcher's port their models to be web friendly, things are only going to get better in this regard.

Seen something cool? Send me your finest TensorFlow.js finds

Made something cool or seen a demo out in the wild? Be sure to tag it with #MadeWithTFJS on LinkedIn / Twitter / social so our team can find it for a chance to be in our newsletter, future events, or even our YouTube show and tell!

That's all folks for this round, see you next month for even more adventures! I shall leave you with this appropriate ML meme that sums up this month's newsletter:

Until next time. Jason.