Scribble Diffusion - convert scribbles and sketches into AI images

What more do you need when artificial intelligence can turn your crude scribbles into an exquisite image and make it look like a masterpiece? An engineer at Replicate under the nickname zeke on GitHub has created just such a feature.

What is Scribble Diffusion?

Scribble Diffusion is an open-source application that can turn your hand-drawn drawing along with hint text into an artwork. The Scribble Diffusion model is built on ControlNet, a neural network framework developed by Lvmin Zhang and Manesh Agrawala to control large diffusion models with additional input conditions, which in this case are human doodles. The model drives stable diffusion with other inputs such as edge detection, depth maps and pose estimation.

ControlNet was released only three weeks ago, and developers are already creating applications based on it. Another developer has created an application that generates DreamBooth images based on the pose of the input figure.

How is scribal diffusion related to stable diffusion?

Since the open source release of Stable Diffusion, ML developers and enthusiasts have been experimenting with it to create different types of models.

First, it doesn't take long before a new technology comes along and users start trying to make NSFW content with it. This is exactly what happened with Unstable Diffusion. After the release of this model, Stability AI came under scrutiny.

Other models include Riffusion to create real-time music using artificial intelligence by converting audio images into sound. The approach involves fine-tuning Stable Diffusion to create spectrograms that are visualizations of audio. Another model, Amazing AI, was built on top of Stable Diffusion 1.5, which runs on macOS without the use of the internet.