Add simple image edit/mask component

[iwr-redmond]

Much ML work involves image generation and refining, which means that a component capable of simple edits (e.g. crop) and especially adding an inpainting mask could be very helpful.

Some libraries good enough for government work:

• Mini Canvas Editor: lightweight and can be customized for specific use cases
• TUI Image Editor:: customizable interface with all the various features
• Pikaso: can have a background image to paint on a mask, and separately has a crop function

[iwr-redmond]

Looking at the Mini Canvas Editor configuration file, I think this would be best.

Demos of the features I suggested might be a priority for ML apps:

• Inpainting: config file
• Cropping: config file

[iwr-redmond]

Inpainting

Each line drawn is a separate layer on the canvas. The layer-item.ts code has at L100:

private readonly onDeleteClicked = () => {
		this.state.canvas.remove(this.object);
	};

Suggesting that all of the markers could be deleted with a forEach loop similar to template-creator.ts L88. I suppose this could also allow the first click to be recognized and used to activate Segment Anything in the future.

The configuration file has code at L28:

export interface BrushModeConfiguration {
	brushSize?: number;
	brushColor?: string;
}

Allowing easy theme integration.

Cropping

The demo resizes the image on the canvas rather than cropping. However, "crop" might be better expressed in the Template Creator example, with the white box representing the fixed canvas and the green $cat box representing the crop mask.

Per the backend code template-creator.ts, this would look something like:

const editor = Editor.createFromImage(
			placeholder,
			image,
			{
				selectable: false
			},
editor.add(
	new MceRect({
		fill: 'transparent_png_file',
		left: 0,
		top: 0,
		width: 100%,
		height: 100%,
		label: 'Mask'
	})
);

Bearing in mind that my code is good enough for government work.

[mad-moo]

I am confliced. As you've said, components like this are currently very in demand. The thing is, they're highly specialized for this exact subfield of AI, and given how rapidly AI is moving, the component probably won't be used for long, or would at least have to undergo massive changes.

This would be a perfect candidate for a user defined HTML component. Something that doesn't ship with Rio, but has access to the HTML DOM & JavaScript. Except, we don't have those yet, at least not publicly. If you need this urgently I can help you create your own HTML component using the internal, not-yet-stable API. If not, I think this should wait for that API to come online, at which point it could live in a separate python package. What do you think?

[iwr-redmond]

I think the right path depends on the complexity of integration. If the complexity level is low, then a custom component would make sense; if high, forget it. However, I suspect the complexity level is moderate, which makes me think that an image editor function makes most sense in core.

In terms of long-term need, beyond current popularity, revenue from AI image generation is predicted to grow by 17.7% (compound) each year to 2030.

[iwr-redmond]

I suggest adding Jimp to the list of possible editor components. It's lower level than Mini Canvas Editor, which means it would probably be more work to implement initially but might provide more long term flexibility. Customization options are documented here.

A few more data points supporting this feature request:

According to a recent paper by Zhang and Tang, ML image research is growing rapidly - see Table 3 (p9). (Table 4 paints a similar picture, so to speak, but is not sorted by year.)

Black Forest Labs have recently released a model dedicated to inpainting and outpainting, giving it pride of place in the press release.

Inpainting research goes back over two decades and is so common that there have been at least 45 reviews summarizing earlier papers, according to the October 2024 special issue of the Computers & Graphics journal (Barcelos, Rabelo, Bernadini, et al, 2024).