Training a coding model to paint watercolours with TRL and OpenEnv
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
An official release brings Qwen model updates — worth tracking for capability changes, ecosystem impact, and follow-up.
Surya Narreddi demonstrated a language model painting watercolors using JavaScript and the p5.brush library, which extends p5.js with natural drawing tools. The viral video, posted on August 23, garnered over 1.5 million views. The model was trained with TRL and OpenEnv, utilizing a Qwen/Qwen3.5-35B-A3B model with LoRA, bf16, and gradient-checkpointing. The training involved specific parameters like --subject 'a peach hibiscus', --steps 110, and --n-episodes 240, ensuring the model compiles code, uses the library correctly, and applies pigment without attempting to trick the scorer.
On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush , a library that "adds natural drawing tools to p5.js". The video went viral fast, over 1.5M views at the time of writing.
The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine . My attempt is on the engineering side, reproducing the recipe in the open with every piece published.
Note: for the context behind the project, told by Surya himself, watch this video of his thesis .
In this article I try and reproduce his idea with TRL and OpenEnv . The reference pool dataset, the RL environment, the training scripts and the trained models, all open.
The whole pipeline runs on Hugging Face, end to end:
- training on Jobs
- the RL environment and the scorer model as Spaces
- the pairwise judge through Inference Providers
- and every artifact on the Hub, gathered in one collection
Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model , set two environment variables for the reward mix, and launch:
hf jobs uv run train/watercolour_grpo.py --flavor h200 -- timeout 48h --secrets HF_TOKEN -- \ --env-url https:// -watercolour-env.hf.space \ --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \ --subject 'a peach hibiscus' --references 4 \ --top-p 0.95 --top-k 20 \ --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \ --scale-rewards none \ --steps 110 --n-episodes 240 --num-generations 8 \ --per-device-batch-size 1 --gradient-accumulation-steps 8 \ --max-completion-length 8192 \ --run-tag my-run --out /watercolour-grpo --push-to-hub
The rest of this article is the story of getting there, and every piece is in the repo .
I have followed the original blog step by step, and only changed something when strictly needed. Every idea of my own went into a list instead of into the experiment, and that list became "What I would try next" at the end, next to the full list of published artifacts. If you have already read his post, the framing and the reward design will be familiar. The new material is the open implementation, the hand-rated pool, and three reward mixes trained and compared, and it starts at The RL environment you need to build .
Three runs, one per reward mix, evolving in parallel. Each frame shows the median painting of a step. No need to tell them apart yet, the article explains which run is which.
Why people loved it
The paintings look loose, imperfect, handmade, at a moment when image models produce perfect (statistically average) pictures. My guess is that this contrast is a big part of why the video went viral. It reminded me of the early days of generative AI art, when the point was to explore the medium. DeepDream (2015) was a debugging tool that people turned into art, works like Edmond de Belamy (2018) came from artists probing what a GAN could do, and artists like Mario Klingemann spent those years making dreamy portraits with neural networks .
This project feels closer to those early days. In his thesis, Surya describes the path that led here. He started by prompting text-to-image models, where the prompt is the only lever you can pull, and more detail buys more control only up to a point. Training the model itself goes further. The other half of the idea is the medium. The model writes a program of about 150 lines of JavaScript that paints the image. That model output is code. You can read it, edit it and run it again, and the decision behind each brushstroke is visible. And the style comes from a restriction where the model is only allowed ten of the library's methods . More on that below.
In that same period, Anna Ridler photographed thousands of tulips, hand-labelled every one, exhibited the dataset itself as the artwork, and later trained a model on it. I found her work through the references AI agents brought back while building this project and loved it because this project does something very similar by curating a set of images by hand, and then training against them.
RL over taste
Most of the recent RL work on language models uses rewards you can verify. For example, math problems with a known answer, code that passes tests, or graders that are right or wrong and cheap to run. This project is closer to the older exception, RLHF, where the model learns a reward model from human preferences.
Here the reward is aesthetic preference. There is no correct answer. The real question of the project is whether you can do RL over taste.
The reward, as his blog defines it and as the RL environment I built implements it:
term weight what it measures
gate 0.05 the sketch compiles, paints something, does not cheat