The card game we play at home
Zheng Shang You is a card game we play in my family. Four players try to get rid of their hands, beating the last combination or passing, with bombs that outrank the ordinary combinations.
I wanted to see what a learning agent would make of it. There's a playable version here too, if you'd like to get a feel for the game.
Self-play wasn't getting me very far
An early Monte Carlo self-play setup kept returning scores around 33%, without giving me a useful way to tell whether the networks were improving. The training log records that dead end. I moved to cloning a scripted player, then using it as a starting point for further training.
The later setup combines imitation learning and PPO. The critic can see all the hands during training, while the policy only gets the player's information. There's also a small oracle that estimates how many moves a hand needs to shed.
What helped in the recorded runs
These are four-player evaluations against three copies of the scripted strategist. A quarter of wins is the equal-seat reference point.
One result I found interesting was changing how the search scored a rollout. Looking only at a win or loss made second and fourth place equally bad. Giving them different scores, along with more sampled hands, helped substantially in the logged comparison. It wasn't just a matter of searching deeper.
Training against earlier versions
The league runs below include the attempts that levelled off.
Each generation plays a mixture of the previous champion, older snapshots and itself. Fixed opponents and pure self-play each ran into problems. The third generation's logged proxy results were flat, and I hadn't settled whether that was the model or the opponents.
There were some less glamorous problems too. Crossing between Python and Rust only paid off once whole workloads were batched. And wildcard hands can have so many legal moves that the cloning pipeline ran out of memory. Sampling alternative moves bounded that cost.
Where the next improvement might come from
Open question
I'd like to separate the value of a larger policy from the value of more search. The flat third-generation runs didn't answer that.
Open question II
The logs also suggest passive endgames are a problem. It would be interesting to see whether training can learn when to spend a bomb without me encoding that advice into the reward.