Many of its games are available with commentary on youtube; notes taken on this pair of PvT matches against terran pro Scan.
At this point game-shaped domains falling to superhuman AI is a familiar experience, so there’s nothing surprising in the abstract about this happening to SCBW, untouched only because of relative neglect (SC2 got AlphaStar in 2019 (!)); still, the jagged particulars of the AI playstyle (what goes superhuman while what remains ab- or subhuman) are interesting.
SCBW skills could be broken into “micromanagement” (efficient unit combat), “macromanagement” (economic growth and production), and “strategy” including both general skill at long term decisionmaking and specific metagame skill of reading opponent’s decisions and making the appropriate counter moves. In this crude division, the first two are close to purely mechanical, basically performing simple-ish action quickly and with precision at regular intervals; some difficult combinatorial decisionmaking can emerge in moments where many factors overlap (like how to split allocation units between two close battles, or when to cut optimal production cycles to spike resources into some other area) but it’s mostly ‘just’ mechanical [one could ask if there isn’t some moravec’s asterisk here of what looks just mechanical or not? I don’t know/TK].
SCBW is a game of mechanical restrictions (human speed, only one computer screen of attention at a time, only one set of mouse and keyboard inputs at a time; in ‘actions per minute’, this budgets out to 300-500apm at the highest level); so an easy way to get better play is just go superhuman mechanically, with thousands of apm for perfect micromanagement (rotating individual units to fire and retreat) or attention management (perfect control on two or more screens simultaneously) or macromanagement (perfectly timed production cycles).
But because it’s also a game of imperfect information, and because certain strategies are very volatile and high variance, one still needs a lot of game understanding to not “still die” to some unforeseen haymaker, even when playing ~perfectly in a generalist way.
Naively, one might expect a superhuman AI bot to do perfect micromanagement and then fall apart at a more abstract, long horizon strategic level (I did), and previous SCBW bots as well as AlphaStar leaned hard on degenerately near-perfect superhuman micro. That’s not how this bot looks at all. Not only is its individual unit micro just ‘okay’ (loses equivalent battles to humans), it also regularly makes what are just blunders that a professional would never make (eg running dragoons into mines it knows are there), and is missing some techniques (like dragoon stutter step mine defusing micro). So it seems to be ‘good enough’ at micro to get by but then being extremely strong at macromanagement, or to have its micro kick in more nonlinearly in the critical late game battles, where human attention is more taxed whereas the early game battles can have one screen of attention and near frame perfect unit management, for a human. Human micro perfection falls off much more significantly with too many units and too many screens, whereas the bot can shoot up to 1500 apm and remain closer to its ‘good but not great’ micro level. (This may be what going on but I’d want to look at the later battles more closely to confirm it was winning ‘more than expected’; a lot of this is very cursory and provisional. It is notable that it often seems to be behind in the early to midgame from some predictable blunders but then catches up or isn’t behind as much as it should be, from that, and then wins some critical late game battle that looks equal-ish before it happens).
A lot of its specific decisions are just very weird and are unmissable ‘engine moves’, in the chess terminology, moves a human would never make, either at the professional level or sometimes any human at all, ever. Examples from this set of 2:
Basically random gateway placement. (Humans get an advantage from visually clumped gateways on one screen allowing easier macro cycles, a bot doesn’t care. This has ~0 practical effect as far as gameplay but is visually very noticeable)
Running dragoons into mines for no benefit/that it knows are there, not even a tactical gamble. Amateur players will do this but pros can generally keep them alive. (As noted before, also has no knowledge of stutter step defusion micro)
Slow walking reavers around with no shuttle (and not harassing with reavers at all, in this game at least). It’s an interesting ‘concept’ to just use them for defensive zoning and saving the minerals and robo build time on shuttles (‘looks fosbury flop shaped’) but very confident this is just strictly worse, and given Snow can kill even high level terrans with just one reaver/shuttle, as in recent ASL match, a bot with superhuman reaver/shuttle micro + all these existing skills would be enormously stronger.
Similarly it just walks HT around, no shuttles. Although the shuttle incorporation is also a somewhat recent meta innovation, that humans didn’t do for a while.
Feeding in DTs one at a time, endlessly; this is a sort of tactic that can work, if it clears minefields or exhausts scans, so it’s not a completely crazy move, but it also didn’t work here and also no pro would really play this way or use gas in this way; and K/D ratios on gas units were extremely disproportionate, from this. So it ‘could be’ a new play style it discovered but seems to just be an un-ironed-out blunder, like the running dragoons into mines problem.
Placing a cannon where the nexus would go, and then making the nexus above the cannon for more inefficient mining. Human pro would probably never do this in their entire career, like the Minsky quote in Society of Mind about people going their entire lives without accidentally walking out a window.
Clumping all three observers over terran army, so all die to one scan. Human would do this from a hotkey mistake but never on purpose.
Making defensive cannons vs. a tank/vulture push at the last second. Not 100% crazy but pretty close.
These were just observed in one game in one matchup, so there are many more latent examples out there.
For any strong difference in human and computer play, we can ask which of three possibilities it is. Is the computer playing better in the human range, and in the human skillset? (These can be productively copied). Is it playing better but superhumanly, in a way humans can’t copy themselves? (eg chess positions that require fragile machine ultra-finesse to play perfectly, and otherwise blow up; or here, using lockdown in PvT vs dragoons with hundreds of extra APM requirements at critical moments; humans have tried this for a long time and just failed, so while it’s not totally impossible it could be revisited, I’d bet against it). Or is it just playing worse and the seeming blunders are real blunders, but it’s more than compensating in other game areas. These kind of all seem like the third category although more research is needed.
Not going to do timestamped commentary but general impressions. GAME 1: Pluto opens with a risky but in-meta economic build. It then loses an initial tactical skirmish that was probably a draw, and also runs dragoons into mines avoidably. After this it loses its early 3rd expansion and all the probes, and seemingly didn’t try to defend it; in a normal game this would be a crushing setback and maybe close to losing, although Scan didn’t push further and objective $ and worker count metrics may show it was more of a reset than a strict deficit, would have to rewatch. At any rate bot is taking punches and not looking superhuman in the opening. Stylized picture is a huge disproportion in K/D, especially gas losses, but protoss still keeps up economically, with workers, etc; same supply, same resources mined, same worker counts. So the macro is pulling it ahead but then strategy and maybe micro is bleeding it off, although this further needs to be adjusted by protoss ‘supposed to’ being ahead and less efficient in this matchup, at this point. The rest of the game proceeds normally enough, with Scan playing somewhat conservatively and armies posturing back and forth but no critical damage inflicted, and then in a final large climactic battle the bot comes out on top and wins, beating a pro terran in a “normal” game despite serious damage in the opening.
GAME 2: Bot goes with 1 base DT rush, a very off meta and pretty weak build when performed optimally, but then it was performed very badly here with no attempt to hide it from the first scout at all AND a failed attempt to block it from the 2nd, confirmatory scout SCV, which was using mining glitch to get through the walling unit (I think from vision from the earlier first scout). Seeing the punch a mile away, Scan blocked it incredibly easily and then, in an echo of the (bad meta build then executed badly), the bot ran in anyway hyperaggressively and took huge losses (although there’s some argument that this was ‘playing to its outs’ as it was maybe losing anyway). Then the T kind of macros normally and steamrolls the bot with a timing push, but also during which the bot made some odd army movement choices like cutting its forces in half and leaving half stuck below an easily blocked ramp near its natural. Overall looked very not good, in this mode, which kind of suggests the jagged shape of the weaknesses; but it did look extremely strong, at its strengths, in game 1.
Some related points:
Cool analysis. Some thoughts:
Did Alphastar ever get good outside PvP? Generalizing across matchups seems really hard, so I am impressed that Pluto pays that handicap. I guess because self-play requires learning all races too.
On Pluto blundering well known micro tech, I would separate micro into two categories. A lot of the stuff that makes BW mentally demanding to play, is stuff that SC2 does for you out of the box even as a human player. So if a BW bot beats a human for the first time, and an SC2 bot beats a human for the first time, you would expect the SC2 bot to have better 'techy' micro and better strategy, because its human opponent isn't bogged down with idle workers and clicking on production buildings one at a time and routing the four fragments of his deathball around each other.
On writing that last point, I suspect game balance compounds this. Players have begged for Protoss deathballs and Zerg A-moves to be nerfed as long as I remember, sheer mass beating spellcasting is anathema... But in BW sheer mass could well be optimal.
I would love to see players learning to beat Pluto's shortcomings, but I would also love to see Pluto develop even more, because it sounds quite impressive considering the 9x greater matchup number vs the Alphastar debut, big shoes to gradient descend in!