In a typical standoff, such as a hostage situation, two characters face off without immediately shooting at each other. One or both of them may be using a human shield, or be reluctant to shoot first for fear that the other will take them with him. A Mexican standoff, however, takes that up to a whole new level.
Perhaps the best explanation is the one from Wikipedia:
A Mexican standoff is most precisely a confrontation among three opponents. The tactics for such a confrontation are substantially different than for a duel, where the first to shoot has the advantage. In a confrontation among three mutually hostile participants, the first to shoot is at a tactical disadvantage. If opponent A shoots opponent B, then while so occupied, opponent C can shoot A, thus winning the conflict. Since it is the second opponent to shoot that has the advantage, no one wants to go first.
Basically, it’s Mutually Assured Destruction (MAD) on the level of individuals rather than nations. The classic example is the finale from The Good, the Bad, and the Ugly. Clint Eastwood’s character rigged it, though, so the outcome was never really in dispute. Elsewhere, things rarely end so cleanly.
If the resulting fight has a little more thought and strategy to it, though, you’ll get a Mêlée à Trois. This happens in The Hobbit with the Battle of Five Armies, and in Downbelow Station with the stationers, merchanters, and Captain Mallory. It also tends to happen a lot in A Song of Ice and Fire, with the various factions in that world. Heck, it happened in real life during the Lebanese civil war, and is probably happening right now between the Christians, Sunnis, Shia, and Alawites of Syria.
When the fight has room for a little more scheming, you’ll often see things like a gambit pileup or a kingmaker scenario. This is where things get really tricky, and the intrigue becomes positively delicious. It isn’t enough just to have two parties duking it out–you’ve got to have lots of characters and factions, each with their own agenda. That way, even the weakest party can win if they can convince everyone else to fight each other.
All of that happens after the Mexican standoff, though. The standoff itself is the moment of tension and uncertainty before the crap hits the fan. When done well, it’s an awesome moment of tension that can really boost the suspense. When done poorly (or just for laughs), it’s over-the-top crazy, like Duke Nukem meets Scooby Doo.
I haven’t played around with this trope too much yet. Stars of Blood and Glory has a little more intrigue than some of my other works, but I can’t think of a specific Mexican standoff moment in the book. The situation near the end of Desert Stars is kind of like this, but with relationships and marriage alliances rather than guns. I’m sure that’s a different trope, but I’m not sure which one.
Yet another reason to keep trawling tvtropes…