I think the root-problem is voice-chat cannot be "windowed" by the human brain in the same way text conversations can be, it's the single biggest achilles heel of voice-comms & there is no real way of getting round it (unless human hearing makes an evolutionary leap in that direction).

However, on alternatives to "shut up" anything will work, it all depends on it being a recognisable word or phrase that everyone on the raid is expected to respect, "Squelch On!" "Chats Off" or some other phrase not common in conversation work best to catch attention, ideally coupled with a Text command it just comes down to setting something up that everyone is expected to be quiet after hearing.