Short answer: Not really.
Since Xiao is not a character who depends on elemental reactions for his damage, elemental mastery is almost useless for him.
Xiao is of the anemo element, meaning the one and only reaction he can trigger is Swirl, which spreads and increases the damage of pyro, hydro, cryo, or electro. Swirl isn’t considered a very powerful reaction, because about half the time a Swirlable element is on the field, it belongs to an enemy that has high or total resistance to that element (ex: slimes, who are completely immune to the element they embody).
In addition, most of the time that Xiao is able to swirl, he’s in his elemental burst. Xiao’s burst is when he does most of his damage, and it ends when he gets swapped off the field. This, combined with how you’re encouraged to plunge attack during his burst, means you won’t have constant application of a Swirlable element. Plunge attacks do not trigger Xingqiu’s rain swords, and all Xiao’s damage is converted into Anemo damage that cannot be overridden by another element, so he will be unaffected by Chongyun’s skill and C6 Bennett’s burst. Plunge attacks also mean you’ll be jumping up a lot, so abilities like Xiangling’s pyronado will be awkwardly positioned at best and visual distraction at worst.
Because Xiao relies most completely on his burst for damage, meaning he’s the one triggering the elemental reaction, sacrificing attack or Crit stuff for a little more swirl damage really isn’t worth it. Focus on substats Crit Rate and Crit Damage, then Attack %, then Energy Recharge for his burst. Elemental Mastery is okay, better than HP or Defense, but building a ton of EM on Xiao is not his best. Give it to Sucrose, instead.
For Xiao, since anemo damage is his main form of damage, smack 2pc Viridescent and 2pc Gladiator’s or something on him. 4pc Viridescent isn’t his best in slot because he doesn’t rely on Swirl for his damage output.