There is no one “rarest” character in the game. But there more rare and less rare characters.
4-star characters are common (it is reasonable to expect to eventually have them all), while 5-star characters are rare (you can’t really expect to get any specific one of them). But the most rare category, is 5-star characters unavailable through general wishes, only event wishes.
For example, now there is an event wish “invitation to mundane life”. It should be over later this month. And as long as it continues, you can try to roll Xiao, the Vigliant Yaksha as a playable character.
Event-only 5-star characters include:
Albedo the genius alchemist and a Geo swordsman
Ganyu the half-adepti secretary and a Cryo archer
Klee, an adorable child with too much Pyro explosions
Tartaglia, the Fatui harbinger, in his role as Hydro archer/swordsman (you don’t get to access any other forms he has as a boss opponent)
Venti the bard, the Anemo archer
Xiao the vigilant yaksha, an Anemo spearman
Zhongli the expert, the Geo spearman
Still, in my multiplayer experience, there are plenty of people with these characters. But of all that list, I only have Albedo.
It is likely that in the future, events where you can get one of the event-only 5-star characters will be repeated. But they’ll still remain quite rare.