I understand that he identified you by your race, but how is that actually racist? If he used some form of derogatory term or a racist connotation then maybe, but if he just calls you your race, then I don't really get the racism. (Not that calling people by their races is totally cool, but it certainly impacts the gravity of his words)
For example (racist example warning)
Cool: "dang you, you arab"
Uncool:
Really depends on what he said. Either way I'd appeal to get back in for the 2 games. Oh, and leave the timekeepers alone, its the ref's job to deal with you.