Well yes, you do wear underwear under your costume- your tights! Just like you do not wear underwear with your swimsuit, the same is for dance costumes and leotards. Dance tights are the appropriate under garment for class and performance, and the younger that dancers get used to the concept, the easier it is, and no one will be noticing the unsightly bulk of underwear showing at dance and on stage.
All dancers will need tights for the recital! Make sure you have the right color to go with your costume by checking with your instructor or at the office.