The group Rhythmix were put together after entering the show as four soloists and are in the final 16 acts.
Brighton-based charity Rhythmix, which has been operating in Kent, Surrey and Sussex, fears it will cause confusion.
X Factor said there was no conflict because the charity used the name for educational purposes only.
Rhythmix charity, which receives National Lottery funding, has 50 musicians who have worked with 40,000 young people through youth support services.
The charity just want to make it clear that there is NO connection between them and the TV programme.
"We contacted them (X Factor) on Sunday to say 'do you realise that this is the name of a music education charity working with vulnerable young people?'" said consultant Lucy Stone.
"We are just asking them to stop and think about what they are doing and what impact they are going to have on a charity at such an already really challenging financial time. "Their response was, 'Get a lawyer'."
Media lawyer Paul Gilbert said X Factor was being "a bit disingenuous".
"I think they could at least make some statement to make it clear the distinction between the two and to support the charity in the good work they are doing," he said.
An X Factor spokesman said, "The charity in question have trademarked the name Rhythmix for educational purposes only, which is not in conflict with the band or the show"
Now we can understand the charities frustration but if the X Factor Follows the rules and regulations then what can be done? Not alot.
IF anything, the charity are going to benefit from all the exposure they are receiving regarding the dispute.
