The 31-year-old radio presenter replaced X factor veteran Louis Walsh, and fans of the show have been in uproar after losing favourite faces including the long-standing judge, Mel B and presenter Dermot O'Leary .
Nick is apparently keen to return to the show in 2016, but a source speaking to The Sun said: “It feels like the writing has been on the wall for him. He has got better, but it's too little too late.”
X Factor 2015: Rita, Nick, Cheryl, Simon, Caroline and Olly. Will the line-up look the same in 2016?
Rita Ora, who was also added to the judging panel this year, has proven a hit with viewers. But she has a new album planned for 2016 and has said: "I don't know if there'll be time [for The X Factor], my main priority is my album."
The X Factor Final starts tonight on ITV, 8.00pm and concludes Sunday night, when the winner will be announced.
And Nick could have the winning act in Che Chesterman, who is up against Rita Ora’s act Louisa Johnson and Cheryl Fernandez-Versini’s Reggie N Bollie.