The man, who would only give his name as Barry, said the disgraced DJ showed no alarm and carried on having sex with the girl while smoking a trademark cigar.
He has now told police about the attack, which happened in 1961 at the Mecca Locarno ballroom in Leeds, which Savile ran before becoming famous at the BBC.
Speaking to the Sunday Express, Barry said: “I knocked on the door to what I thought was my band’s room and went in but it was actually Savile’s dressing room. He was having sex with a very young girl, she was no older than 12 or 13.
“The only thing I could blurt out was, ‘I’m very sorry’.
“He didn’t even flinch. He laughed it off and said, ‘That’s all right kid, go down the corridor and turn right’.
The only thing I could blurt out was, ‘I’m very sorry’
“What I remember most, though, is the girl’s face.
“She was a pretty, innocent- looking blonde who turned and looked straight at me with an expression of sheer terror and with these sad eyes.”
Barry, then 20, eventually found his bandmates and told them what he’d seen. “They were completely unfazed and said, ‘Oh, that’s just Jimmy’. He was well known for that behaviour, even then.
“As that poor girl was being raped there were four or five others queuing up outside for autographs.”
He said: “At the time I’d just become a musician and did not want to lose my place in the band. I’d told the other senior members what had happened and they didn’t seem too bothered so I was confused as to what to do.
“Savile was a big name in Leeds. I saw him a few times afterwards but never spoke to him again.
“As for the girl, I never saw her again, but her face has tormented me ever since and I’ve now reported the attack to West Yorkshire Police.”