I do understand what you mean.
I personally think that his having called Charlotte my lady has a special meaning judging from a lot of evidences though what you say might be a reality.
She was never a groupie, and never a part time lover for Jimmy. She was his real love, and is the mother of his daughter and the grandmother of his grandchild.
She should not be compared to another women(especially from that time).
Anyway Jimmy calls all his children and all the mums 'a close-knit family'. Recently I found the meaning of this word clearly at last! They are really going well.
From ''When Giants Walked the Earth'' by Mick Wall(P.173)
Jimmy's affair with Charlotte would remain in place a lot longer than his dalliance with the prettiest GTO. It was,in fact, the start of the first really significant love affair of the guitarist's life, and one he would remain true to even after the band had returned to the road in America. An almost unheard of sacrifice, even for the married men in the band, for Jimmy it was a sign of something much deeper.''