I really don't think it's because we associate his voice with it. Today it's been so many incarnations of Sabbath, so, at least the new crowd of Sabbath fans (80's and forward), has probably heard the Dio line-up as much as the Ozzy line-up.
Even I, who grew up in the Heaven & Hell and Born Again-era, prefered the Dio line-up at the time.
But no matter how you see it, IMHO, it's the first line-up which probably has the most personal and uniqe sound, much because of Ozzy's voice I think, and that has a great deal to do why they got so popular in the first place. Today, I think Sabbath was best with Ozzy. It's all a matter of taste.