If I am riding a horse, the fact that I wear glasses is irrelevant. If I am talking to a friend about whether he should or should not get bifocals, the fact that I wear glasses is very relevant. If I am meeting a friend at a restaurant, the fact that she uses a wheelchair is irrelevant (or at least should be, and generally speaking in the US it is irrelevant). However, if I am asking her to discuss how her disability has impacted her desire to be employed, that she uses a wheelchair is very relevant.
The key is to keep the relevant relevant and the irrelevant irrelevant.
If I have an intellectual disability and as a result I am segregated, that is making what should be irrelevant relevant.
If I use a wheelchair and because of that I cannot find a job, that is making what should be irrelevant relevant.
It is critical for people to look on their neighbors wisely. If something is considered relevant for some reason, it should be logical and defensible. So much of the discrimination people face is the basis of making irrelevant things, ethnicity, gender, disability relevant to judgements about people when it should not be so.
Yes there are aspects of human impairment that are best addressed by a medical model approach. There are other aspects that are best addressed by a social model approach. It is not one or the other. As Tom Shakespeare has related, he cannot blame society and discrimination by society for a bladder infection resultimg from his impairment. He can blame society for making personal characteristics relevant as a cause for discrimination.
The key is understanding the difference and only making personal characteristics relevant when they should be relevant.