He was often looked down on for being different.
The scholar looked down on the farmer.
The rich man looked down on the beggar.

A: Why do those rich kids look down on Sarah?
B: Because Sarah's family is poor and she wears old clothes.