There some situations in which eye contact is considered inappropriate.
In neurotypical body language, initiating eye contact with someone means that you want to interact with them. It’s often the first stage of a conversation, or of flirtation.
This can lead to awkward situations for those of us who don’t make eye contact naturally and don’t have it in mind much.
For instance, on the subway it is not considered appropriate to make eye contact with strangers. On the subway, people are supposed to leave each other alone.
People who make eye contact naturally kind of know where not to look, and don’t have to think about it much. For people who don’t use eye contact as a natural part of their communication, avoiding inappropriate eye contact can actually be difficult, since they don’t automatically pay attention to where not to look.
If you look in the direction that would be for eye contact if you did that sort of thing, people will interpret it as an attempt to initiate eye contact with them. And they will often look back and smile weakly, because it is considered rude to ignore eye contact. But since they don’t want to talk to you, and it’s in a situation in which people expect not to talk to each other, it’s invasive.
For that reason, if you have this problem, it might help to intentionally figure out some other place to look in order to avoid inappropriate eye contact. (Eg, your bag, your phone, the floor, the ads).