My brother and I were always incredibly close. I'm 2.5 years older and I never had any thoughts about it, to be honest. I was always very much watching out for him and making sure no one said anything or did anything to him. Luckily only had one unpleasant episode. Frankly, I felt as far back as I can remember, that if someone made anything difficult, fuck them.
I think the thing for you and his siblings is to accept him for what he is and what he likes, not what you would hope him to be. Many times I've argued with my mom that she forces him to do activities that she wants him to do, as opposed to his interests (going to prom, for example, wasn't something he knew existed until my mom decided he was going.) He is incredibly happy being around family, going out to eat, having ice cream and watching about 10 disney movies that he loves.
It also depends on behavioral and health issues. My brother has been very lucky, without any major health issue. Communication is challenging, and has been forever. He didn't speak until he was 5, and learned sign language prior to that. Some of the other kids I've known in his cohort would have been very challenging for parents and siblings.