Did I do the wrong thing?

Recently I had the opportunity to be the president of a club at my university. The teacher in charge really wanted me to be the president due to my abilities and personality, and said so many nice things about me. But two of the other three members were so awful and unkind and every time I interact with them I get made fun of for my health conditions and the way I talk/do things, not to mention the other members wouldn't let me change things. Also, one of the members actively tries to mess things up for me and lies about/to me, and I just don't want to deal with the stress on top of my 5 classes (also due to my health condition I prefer to avoid stress because it makes the condition worse/potentially life threatening). But when I told the teacher I couldn't take the position I felt really bad and she was very disappointed, it makes it worse she is a really kind teacher and person. Also, next semester I have her as my teacher as well, which I'm now nervous about. Also, one of the two classmates that's awful are going to be in the class as well. I just don't know if I did the right thing.

Did I do the wrong thing?
Post Opinion