Welcome to the Internet. It means you say something to get a reaction out of someone. Usually you'll say something that's extremely offensive or annoying in hopes that people will become angry at you then you can follow it it up with a "umad?" Which just pisses them off even more.
When someone is trolling, they're basically saying something controversial/offensive just because they know it'll rile people up. They say these things because they want attention, and usually they get it too. This is why some people might say "don't feed the troll". They basically don't want you to respond to whatever the troll has said, because if you do, the troll will keep trolling. But if you don't pay any attention to the troll, they will most likely stop, since they're not getting any satisfaction out of it (i. e. their attempts at fooling others have failed).