Men (and women) should be gawking with what you have now. Going bigger will only increase the probability. Question is, have you been able to deal with the gawking and increased attention to date? If so, then don't worry about more of the same. Surely you should know that it's not just what you have on your chest but how you present them, ie, dress them up/down.
Feeling norty? Go for the bra which really lifts them up and out there, and wear that top with the plunging neckline.
Feeling like having a quiet day? Time for the minimizer bra and the baggy sweatshirt.
My wife has never had any problems dealing with people who make comments about her boobs, and isn't too concerned with the increased gawking/comments after she goes to get pumped up. If anything, her biggest concern is dealing with women who make snippy comments when our boy is around. So if you're anything like her, you've got nothing to worry about.
Still wondering if you should go for it? As I often tell my wife, "You know you wanna."
MD{s^b}