They do stink when you pick them up! The stink glands are located between the first and second pairs of legs.
Stink bugs have a needle-like mouth that is used to pierce the skin of the fruit (and insects) and suck the juice. When these "needles" are not in use, they are tucked between front legs.
Stink bugs can bite when it is threatened. Bites can be painful and it can induce redness and swelling of the skin.
Stink bugs are considered true bugs, which means they are part of a special group of insects. True bugs have mouth parts that look like bird beaks and can pierce and suck. They use their mouthparts the way you might use a straw to drink a soda or shake.