Why do people say taking candy away from a baby is easy?

People don't say taking candy away from a baby is easy, but rather the opposite - that you will have a hard time retrieving something a baby is holding because they will grip it tightly and start crying if it is taken away.