The penny is a worthless coin and a burden to not only the government but to the common taxpayer and society as a whole.
The penny costs more to make than it is worth at around 1.8 cents per every one cent made, resulting in the loss of millions of dollars each year. Not only that, but the time wasted because of pennies results in a loss of as much as $1 billion.
Pennies have become rejected in many ways by society. They are not accepted by most vending machines, parking meters, toll booths, etc. Thus, they accumulate and become an inconvienience.
Many other countries (such as Canada) have retired their one cent without great disruption, and there is no benifit to keeping it in the modern economic system.