A penny costs 2.5c to make, and the US Government make billions of them a year, at a cost of $100s of millions a year. But with inflation, a quarter today is worth the same as a penny only 100 years ago! (so by that, we should get rid of the nickle and dime as well)!
A dollar bill costs 6-8c to make, only lasts 18 months, and we print billions of these each year as well. A dollar coin costs 5x to make, but lasts 20x as long (and can be recycled), again saving $100s of millions each year.