Festivals are held all across the country. Georgia has their World Grits Festival, South Carolina the Cooter Festival. There are music festivals, eating festivals, rodeos, flowers - you name it. Most of them are either free or have a minimal admission fee.
There's a few web sites out there to help you find festivals - whether they're close to home or near a vacation spot. One is www.festivals.com. You put in the location and date and you get a list of events. Another is www.southfest.com, where you can fine festivals all over the southeast. One more is www.festivalsandevents.com which allows you to search anywhere.
Not only do you save money by going to a relatively inexpensive festival, it helps the local economy!