If the country goes over the fiscal cliff, the impact would be borne most heavily by the wealthy. But the average middle class family could take a steep hit as well. Take for, instance, a family making $82,000 a year with four children. Both spouses work. Due to a combination of expiring tax cuts, that family would pay an extra $530 a month in taxes, or more than $6,000 a year.