Work itself doesn't seem like much of a curse to me. Most people I know like their jobs and really don't mind working. Some would comment that they'd give anything to not have to work and simply bang on a drum all day. We usually roll our eyes and call such people lazy (all the while resonating with them at various levels).
But God gave Adam work to do BEFORE the fall: he was told to rule over the creation in Genesis 1. God is OK with us working. The reason why Adam' curse is so cursed comes in how hard it will be to work & provide:
"Cursed is the ground because of you; through painful toil you will eat of it all the days of your life. It will produce thorns and thistles for you, and you will eat the plants of the field. By the sweat of your brow you will eat your food until you return to the ground, since from it you were taken; for dust you are and to dust you will return." Genesis 3:17-19
Providing food for the family wouldn't be easy, now He would have to sweat for it. Now he would have to take the time and the effort required to grow and gather the food.
The implications of this time & effort go even farther.
Yesterday my wife, Jeannette, said to me as I was leaving, "I wish you didn't have to go today."
Every morning before I leave I get hugs and kisses from my 2 1/2yr old daughter, Baylee. All the while she says with pouty lip, "Won't you stay with me today?" Normally she cries after this, daily breaking my heart.
And that is why the work element of the curse is so cursed to me: I'd rather be with them. Shoot, I'd even work with them if I could. But I can't, I have to walk out the door with the tears behind me, knowing I can't participate in what they'll be doing today.
The curse is horrible because of how it puts strain on the relationships that are most important.