If this is a 'nature' trail, not just a jogger/dog walking trail then...
Not asphalt. Who wants to go hiking on the same stuff that's all around town. Also, depending on your climate, it may be hot and sticky in the summer. Then there is that early melt thing to bum out the skiers.
Not gravel. Riding bikes in gravel is no fun, especially trying to make that turn at the bottom of the hill. While hiking you can't quite hear nature over the crunch, crunch, crunch of gravel under your feet. And yes, those spring skiers will be cursing the gravel that seems to find it's way up a few inches of snow just to scratch the bottoms of their skis.
Dirt. That's the ticket. Free, easy to maintain. Easy on the feet. I look for the mud when on my mountain bike. If there is a problem with too much mud then do the wood chip thing like groomerguy said. It's insulating, quiet, soft, and smells good too.
That polypavement looks interesting, but how hard and how much?