In answer to your last question first, a god quality unicycle, designed for your preferred type of riding, makes a huge difference.
However, it is possible to challenge yourself on uneven surfaces on any size and quality of unicycle.
My first uni was a Pashley UMX: heavy steel wheel, awful seat, cottered cranks, general purpose cheap BMX tyre. I rode it many miles around the local nature reserve including short sections of single track and a few steep little hills.
I have a custom 700c unicycle with a narrow rim and a high pressure 23mm section road tyre. On 14mm cranks I enjoy riding it on dry uneven surfaces, fencing with the trail instead of bludgeoning it.
I also ride cross country on a purpose made KH24, a KH29 and a 36.
So don’t let your present uni stop you trying things, but do start to dream about owning a more suitable one.
As for how to make your first attempts at Muni:
Down hill is easier than uphill, but just as exciting.
Find easy paths with interesting bits at the side.
Ride for a bit until you feel comfortable, then divert along something “interesting” until you feel tired, then go back onto the easy tracks to get your breat back and reestablish a rhythm.
Do things like cutting corners by going over the raised grass at the side of the main path.
Look for small lumps and bumps you can ride over, everywhere you go.
Eventually, what you class as “easy” will be what you used to class as “interesting”, and you will gradually develop your skills almost without realising it.
I found it was a mistake to push myself too hard and to be too ambitous. It is a miserable feeling to be a mile from the nearest path, so tired you can barely freemount, and so bogged down in long grass or mud that you can’t ride more than a few feet at a time.
And whatever you do, there is no substitute for spending a lot of time doing it. Eventally, it becomes as natural as walking.