it is just part of the learning process. We all went through it. Small changes like more weight in the seat (which will come naturally as you practise) will make the seemingly impossible, suddenly possible. Don’t stress it, just keep practising and you will get there.
To answer your specific question, no it is not a limitation of a smaller wheel. It will be more comfortable to cycle distance on a big wheel but it can certainly be done on a smaller wheel for normal people. A couple of recent examples. I cycled 25km (15.5 miles) on a single ride using a 24" at the end of last month. I certainly could have gone further. I am just a regular 45 year old guy. We also had a 10 year old on Unicycle Tour of Oslo. She rode around 17km/10.6 miles (IIRC).