At small to medium wheel sizes, an inch or two makes more difference.
For example, a 24" uni is 4/20 (20%) bigger than a 20.
A 28" uni is only 4/24 (16.7%) bigger than a 24.
This means that, all other things being equal, a 24 will be 20% faster than a 20, and a 26 will be only 8% faster than a 24.
Assuming a riding speed around 8 mph/13 kmh, all other things being equal, a 26 will only go 2/3 mph hour faster than a 24 - which is negligible.
However, all other things aren’t equal, because you need to consider the tyre section, the availability of tyres, your level of skill and confidence, and so on.
Also, because of the angles involved, a 26 will roll over things more easily than a 24.
Rather than going into the maths, think of these extreme examples:
A 2 inch wheel hits a 2 inch vertical step. It stops dead.
A 36 inch wheel hits a 2 inch vertical step: it barely notices.
Now apply this obvious principle to the 24 and 26 meeting a typical tree root. Whatever you decide to call it (angle of incidence? Angle of attack? Rollover factor?) the bigger wheel will always make less fuss about any given obstacle.
A fatter and softer tyre will also smooth out these small obstacles.
On the other hand, if you want to hop over an obstacle, the smaller and lighter the uni, the easier it is.
I ride and have ridden several unis off road on forest trails, rough tracks, uncultivated ground, and sometimes moorland. These include:
20 wth 125s
24 x 1.75 with 102mm cranks
26 with variosu cranks
28 x 23 with 114 mm cranks
29 with 150s and 125s
24 x 2.3 with 170s and with 150s
36 with 150s and with 125s
While you are just riding along the trail, there is no doubt that a 29 is easier than a 26 and a 26 is easier than a 24, as long as you are confident on that wheel size.
As soon as you need to stop, idle, reverse, turn suddenly, etc., the smaller size is easier.
End result: 3 recommendations:
If I could only afford one uni for general purpose including light off road, trails and the like, it would be a 29.
If I was never going to ride it on tarmac, but only on rough ground and trails, I’d probably go for the 26.
If I was going to ride the toughest stuff I could manage, and take risks, I’d go for a fat tyred 24.