As stated above, the total gear ratio is the guiding principle: shorter cranks require less leg movement which allow higher RPMs and thus higher speeds.
As to going downhill with short cranks: that is almost completely skill related. Unless you are doing slower speed trial-like movement on super technical terrain, a good rider is faster on short cranks. For example, I generally ride 110mm for DH on my 29+. If I am riding a trail above my technical ability or just want to be more relaxed, then both switching down 27.5" wheel size or switching to longer cranks up to 140mm improves my technical ability to clear sections but at a reduction in overall speed. My DH technical ability is probably best at about 125-136mm cranks, i.e. near trials length, (but faster with 110/117).
When considering uphill it is more complicated and basically a comprise.
If your goal is to go faster, then choose the shortest crank you can manage on uphills at a speed significantly faster than walking: If you’re fighting uphill with longer cranks, it requires so much energy while only being slightly faster than walking and this can then be made up on the less steep sections (or downhill) where shorter cranks are then faster.
On longer ascents (say 200-1000m vertical), I generally cannot ride the sustained pace required for short 110mm cranks (on road or good surfaces I sometimes can if I’m really fit) . For recreational riding I generally switch the cranks using dual hole (my current favorite is QX 110/136). Although for really long sustained ascents 150mm are definitely easier and less exhausting.
For shorter uphills like in XC or in racing where you don’t change crank length, there are mainly 2 guidelines/options: choose the shortest crank you can push for most sections significantly faster than walking speed (and walk the steeper). Or if you’re really good, then use a Slumpf. Note that this is mostly only for really top riders like Maxence or Ben (I think my improvement at the XC of the last 2 Unicons had I had a Schlumpf would have been minimal). I think I rode last Unicon XC with 29" with 125s (only walked at the very end when I was too tired due to no more cardio) and Grenoble with 117s as I walked the few short steep sections.
I think the top riders mostly used sub 125mm (mostly 110mm), or geared with Schlumpf. DH was mostly 110mm with almost no one over 125mm.
Also uphill depends on the surface. Road uphill you can push crazy short cranks if you’re strong: last year at Elsbet, I rode with Maxence where he rode uphill grades around 20% with around 125mm cranks in high gear!! I was walking with my 110mm cranks. But Maxence is of course an exceptional uphill rider.
For bumpy or off-road uphills then it’s a mixed bag. As said above, there is a small range of incline/terrain where longer crank length allows you to ride without dismounting, but concerning overall speed the advantage usually dissapears quickly.
Before I was racing and just riding for fun, I generally used dual 125/150mm cranks. With the 150s I can ride steep extended sustained climbs of 300-1000m vertical. Then I switched to 125s for downhill.
In the last years I have really enjoyed running shorter cranks, but you have to improve technique for downhill as well as strength and fitness for uphill.
My recommendation: give shorter cranks a try. For DH you don’t need anything longer than 140, so try 110-136mm for 29/27.5" and max 125mm for 26/24".
Many top riders also forgoe the dual cranks or Schlumpf and stick with one crank length for all XC/DH and concentrate on spinning fast for DH. I think 125mm is here the most common, although many strong riders also use 117mm or 110mm.
Uphill depends on what you ride. Ultra steep: just walk. Moderate or road: short cranks like 110-125 are totally doable with training. Long climbs or mixed steep, it often makes sense to go longer to 136 or even 150mm as it’s generally more fun to stay on the uni than walk a long time.