Just don’t over tighten the bearing holders once you do. Once you shim them, they’ll have a smaller internal diameter and there will no longer be an even amount of pressure applied to the bearings.
As a result, this can happen - Split open bearing casings! (although that wasn’t from shimmed bearing holders…they were too small to begin with)
Andrew