I have used little strips of rubber (like what you use when mounting cycle computers or lights) as shims between the bearing and holders.
In my case I needed to increase the tyre clearance on the frame by a few mill but I think rubber would make a fairly good packing material. It should distribute the load, is deformable so it isn’t too hard to squeeze into a hole and won’t be nearly as messy as epoxy. You may need to cut and trim to get exactly what you want but I reckon it is worth trying.
Remember you don’t want to overtighten your bearing holders as that will destroy your bearings eventually.