Ok, I'll jump in here for a moment. Using Petes' diagram above I'll try to explain it this way:
The bearings are designed for a LOAD of X with a FORCE CENTER at D.
If the load (X) is the same and D remains the same it makes no difference if D includes a spacer. The forces on the bearings are the same. If the design for D is 3" it doesn't matter if 2" is from the rim and 1" is from a spacer or if 3" is from the rim, the bearings see the same force.
If D changes (+ or -) then the force exerted on the bearings changes and does nasty things to the bearings.
If load (X) changes we get the same effect, changes to the forces exerted on the bearings. Load (X) is effected by almost everything if you really get into it, but it does not have the same impact as the moment arm (D). Things get interesting from the load aspect when you take suspension into account .
Think of it like a lever if that helps.
Spacers may have some issues, but they shouldn't fry bearings if they're used correctly.
-VRJoe