Right, I agree boxing every number would be problematic, although Guava's unsigned support works with primitives. With inline functions the cost hit could be not terrible, if it was possible to avoid the boxing.
I’m not suggesting ruling things out. As mentioned, Swift gives you “fast but less safe” operators that don’t check for overflow, suitable for use in matrix mult routines and the like.
I cannot offer any data on how often overflow bugs crop up in various apps. Perhaps someone has studied it, I’ll have a look around. For one pretty disasterous example, check out a bug that occurred in Bitcoin in 2010. Someone discovered a place where value fields in the protocol were being summed without checking for overflow. The result was someone managed to create billions of bitcoins, more than should ever have existed. The entire system had to be rolled back and replayed without the bad transaction. Overflow bugs in security sensitive software can be quite damaging, but hard to find in the normal course of operations and testing.