This is why I don't understand why people love shoulder totes. All shoulder bags are also practically impossible with raglan, or round-shouldered sleeves or most oversized style outerwear, both fashionable right now.
You may actually find it easier to make it stay on when a bit heavier. Also, like hobos, it will depend on what you're wearing underneath.