If you continue to hold it the way you're holding it now then it will always be darker due to the amount of natural oil and dirt that is coming off your hands onto that part of the handles.
However you can always "tan" your bag and cover the parts of the handles where you hold it with a thick towel or something equivalent. Or you can start carrying your bag with a scarf and have the scarf absorb the dirt and oils from your hand.