I like to brush because I think it easier to watch TV and brush at the same time. If a cloth is working for you, then I think that is the way to go.
Is your brush soft? If not, then your brush might be the problem. I have several Kiwi brand like this and love them:
https://www.amazon.com/Kiwi-100%-Ho...id=1469105908&sr=8-2&keywords=horsehair+brush
I brush in long gentle strokes and brush lots and lots of times. I sometimes brush right after applying conditioner, then usually brush several more times before applying the next round. The drier the bag, the more I tend to brush it. I usually only buff with a cloth after the final coat of Blackrocks.