ST ji, just for my knowledge - I believe this is subjective to, how they show this in their account books. If they are showing as discount in brokerage, the service tax may be applicable only on the brokerage paid.
For ex - brokerage charges - Rs 1000 . refunded - Rs 500.
They can show actual brokerage charged of Rs 500 , instead of showing 2 transactions above and submit the service tax to govt as per Rs 500 only, irrespective of the tax collected for Rs 1000 from client.
Alternatively, they can also pay entire service tax to govt - as u said, and show the refunds as incentives.
Not sure if there are exclusively laid down rules for such accounting for brokerage houses.
Also from the recent changes, they have recently reduced the refund amount to Rs 30000 , which is coincidentally also the slab limit to deduct TDS in case of incentive paid. So most likely they are showing as incentive payout