Previously if there are mul like ops, the final output would be of
scale = 2 * scaling_factor. Now we introduce a scale down (if required)
so that the scale of the model output is = scaling_factor.