Also added tests which verify the intended behaviour. GitHub:#3452 ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=175656478
