When dropping the remainder, the decoder and encoder timestamps start diverging after a few buffers when no speed changes are supposed to occur. Tracking the remainder keeps them in sync. PiperOrigin-RevId: 408341074
