Commits


Hariharan Seshadri authored and GitHub committed f1044e3b9a9
CUDA GreedySearch ProcessLogits optimization (#13823) ### Description Explore the possible re-use of the logits buffer in `GreedySearch` for cases where sequence length == 1 (Post the first decoding run, the sequence length is guaranteed to be 1). This re-use will ensure that we do not have to make copies of the logits before processing them. Currently, we make a copy of the logits even if the sequence length == 1 which is not necessary as we can directly re-use the logits buffer for the token generation step. A similar optimization exists in `BeamSearch`, but seems lacking in `GreedySearch`. Since, the logits buffer may contain padded data, we need to adjust the pieces consuming the logits buffer directly to account for any padding. A more invasive change (needs changes in a few places) will be to adjust the interfaces of `ProcessLogits()` such that it takes a reference to the logits and not a const reference as (based on my understanding) this is the only place where the logits from the decoder subgraph will ever be used and giving the `ProcessLogits()` method license to mutate/process the underlying buffer of the logits OrtValue seems reasonable (instead of making a copy and then mutating/processing them). The will also remove the ugly `const_cast`(s) seen in this change.