onnxruntime

Device handling fixes in ORTModule (#6187)

* Fix typo in ORTModule.to()

The `args` and `kwargs` should be expanded in the call to `super(...).to()`.

* Add fixes for multiple CUDA devices.

* Add simple DeepSpeed test script and configuration.

* Fixes for test script and config.

* Add trailing newline.

* Fix formatting for config.

* Set InferenceSession provider options at construction.

* Make the local_rank arg required.

* Convert ORTModule._device to a torch.device() before using its accessors.

* Refactor device handling and fix regressions on BERT fine tuning

Co-authored-by: Derek Murray <demurra@microsoft.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>