Commits


pengwa authored and GitHub committed 15cb2f5a8aa
Warn the user when nondet kernels are invoked in det mode (#16571) ### Give user warnings if nondeterministic kernels got called when Deterministic flag is set When we do accuracy investigation (for example training convergence issue debug), usually we will set `use_deterministic_compute ` to be true. ``` SessionOptions sess_options; sess_options.use_deterministic_compute = true; ``` While in recent investigation, it is found GatherElementsGrad kernel (who used atomic add) generate non-deterministic results, making a deberta model ouput pretty different loss curve every time we run it even we fix the seed, remove the dropout ratio, and set use_deterministic_compute to be true. It turned out to be an expected problem if we do the add in different order by cuda threads. The order cannot be guaranteed. So this PR will give warnings when users set `use_deterministic_compute `, but some kernels don't have determinstic kernel impl, has to run with non-determinstic impls. This would at least let users know the results is not determinstic though that flag is set to be True.  Only print the message once in case it floods training logs.