Preprocessing a function for remote call

Currently, we preprocess a function that is performed in a ray remote call putting the function into plasma via ray.put(func). However, in case of dask engine we preprocess a function that is performed in a dask remote call via cloudpickle.dumps(func) for single partition only, but not for other remote calls. We leave the function as it is for those cases. I think it is an issue of design and we need to think of it to resolve this. Also, we might preprocess a function for dask engine via client.scatter(func). Is there an issue with it or use of cloudpickle has an advantage over scatter?

@YarShev We had some early issues with Dask and function dependencies because of how we submit the tasks. It might be possible to change that now, and not use cloudpickle.dumps or client.scatter. Ultimately, client.scatter will likely call cloudpickle.dumps internally.

Maybe the first thing to do would be to verify that we can operate without any function pickling. The issue may be resolved by now.

@devin-petersohn , I was verifying the case, when we operate without any function pickling, on Ubuntu 20.04.2 LTS some time ago and all our tests were passed. Yes, we might fix this. Anyway, preprocessing a function would be good for performance. Do you think we need some internal refactor?

@YarShev Yes, and internal refactor will be useful.