Set scheduling_strategy to SPREAD when using Modin on Ray

Hi all, I’m using Modin API on Ray. When I call function such as .count(), from the dashboard I saw that tasks are distributed unevenly among nodes, one node is heavily loaded, while the cpu utilization of other nodes is quite low, and I saw there is one configuration in Ray called scheduling_strategy, by default it is set to value DEFAULT, and there is another value SPREAD, I think maybe by setting the scheduling_strategy to SPREAD the tasks can be more evenly distributed. However I was not able to find where I can set the configuration in the code, do you guys have any idea on this? Any suggestion would be appreciated.

HI @j_j12, thank you for posting here again.

count on ray should be implemented with parallelism across both rows and columns. ray will normally do a good job of balancing work across workers even with the DEFAULT scheduling. It’s possible that your modin datafram eonly has one partition.

Are you able to give a complete reproducer for the unbalanced execution? This would be a script that we can use that behaves similarly to your code and also shows the single node execution.

If you can’t give a reproducer, could you please share the results of the following, where obj is the modin dataframe or series that is slow for you:

print(obj.shape)
print(obj._query_compiler._modin_frame._partitions.shape)
print([(p.length(), p.width()) for p in obj._query_compiler._modin_frame._partitions.flatten()])

Also, please post your versions of:

  • python
  • modin
  • ray

Hi @mahesh , thank you for the reply!

Here is the version information:

  • Python: 3.7
  • Ray: 2.0.0
  • Modin: 0.12.1

Here is the result of running the provided code:

(30656013, 109)
(32, 28)
[(958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 4), (958001, 1), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4), (957982, 4)

And I ran the .count() just now, but this time it seems that the tasks are distributed more evenly, but I did not make any change to code or config, does that mean the distribution is random?

@j_j12 the scheduling shouldn’t be random. Ray should generally try to distribute work evenly across workers, though depending on partitioning and exactly how your code is using modin, ray may not be able to give an even split.

It’s hard to tell without an example why you were seeing biased scheduling in your first example.

By the way, are you able to use Python 3.8? If you’re on python 3.8, you can use the latest version of Modin, 0.16.1, which should have better performance and many more features than 0.12.1.

Hey @mahesh. I have a couple of follow-up queries:

  • I am running modin[ray] on my local setup(quad-core intel processor MacBook Pro). Based on my ray dashboard, when I change from modin.config.CpuCount.put(1) to modin.config.CpuCount.put(2) I see an improvement in run-time by a factor of ~0.6 but increasing the cpu_count beyond 2 doesn’t improve the performance any further.
  • The run-time for modin[ray] is very high even for a read_parquet or mean by a factor of almost 10 compared to pandas on my local setup. What can be the possible reasons to this?

@Krish modin will not necessarily be faster than pandas in all use cases, and sometimes it can be slower than pandas.

I see an improvement in run-time by a factor of ~0.6 but increasing the cpu_count beyond 2 doesn’t improve the performance any further.

This could happen because the operation you’re testing doesn’t leverage all the cores. Your dataframe might not have enough partitions for Modin to leverage all cores. I can’t say much more without seeing what your script is doing and without some more information about the Modin object, e.g. what is its shape?

What can be the possible reasons to this?

There are a few possible reasons:

  • you might be looking at an operation that is defaulting to pandas. These will always be at least slightly slower than pandas
  • Often, the functions that Modin implements in a distributed way will not perform better than pandas on smaller data (say less than 50 MB)
  • Modin may simply have a bad implementation of the functions you’re testing, in which case we should see how we can improve Modin

It would be helpful if you could provide:

  • Reproducible scripts for all the performance issues you have.
  • If you can’t provide a reproducible script, the names of the functions you’re calling, and a description of how you’re calling them and with what kind of data.
  • Your modin version
  • your python version

If you want to respond, let’s please continue in a new thread. I want to keep this thread focused on @j_j12 's issue.

Hi @mahesh , after I upgrade python to 3.8 and Modin to 0.16.1, I am not able to read the parquet from my s3 bucket, it throws error pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path '<bucket>/data/xxx.snappy.parquet', which is outside base dir 's3://<bucket>/data/', may I know if you have any idea on this issue? The version of pyarrow is 9.0.0.

@j_j12 it seems that you’ve found a Modin bug. Could you please report it here and make absolutely sure to include:

  • an entire script that reproduces the error on your end (it’s ok if the public doesn’t have access to your s3 bucket and/or if you anonymize the s3 path)
  • the complete stack trace

The other modin contributors and I would be happy to help with your issue.

Hi @mahesh, I tried to set the engine to fastparquet, and now I am able to read from s3 bucket. However, I found that it is a bit slow when using fastparquet.
In Modin 0.12.1, pd.read_parquet(input_path) took 565 seconds, while in Modin 0.16.1, pd.read_parquet(input_path, engine='fastparquet') took 630 seconds.
May I know if I should still report this issue?

@j_j12 it would be great if you could report two separate issues:

  1. ArrowInvalid error when using the default engine
  2. slow performance when using fastparquet

For each issue, having a minimal, complete, verifiable reproducer script will make it much easier for the Modin community to help.

Hi @mahesh, thank you for the suggestion! I have reported the bug here BUG: Cannot read parquet file from s3 bucket when using Modin 0.16.1 on Ray · Issue #5128 · modin-project/modin · GitHub.
I also tested read_parquet() by using default engine and fastparquet engine both with Modin 0.12.1, however this time the performance is almost the same. So for now, I only posted the first issue.