Error with new version of modin

with modin 0.5.1 and ray 0.7, I get the following error when I run query on a dataframe. Something with my setup or a genuine bug? If the latter, I’ll file an issue.

Dinesh

File “/home/ddutt/work/a/engines/modin/engine.py”, line 91, in get_table_df
.query(query_str)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/pandas/dataframe.py”, line 1260, in query
new_query_compiler = self._query_compiler.query(expr, **kwargs)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/backends/pandas/query_compiler.py”, line 1669, in query
new_index = self.compute_index(0, new_data, True)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/backends/pandas/query_compiler.py”, line 93, in compute_index
old_blocks=old_blocks,
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/base/frame/partition_manager.py”, line 535, in get_indices
if len(self._partitions_cache.T)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/base/frame/partition_manager.py”, line 534, in
[idx.apply(func).get() for idx in self._partitions_cache.T[0]]
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/ray/pandas_on_ray/frame/partition.py”, line 34, in get
handle_ray_task_error(e)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/ray/utils.py”, line 13, in handle_ray_task_error
raise getattr(builtins, s.split(":")[0])("".join(s.split(":")[1:]))
ValueError: Length mismatch Expected axis has 28 elements, new values have 10 elements

This looks like a bug to me.

Does get_table_df use .get or other indexing method before query? We made some performance updates related to indexing and it might not be playing nice with other operations.

Yes, it does. The exact code is here:

        final_df = pd.read_parquet(folder, columns=fields, filters=filters or None) \
                            .query(query_str)

Let me know if I should file a bug,

Dinesh

Yes, go ahead and file a bug report. I don’t see the indexing method in the code snippet you provided, what is the indexing method used?

I misspoke. I’m not using any index.

Dinesh

This was fixed in 0.5.2 release: https://github.com/modin-project/modin/issues/642

Thanks again for the report.