Error with new version of modin

#1

with modin 0.5.1 and ray 0.7, I get the following error when I run query on a dataframe. Something with my setup or a genuine bug? If the latter, I’ll file an issue.

Dinesh

File “/home/ddutt/work/a/engines/modin/engine.py”, line 91, in get_table_df
.query(query_str)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/pandas/dataframe.py”, line 1260, in query
new_query_compiler = self._query_compiler.query(expr, **kwargs)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/backends/pandas/query_compiler.py”, line 1669, in query
new_index = self.compute_index(0, new_data, True)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/backends/pandas/query_compiler.py”, line 93, in compute_index
old_blocks=old_blocks,
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/base/frame/partition_manager.py”, line 535, in get_indices
if len(self._partitions_cache.T)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/base/frame/partition_manager.py”, line 534, in
[idx.apply(func).get() for idx in self._partitions_cache.T[0]]
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/ray/pandas_on_ray/frame/partition.py”, line 34, in get
handle_ray_task_error(e)
File “/home/ddutt/.local/share/virtualenvs/a-cFmMv4Vf/lib/python3.7/site-packages/modin/engines/ray/utils.py”, line 13, in handle_ray_task_error
raise getattr(builtins, s.split(":")[0])("".join(s.split(":")[1:]))
ValueError: Length mismatch Expected axis has 28 elements, new values have 10 elements

#2

This looks like a bug to me.

Does get_table_df use .get or other indexing method before query? We made some performance updates related to indexing and it might not be playing nice with other operations.

#3

Yes, it does. The exact code is here:

        final_df = pd.read_parquet(folder, columns=fields, filters=filters or None) \
                            .query(query_str)

Let me know if I should file a bug,

Dinesh

#4

Yes, go ahead and file a bug report. I don’t see the indexing method in the code snippet you provided, what is the indexing method used?

#5

I misspoke. I’m not using any index.

Dinesh

#6

This was fixed in 0.5.2 release: https://github.com/modin-project/modin/issues/642

Thanks again for the report.