Key Error: 'MODIN_ENGINE'

Hi guys,

I am facing an issue in which my script which includes swifter is running fine in
the virtual environment i created with python 3.6.2, but when i create a .exe file
from the same virtual environment and when i run the exe i get a Traceback.

Traceback (most recent call last):
File “modin\config\pubsub.py”, line 118, in get
File “modin\config\envvars.py”, line 32, in _get_raw_from_config
File “os.py”, line 669, in getitem
KeyError: ‘MODIN_ENGINE’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “file.py”, line 1
118, in
File “file.py”, line 8
38, in commonsteps
File “swifter\swifter.py”, line 358, in apply
File “swifter\swifter.py”, line 259, in _modin_apply
File “”, line 961, in _find_and_load
File “”, line 950, in find_and_load_unlocked
File “”, line 655, in load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File "modin\pandas_init
.py", line 180, in
File “modin\config\pubsub.py”, line 107, in subscribe
File "modin\pandas_init
.py", line 109, in _update_engine
File “modin\config\pubsub.py”, line 120, in get
File “modin\config\envvars.py”, line 75, in _get_default
File “”, line 961, in _find_and_load
File “”, line 950, in _find_and_load_unlocked
File “”, line 655, in load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File "distributed_init
.py", line 1, in
File “”, line 961, in _find_and_load
File “”, line 950, in _find_and_load_unlocked
File “”, line 655, in _load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File “distributed\config.py”, line 17, in
FileNotFoundError: [Errno 2] No such file or directory: 'D:\01_xew\02_ret

Can anyone help…!!

Hi @sohail22b, welcome to the board!

The KeyError here is caught and handled, and there’s another error coming from distributed. One temporary workaround for this is to add this to your script:

import os

os.environ["MODIN_ENGINE"] = "dask"

Unusual that it would happen as an exe file. I’ll have others weigh in on why this might be happening.

@sohail22b , I see that your python version is 3.6.2. Please note that Modin currently requires python version >= 3.7.1.

Hi devin i tried that in python version 3.9.2…still the exe wont work
getting the following traceback:
Traceback (most recent call last):
File “swifter\swifter.py”, line 340, in apply
File “DTC_dask.py”, line 840, in compare
File “pandas\core\ops\common.py”, line 65, in new_method
File “pandas\core\ops_init_.py”, line 365, in wrapper
ValueError: Can only compare identically-labeled Series objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “DTC_dask.py”, line 1127, in
File “DTC_dask.py”, line 847, in commonsteps
File “swifter\swifter.py”, line 358, in apply
File “swifter\swifter.py”, line 259, in _modin_apply
File “”, line 1007, in _find_and_load
File “”, line 986, in find_and_load_unlocked
File “”, line 680, in load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File "modin\pandas_init
.py", line 180, in
File “modin\config\pubsub.py”, line 107, in subscribe
File "modin\pandas_init
.py", line 122, in _update_engine
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File "distributed_init
.py", line 1, in
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File “distributed\config.py”, line 17, in
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\TTC01017\Documents\My Received Files\exe_392_dask\distributed\distributed.yaml’
[16400] Failed to execute script ‘DTC_dask’ due to unhandled exception!

you can also refer here for my issue: virtualenv - I am finding it difficult to create an .exe from python script using pyinstaller. My script executes perfectly but when i run the .exe i get traceback - Stack Overflow

Thanks for the update @sohail22b.

Does the same issue occur if you don’t use swifter? I wonder if there’s some combination of settings between Modin and Swifter that are causing this. We can narrow down where the issue is coming from first, then fix it.

Sure Devin i’ll try without swifter and update you. Also is it anything to do with the pyinstaller not considering all the modules for creating the exe…? and can you help with what other modules are modin and swifter dependent on …?

@sohail22b Your comment about pyinstaller inspired my search :slight_smile:. I think I may have found the issue, it’s coming from Dask: python - Dask + pyinstaller fails - Stack Overflow

Can you try this?

Hi devin tried it…no luck…!!

Traceback
Traceback (most recent call last):
File “modin\pandas_init_.py”, line 132, in _update_engine
File “distributed\worker.py”, line 4088, in get_client
ValueError: No global client found and no address provided

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “DTC_dask.py”, line 1127, in
File “DTC_dask.py”, line 847, in commonsteps
File “swifter\swifter.py”, line 358, in apply
File “swifter\swifter.py”, line 259, in modin_apply
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File "modin\pandas_init
.py", line 180, in
File “modin\config\pubsub.py”, line 107, in subscribe
File “modin\pandas_init_.py”, line 136, in _update_engine
File “distributed\client.py”, line 766, in init
File “distributed\client.py”, line 948, in start
File “distributed\utils.py”, line 332, in sync
File “distributed\utils.py”, line 315, in f
File “tornado\gen.py”, line 762, in run
File “distributed\client.py”, line 1013, in _start
File “distributed\deploy\spec.py”, line 392, in _
File “distributed\deploy\spec.py”, line 303, in start
File “distributed\scheduler.py”, line 3695, in init
File “distributed\http\utils.py”, line 40, in get_handlers
File "importlib_init
.py", line 127, in import_module
ModuleNotFoundError: No module named ‘distributed.http.scheduler’
[13496] Failed to execute script ‘DTC_dask’ due to unhandled exception!

[UPDATE] Hi Devin the exe works perfectly without swifter. With swifter still getting tracebacks, even after referring to the stack overflow you asked me to refer

getting the following traceback:
Traceback (most recent call last):
File “modin\core\execution\dask\common\utils.py”, line 25, in initialize_dask
File “distributed\client.py”, line 4763, in default_client
ValueError: No clients found
Start a client and point it to the scheduler address
from distributed import Client
client = Client(‘ip-addr-of-scheduler:8786’)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “DTC_dask.py”, line 1129, in
File “DTC_dask.py”, line 866, in commonsteps
File “swifter\swifter.py”, line 358, in apply
File “swifter\swifter.py”, line 264, in modin_apply
File “modin\pandas\dataframe.py”, line 109, in init
File “modin\config\pubsub.py”, line 213, in subscribe
File "modin\pandas_init
.py", line 141, in _update_engine
File “modin\core\execution\dask\common\utils.py”, line 41, in initialize_dask
File “distributed\client.py”, line 766, in init
File “distributed\client.py”, line 948, in start
File “distributed\utils.py”, line 332, in sync
File “distributed\utils.py”, line 315, in f
File “tornado\gen.py”, line 762, in run
File “distributed\client.py”, line 1013, in _start
File “distributed\deploy\spec.py”, line 392, in _
File “distributed\deploy\spec.py”, line 303, in _start
File “distributed\scheduler.py”, line 3691, in init
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File “distributed\dashboard\scheduler.py”, line 6, in
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File “distributed\dashboard\components\nvml.py”, line 18, in
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “PyInstaller\loader\pyimod03_importers.py”, line 476, in exec_module
File “distributed\dashboard\components\scheduler.py”, line 83, in
File “bokeh\themes\theme.py”, line 165, in init
FileNotFoundError: [Errno 2] No such file or directory: ‘E:\Share\NAT\SIDIS\Application\Ritesh\PyExes\DTC_dask\distributed\dashboard\components\…\theme.yaml’
[43256] Failed to execute script ‘DTC_dask’ due to unhandled exception!