Logging within a worker function

I was wondering if it’s possible to log within a custom function run within a modin worker.

I’m using modin with a ray backend and have read ray’s documentation on logging here.

My code mostly resembles the following:

import logging 
import time
import modin.pandas as pd

def custom_func(df_row):
    """some long running custom func"""
    start_time = time.time()
    ... 
    # do a bunch of stuff on each row in parallel within a worker 
    logging.info(f"really long running func took: {time.time() - start_time}...")


def main():
    # setup a logging config that generates a log file
    modin_csv_df = pd.read_csv("some_csv_file_to_parallelise.csv")
    modin_csv_df["new_column"] = modin_csv_df.apply(lamda row: custom_func(row), axis=1)

I’m not getting any logging from the worker, obviously it’s not outputting to the same logging stream.
The ray documentation descripes the remote like api which doesn’t quite aline with modin’s implementation, so I haven’t been able to figure out how to correctly log to file.
Logging to stdout/terminal works, just not to file.

This is something we should definitely look into. I have been thinking about this as well, so I will open an issue to enable this.