python - Exception thrown in multiprocessing Pool not detected -

python - Exception thrown in multiprocessing Pool not detected -

it seems when exception raised multiprocessing.pool process, there no stack trace or other indication has failed. example:

from multiprocessing import pool def go(): print(1) raise exception() print(2) p = pool() p.apply_async(go) p.close() p.join()

prints 1 , stops silently. interestingly, raising baseexception instead works. there way create behavior exceptions same baseexception?

i have reasonable solution problem, @ to the lowest degree debugging purposes. not have solution raise exception in main processes. first thought utilize decorator, can pickle functions defined @ top level of module, that's right out.

instead, simple wrapping class , pool subclass uses apply_async (and hence apply). i'll leave map_async exercise reader.

import traceback multiprocessing.pool import pool import multiprocessing # shortcut multiprocessing's logger def error(msg, *args): homecoming multiprocessing.get_logger().error(msg, *args) class logexceptions(object): def __init__(self, callable): self.__callable = callable homecoming def __call__(self, *args, **kwargs): try: result = self.__callable(*args, **kwargs) except exception e: # here add together debugging help. if multiprocessing's # debugging on, arrange log traceback error(traceback.format_exc()) # re-raise original exception pool worker can # clean raise # fine, give normal reply homecoming result pass class loggingpool(pool): def apply_async(self, func, args=(), kwds={}, callback=none): homecoming pool.apply_async(self, logexceptions(func), args, kwds, callback) def go(): print(1) raise exception() print(2) multiprocessing.log_to_stderr() p = loggingpool(processes=1) p.apply_async(go) p.close() p.join()

this gives me:

1 [error/poolworker-1] traceback (most recent phone call last): file "mpdebug.py", line 24, in __call__ result = self.__callable(*args, **kwargs) file "mpdebug.py", line 44, in go raise exception() exception

python exception multiprocessing