pttools.speedup.parallel

Classes

class pttools.speedup.parallel.FakeFuture(func, *args, **kwargs)

Bases: object

Parameters:

func (callable)

result()
class pttools.speedup.parallel.LoggingRunner(func, arr_size, unpack_params, args=(), kwargs=None, log_progress_element=None, log_progress_percentage=None)

Bases: object

A handler for logging the execution status of a function that is run in parallel

Parameters:

  • func (callable)

  • arr_size (int)

  • unpack_params (bool)

  • args (tuple)

  • kwargs (Dict[str, any])

  • log_progress_element (int)

  • log_progress_percentage (float)

run(param, index=None, multi_index=None)
Parameters:

  • index (int)

  • multi_index (Iterable)

Functions

pttools.speedup.parallel.run_parallel(func, params, max_workers=MAX_WORKERS_DEFAULT, multiple_params=False, unpack_params=False, output_dtypes=None, return_arr_shape=None, log_progress_element=None, log_progress_percentage=None, args=(), kwargs=None, single_thread=False)

Run the given function with multiple parameters in parallel

Parameters:

  • func (callable) – The function to be executed in parallel

  • params (ndarray) – Array of the function parameters

  • max_workers (int) – Maximum number of worker processes

  • multiple_params (bool) – Whether the last dimension of the parameter array contains multiple parameters for each function call

  • unpack_params (bool) – Whether the multiple parameters should be unpacked before giving them to the function

  • output_dtypes (Tuple[Type, ...] | List[Type]) – If the function has multiple output values, their types should be given here

  • return_arr_shape (Tuple[int, ...]) – Shape of the array given by func. If None, the function should return single values.

  • log_progress_element (int) – Log progress every n element

  • log_progress_percentage (float) – Log progress every x %

  • args (list | tuple) – common arguments for the function

  • kwargs (Dict[str, Any]) – common kwargs for the function

  • single_thread (bool) – disable parallelism for debugging and profiling

Returns:

Numpy arrays for each output value

Return type:

ndarray | Tuple[ndarray, …] | None