Package org.anchoranalysis.inference.concurrency

Class ConcurrentModelPool<T extends InferenceModel>

Object

ConcurrentModelPool<T>

Type Parameters:

T - model-type

All Implemented Interfaces:

AutoCloseable

public class ConcurrentModelPool<T extends InferenceModel>
extends Object
implements AutoCloseable

Keeps concurrent copies of a model to be used by different threads.

The copies van variously use GPU and CPU for execution, with GPU always being given priority.

Author:

Owen Feehan

Constructor Summary

Constructors

Constructor	Description
ConcurrentModelPool(ConcurrencyPlan plan, CreateModelForPool<T> createModel, Logger logger)	Creates with a particular plan and function to create models.

Method Summary

Modifier and Type	Method	Description
void	close()	Close all models, to indicate they are no longer in use, and to perform tidy-up.
<S> S	executeOrWait(CheckedFunction<ConcurrentModel<T>,S,ConcurrentModelException> functionToExecute)	Execute on the next available model (or wait until one becomes available).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ConcurrentModelPool

public ConcurrentModelPool(ConcurrencyPlan plan,
 CreateModelForPool<T> createModel,
 Logger logger)
                    throws CreateModelFailedException

Creates with a particular plan and function to create models.

Parameters:

plan - a plan determining how many CPUs and GPUs to use for inference.

createModel - called to create a new model, as needed.

logger - where feedback is written about how many GPUs or CPUs were selected.

Throws:

CreateModelFailedException - if a model cannot be created.

Method Details

executeOrWait

public <S> S executeOrWait(CheckedFunction<ConcurrentModel<T>,S,ConcurrentModelException> functionToExecute)
                    throws Throwable

Execute on the next available model (or wait until one becomes available).

If an exception is thrown while executing on a GPU, the GPU processor is no longer used, and instead an additional CPU node is added. The failed job is tried again.

Type Parameters:

S - return type

Parameters:

functionToExecute - function to execute on a given model, possibly throwing an exception.

Returns:

the value returned by functionToExecute after it is executed.

Throws:

Throwable - if thrown from functionToExecute while executing on a CPU. It is suppressed if thrown on a GPU.

close

public void close()
           throws Exception

Close all models, to indicate they are no longer in use, and to perform tidy-up.

Specified by:

close in interface AutoCloseable

Throws:

Exception - if a model cannot be successfully closed.