Package org.apache.sysds.resource.enumeration

Class InterestBasedEnumerator

java.lang.Object

org.apache.sysds.resource.enumeration.Enumerator

org.apache.sysds.resource.enumeration.InterestBasedEnumerator

public class InterestBasedEnumerator
extends Enumerator

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.resource.enumeration.Enumerator

Enumerator.Builder, Enumerator.EnumerationStrategy, Enumerator.OptimizationStrategy

Field Summary

Fields

Modifier and Type	Field	Description
static final double	BROADCAST_MEMORY_FACTOR
static final double	CACHE_MEMORY_FACTOR
static final double	MEMORY_DELTA_FRACTION
static final double	MEMORY_FACTOR
static final long	MINIMUM_RELEVANT_MEM_ESTIMATE
static final boolean	USE_MEMORY_DELTA	different instance families can have slightly different memory characteristics and using memory delta allows not ignoring equivalent instances of different families (e.g.

Fields inherited from class org.apache.sysds.resource.enumeration.Enumerator

COST_DELTA_FRACTION, DEFAULT_MAX_EXECUTORS, DEFAULT_MIN_EXECUTORS

Constructor Summary

Constructors

Constructor	Description
InterestBasedEnumerator(Enumerator.Builder builder, boolean interestLargestEstimate, boolean fitDriverMemory, boolean interestBroadcastVars, boolean interestOutputCaching)

Method Summary

Modifier and Type	Method	Description
ArrayList<Integer>	estimateRangeExecutors(int driverCores, long executorMemory, int executorCores)	Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations
boolean	evaluateSingleNodeExecution(long driverMemory, int cores)
static TreeSet<Long>	getMemoryEstimates(Program currentProgram, boolean outputOnly, double memoryFactor)	Extracts the memory estimates which original size is larger than MINIMUM_RELEVANT_MEM_ESTIMATE
boolean	interestBroadcastVars()
boolean	interestEstimatesInCPEnabled()
boolean	interestLargestEstimateEnabled()
boolean	interestOutputCachingEnabled()
void	preprocessing()	Called once to enumerate the search space for VM instances for driver or executor nodes.

Methods inherited from class org.apache.sysds.resource.enumeration.Enumerator

getCostsWeightFactor, getDriverSpace, getEnumStrategy, getExecutorSpace, getInstances, getMaxPrice, getMaxTime, getOptimalSolution, getOptStrategy, postprocessing, processing, setCostsWeightFactor, setCpuQuota, setDriverSpace, setExecutorSpace, setMinPrice, setMinTime, updateOptimalSolution

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MINIMUM_RELEVANT_MEM_ESTIMATE

public static final long MINIMUM_RELEVANT_MEM_ESTIMATE

See Also:

• Constant Field Values

USE_MEMORY_DELTA

public static final boolean USE_MEMORY_DELTA

different instance families can have slightly different memory characteristics and using memory delta allows not ignoring equivalent instances of different families (e.g. newer generation use 7.5/15.25/30.5/... GB memory instead of 8/16/32/...)

See Also:

• Constant Field Values

MEMORY_DELTA_FRACTION

public static final double MEMORY_DELTA_FRACTION

See Also:

• Constant Field Values

MEMORY_FACTOR

public static final double MEMORY_FACTOR

BROADCAST_MEMORY_FACTOR

public static final double BROADCAST_MEMORY_FACTOR

See Also:

• Constant Field Values

CACHE_MEMORY_FACTOR

public static final double CACHE_MEMORY_FACTOR

See Also:

• Constant Field Values

Constructor Details

InterestBasedEnumerator

public InterestBasedEnumerator(Enumerator.Builder builder,
 boolean interestLargestEstimate,
 boolean fitDriverMemory,
 boolean interestBroadcastVars,
 boolean interestOutputCaching)

Method Details

preprocessing

public void preprocessing()

Description copied from class: Enumerator

Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

Specified by:

preprocessing in class Enumerator

evaluateSingleNodeExecution

public boolean evaluateSingleNodeExecution(long driverMemory,
 int cores)

Specified by:

evaluateSingleNodeExecution in class Enumerator

estimateRangeExecutors

public ArrayList<Integer> estimateRangeExecutors(int driverCores,
 long executorMemory,
 int executorCores)

Description copied from class: Enumerator

Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations

Specified by:

estimateRangeExecutors in class Enumerator

Parameters:

driverCores - CPU cores for the currently evaluated driver node

executorMemory - memory of currently evaluated executor node

executorCores - CPU cores of currently evaluated executor node

Returns:

- [min, max]

getMemoryEstimates

public static TreeSet<Long> getMemoryEstimates(Program currentProgram,
 boolean outputOnly,
 double memoryFactor)

Extracts the memory estimates which original size is larger than MINIMUM_RELEVANT_MEM_ESTIMATE

Parameters:

currentProgram - program for extracting the memory estimates from

outputOnly - true - output estimate only; false - sum of input, intermediate and output estimates

memoryFactor - factor for reverse scaling the estimates to avoid scaling the search space parameters representing the nodes' memory budget

Returns:

memory estimates in ascending order ensured by the TreeSet data structure

interestEstimatesInCPEnabled

public boolean interestEstimatesInCPEnabled()

interestBroadcastVars

public boolean interestBroadcastVars()

interestLargestEstimateEnabled

public boolean interestLargestEstimateEnabled()

interestOutputCachingEnabled

public boolean interestOutputCachingEnabled()