Package org.apache.sysds.hops

Class Hop

java.lang.Object
org.apache.sysds.hops.Hop
All Implemented Interfaces:
ParseInfo
Direct Known Subclasses:
DataOp, IndexingOp, LeftIndexingOp, LiteralOp, MultiThreadedHop, NaryOp
public abstract class Hop extends Object implements ParseInfo
Hop is a High level operator, that is the first intermediate representation compiled from the definitions supplied in DML.

  • Field Details

    • CPThreshold

      public static final long CPThreshold
      See Also:

    • _beginLine

      public int _beginLine

    • _beginColumn

      public int _beginColumn

    • _endLine

      public int _endLine

    • _endColumn

      public int _endColumn

    • _filename

      public String _filename

    • _text

      public String _text

  • Constructor Details

  • Method Details

    • getHopID

      public long getHopID()

    • getExecType

      public Types.ExecType getExecType()

    • setExecType

      public void setExecType(Types.ExecType execType)

    • setFederatedOutput

      public void setFederatedOutput(FEDInstruction.FederatedOutput federatedOutput)

    • activatePrefetch

      public void activatePrefetch()
      Activate prefetch of HOP.

    • deactivatePrefetch

      public void deactivatePrefetch()

    • prefetchActivated

      public boolean prefetchActivated()
      Checks if prefetch is activated for this hop.
      Returns:
      true if prefetch is activated

    • resetExecType

      public void resetExecType()

    • getForcedExecType

      public Types.ExecType getForcedExecType()

    • setForcedExecType

      public void setForcedExecType(Types.ExecType etype)

    • allowsAllExecTypes

      public abstract boolean allowsAllExecTypes()

    • isTransposeSafe

      public boolean isTransposeSafe()
      Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)). Usually, this applies to aggregate operations with fixed output dimension. Finally, this information is very useful in order to safely optimize the plan for sparse vectors, which otherwise would be (currently) always represented dense.
      Returns:
      always returns false

    • checkAndSetForcedPlatform

      public void checkAndSetForcedPlatform()

    • checkAndSetInvalidCPDimsAndSize

      public void checkAndSetInvalidCPDimsAndSize()

    • hasValidCPDimsAndSize

      public boolean hasValidCPDimsAndSize()

    • setRequiresReblock

      public void setRequiresReblock(boolean flag)

    • requiresReblock

      public boolean requiresReblock()

    • setRequiresCheckpoint

      public void setRequiresCheckpoint(boolean flag)

    • requiresCheckpoint

      public boolean requiresCheckpoint()

    • setRequiresCompression

      public void setRequiresCompression()

    • setRequiresCompression

      public void setRequiresCompression(AWTreeNode node)

    • setRequiresDeCompression

      public void setRequiresDeCompression()

    • isRequiredDecompression

      public boolean isRequiredDecompression()

    • requiresCompression

      public boolean requiresCompression()

    • setCompressedOutput

      public void setCompressedOutput(boolean value)

    • setCompressedSize

      public void setCompressedSize(long size)

    • getCompressedSize

      public long getCompressedSize()

    • isCompressedOutput

      public boolean isCompressedOutput()

    • hasCompressedInput

      public boolean hasCompressedInput()

    • compressedSize

      public long compressedSize()

    • setRequiresLineageCaching

      public void setRequiresLineageCaching(boolean flag)

    • requiresLineageCaching

      public boolean requiresLineageCaching()

    • updateLopFedOut

      public void updateLopFedOut(Lop lop)

    • updateLopFedOut

      public void updateLopFedOut(Lop lop, Types.ExecType execType, FEDInstruction.FederatedOutput fedOut)

    • constructAndSetLopsDataFlowProperties

      public void constructAndSetLopsDataFlowProperties()

    • createOffsetLop

      public static Lop createOffsetLop(Hop hop, boolean repCols)

    • setOutputEmptyBlocks

      public void setOutputEmptyBlocks(boolean flag)

    • isOutputEmptyBlocks

      public boolean isOutputEmptyBlocks()

    • getInputOutputSize

      public double getInputOutputSize(Collection<String> exclVars)

    • getMemEstimate

      public double getMemEstimate()
      NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table. * Invocation: Intended to be called for ALL root nodes of one Hops DAG with the same (initially empty) memo table.
      Returns:
      memory estimate in bytes

    • setMemEstimate

      public void setMemEstimate(double mem)
      Sets memory estimate in bytes
      Parameters:
      mem - memory estimate

    • clearMemEstimate

      public void clearMemEstimate()

    • isMemEstimated

      public boolean isMemEstimated()

    • getInputMemEstimate

      public double getInputMemEstimate()
      Get the memory estimate of inputs as the sum of input estimates in bytes.
      Returns:
      input memory estimate in bytes

    • getInputMemEstimate

      public double getInputMemEstimate(double injectedDefault)
      Get the memory estimate of inputs as the sum of input estimates in bytes.
      Parameters:
      injectedDefault - default memory estimate (bytes) used when the memory estimate of the input is negative
      Returns:
      input memory estimate in bytes

    • getOutputMemEstimate

      public double getOutputMemEstimate()
      Output memory estimate in bytes.
      Returns:
      output memory estimate in bytes

    • getOutputMemEstimate

      public double getOutputMemEstimate(double injectedDefault)
      Output memory estimate in bytes with negative memory estimates replaced by the injected default. The injected default represents the memory estimate per output cell, hence it is multiplied by the estimated dimensions of the output of the hop.
      Parameters:
      injectedDefault - memory estimate to be returned in case the memory estimate defaults to a negative number
      Returns:
      output memory estimate in bytes

    • getIntermediateMemEstimate

      public double getIntermediateMemEstimate()

    • getSpBroadcastSize

      public double getSpBroadcastSize()

    • computeMemEstimate

      public void computeMemEstimate(MemoTable memo)
      Computes the estimate of memory required to store the input/output of this hop in memory. This is the default implementation (orchestration of hop-specific implementation) that should suffice for most hops. If a hop requires more control, this method should be overwritten with awareness of (1) output estimates, and (2) propagation of worst-case matrix characteristics (dimensions, sparsity). TODO remove memo table and, on constructor refresh, inference in refresh, single compute mem, maybe general computeMemEstimate, flags to indicate if estimate or not.
      Parameters:
      memo - memory table

    • refreshMemEstimates

      public void refreshMemEstimates(MemoTable memo)
      Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by this.
      Parameters:
      memo - memory table

    • isFederated

      public boolean isFederated()
      Checks if ExecType is federated.
      Returns:
      true if ExecType is federated

    • someInputFederated

      public boolean someInputFederated()

    • isFederatedDataOp

      public boolean isFederatedDataOp()
      Checks if the hop is a DataOp with federated data.
      Returns:
      true if hop is a federated DataOp

    • getParent

      public List<Hop> getParent()

    • getInput

      public List<Hop> getInput()

    • getInput

      public Hop getInput(int ix)

    • addInput

      public void addInput(Hop h)

    • addAllInputs

      public void addAllInputs(List<Hop> list)

    • getBlocksize

      public int getBlocksize()

    • setBlocksize

      public void setBlocksize(int blen)

    • setNnz

      public void setNnz(long nnz)

    • getNnz

      public long getNnz()

    • getFederatedOutput

      public FEDInstruction.FederatedOutput getFederatedOutput()

    • hasFederatedOutput

      public boolean hasFederatedOutput()

    • hasLocalOutput

      public boolean hasLocalOutput()

    • setUpdateType

      public void setUpdateType(MatrixObject.UpdateType update)

    • getUpdateType

      public MatrixObject.UpdateType getUpdateType()

    • constructLops

      public abstract Lop constructLops()

    • getOpString

      public abstract String getOpString()

    • toString

      public final String toString()
      Overrides:
      toString in class Object

    • isGPUEnabled

      public abstract boolean isGPUEnabled()
      In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).
      Returns:
      true if the Hop is eligible for GPU Exectype.

    • dimsKnown

      public boolean dimsKnown()

    • dimsKnown

      public boolean dimsKnown(boolean includeNnz)

    • dimsKnownAny

      public boolean dimsKnownAny()

    • rowsKnown

      public boolean rowsKnown()

    • colsKnown

      public boolean colsKnown()

    • resetVisitStatus

      public static void resetVisitStatus(List<Hop> hops)

    • resetVisitStatus

      public static void resetVisitStatus(List<Hop> hops, boolean force)

    • resetVisitStatus

      public Hop resetVisitStatus()

    • resetVisitStatusForced

      public void resetVisitStatusForced(HashSet<Long> memo)

    • resetRecompilationFlag

      public static void resetRecompilationFlag(ArrayList<Hop> hops, Types.ExecType et, Recompiler.ResetType reset)

    • resetRecompilationFlag

      public static void resetRecompilationFlag(Hop hops, Types.ExecType et, Recompiler.ResetType reset)

    • getDim1

      public long getDim1()
      Get the number of rows in the Hop.
      Returns:
      a long.

    • setDim1

      public void setDim1(long dim1)

    • getDim2

      public long getDim2()
      Get the number of columns in the Hop.
      Returns:
      a long.

    • setDim2

      public void setDim2(long dim2)

    • getDim

      public long getDim(int i)

    • setDim

      public void setDim(int i, long dim)

    • getLength

      public long getLength()

    • getSparsity

      public double getSparsity()

    • getDataCharacteristics

      public DataCharacteristics getDataCharacteristics()

    • getLops

      public Lop getLops()

    • setLops

      public void setLops(Lop lops)

    • isVisited

      public boolean isVisited()

    • getDataType

      public Types.DataType getDataType()

    • setDataType

      public void setDataType(Types.DataType dt)

    • isScalar

      public boolean isScalar()

    • isMatrix

      public boolean isMatrix()

    • setVisited

      public void setVisited()

    • setVisited

      public void setVisited(boolean flag)

    • setName

      public void setName(String _name)

    • getName

      public String getName()

    • getValueType

      public Types.ValueType getValueType()

    • setValueType

      public void setValueType(Types.ValueType vt)

    • requiresRecompile

      public boolean requiresRecompile()
      Indicates if dynamic recompilation is required for this hop.
      Returns:
      true if dynamic recompilation required

    • setRequiresRecompile

      public void setRequiresRecompile()
      Marks the hop for dynamic recompilation.

    • refreshSizeInformation

      public abstract void refreshSizeInformation()
      Update the output size information for this hop.

    • computeSizeInformation

      public static long computeSizeInformation(Hop input)

    • refreshRowsParameterInformation

      public void refreshRowsParameterInformation(Hop input, LocalVariableMap vars)

    • refreshRowsParameterInformation

      public void refreshRowsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

    • refreshColsParameterInformation

      public void refreshColsParameterInformation(Hop input, LocalVariableMap vars)

    • refreshColsParameterInformation

      public void refreshColsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

    • computeSizeInformation

      public long computeSizeInformation(Hop input, LocalVariableMap vars)

    • computeSizeInformation

      public long computeSizeInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

    • computeBoundsInformation

      public double computeBoundsInformation(Hop input)

    • computeBoundsInformation

      public static double computeBoundsInformation(Hop input, LocalVariableMap vars)

    • computeBoundsInformation

      public static double computeBoundsInformation(Hop input, LocalVariableMap vars, Map<Long,Double> memo)

    • clone

      public abstract Object clone() throws CloneNotSupportedException
      Throws:
      CloneNotSupportedException

    • compare

      public abstract boolean compare(Hop that)

    • setBeginLine

      public void setBeginLine(int passed)
      Specified by:
      setBeginLine in interface ParseInfo

    • setBeginColumn

      public void setBeginColumn(int passed)
      Specified by:
      setBeginColumn in interface ParseInfo

    • setEndLine

      public void setEndLine(int passed)
      Specified by:
      setEndLine in interface ParseInfo

    • setEndColumn

      public void setEndColumn(int passed)
      Specified by:
      setEndColumn in interface ParseInfo

    • setFilename

      public void setFilename(String passed)
      Specified by:
      setFilename in interface ParseInfo

    • setText

      public void setText(String text)
      Specified by:
      setText in interface ParseInfo

    • getBeginLine

      public int getBeginLine()
      Specified by:
      getBeginLine in interface ParseInfo

    • getBeginColumn

      public int getBeginColumn()
      Specified by:
      getBeginColumn in interface ParseInfo

    • getEndLine

      public int getEndLine()
      Specified by:
      getEndLine in interface ParseInfo

    • getEndColumn

      public int getEndColumn()
      Specified by:
      getEndColumn in interface ParseInfo

    • getFilename

      public String getFilename()
      Specified by:
      getFilename in interface ParseInfo

    • getText

      public String getText()
      Specified by:
      getText in interface ParseInfo

    • printErrorLocation

      public String printErrorLocation()

    • setParseInfo

      public void setParseInfo(ParseInfo parseInfo)
      Set parse information.
      Parameters:
      parseInfo - parse information, such as beginning line position, beginning column position, ending line position, ending column position, text, and filename