Package org.apache.sysds.hops.recompile

Class Recompiler

java.lang.Object
org.apache.sysds.hops.recompile.Recompiler
public class Recompiler extends Object
Dynamic recompilation of hop dags to runtime instructions, which includes the following substeps: (1) deep copy hop dag, (2) refresh matrix characteristics, (3) apply dynamic rewrites, (4) refresh memory estimates, (5) construct lops (incl operator selection), and (6) generate runtime program (incl piggybacking).

  • Constructor Details

    • Recompiler

      public Recompiler()

  • Method Details

    • reinitRecompiler

      public static void reinitRecompiler()
      Re-initializes the recompiler according to the current optimizer flags.

    • recompileHopsDag

      public static ArrayList<Instruction> recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, ExecutionContext ec, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)

    • recompileHopsDag

      public static ArrayList<Instruction> recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)

    • recompileHopsDag

      public static ArrayList<Instruction> recompileHopsDag(Hop hop, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)

    • recompileHopsDag2Forced

      public static ArrayList<Instruction> recompileHopsDag2Forced(StatementBlock sb, ArrayList<Hop> hops, long tid, Types.ExecType et)

    • recompileHopsDag2Forced

      public static ArrayList<Instruction> recompileHopsDag2Forced(Hop hop, long tid, Types.ExecType et)

    • recompileHopsDagInstructions

      public static ArrayList<Instruction> recompileHopsDagInstructions(StatementBlock sb, ArrayList<Hop> hops)

    • recompileHopsDagInstructions

      public static ArrayList<Instruction> recompileHopsDagInstructions(Hop hop)

    • recompile

      public static ArrayList<Instruction> recompile(StatementBlock sb, ArrayList<Hop> hops, ExecutionContext ec, RecompileStatus status, boolean inplace, boolean replaceLit, boolean updateStats, boolean forceEt, boolean pred, Types.ExecType et, long tid)
      Core internal primitive for the dynamic recompilation of any DAGs/predicate, including all variants with slightly different configurations.
      Parameters:
      sb - statement block of DAG, null for predicates
      hops - list of DAG root nodes
      ec - Execution context
      status - recompilation status
      inplace - modify DAG in place, otherwise deep copy
      replaceLit - replace literals (only applicable on deep copy)
      updateStats - update statistics, rewrites, and memory estimates
      forceEt - force a given execution type, null for reset
      pred - recompile for predicate DAG
      et - given execution type
      tid - thread id, 0 for main or before worker creation
      Returns:
      modified list of instructions

    • recompileProgramBlockHierarchy

      public static void recompileProgramBlockHierarchy(List<ProgramBlock> pbs, LocalVariableMap vars, long tid, boolean inplace, Recompiler.ResetType resetRecompile)

    • recompileProgramBlockHierarchy2Forced

      public static void recompileProgramBlockHierarchy2Forced(ArrayList<ProgramBlock> pbs, long tid, Set<String> fnStack, Types.ExecType et)
      Method to recompile program block hierarchy to forced execution time. This affects also referenced functions and chains of functions. Use et==null in order to release the forced exec type.
      Parameters:
      pbs - list of program blocks
      tid - thread id
      fnStack - function stack
      et - execution type

    • recompileProgramBlockInstructions

      public static void recompileProgramBlockInstructions(ProgramBlock pb) throws IOException
      This method does NO full program block recompile (no stats update, no rewrites, no recursion) but only regenerates lops and instructions. The primary use case is recompilation after are hop configuration changes which allows to preserve statistics (e.g., propagated worst case stats from other program blocks) and better performance for recompiling individual program blocks.
      Parameters:
      pb - program block
      Throws:
      IOException - if IOException occurs

    • requiresRecompilation

      public static boolean requiresRecompilation(ArrayList<Hop> hops)

    • requiresRecompilation

      public static boolean requiresRecompilation(Hop hop)

    • deepCopyHopsDag

      public static ArrayList<Hop> deepCopyHopsDag(List<Hop> hops)
      Deep copy of hops dags for parallel recompilation.
      Parameters:
      hops - list of high-level operators
      Returns:
      list of high-level operators

    • deepCopyHopsDag

      public static Hop deepCopyHopsDag(Hop hops)
      Deep copy of hops dags for parallel recompilation.
      Parameters:
      hops - high-level operator
      Returns:
      high-level operator

    • updateFunctionNames

      public static void updateFunctionNames(List<Hop> hops, long pid)

    • rUpdateFunctionNames

      public static void rUpdateFunctionNames(Hop hop, long pid)

    • reconcileUpdatedCallVarsLoops

      public static boolean reconcileUpdatedCallVarsLoops(LocalVariableMap oldCallVars, LocalVariableMap callVars, StatementBlock sb)

    • reconcileUpdatedCallVarsLoops

      public static boolean reconcileUpdatedCallVarsLoops(RecompileStatus oldCallStatus, RecompileStatus callStatus, StatementBlock sb)

    • reconcileUpdatedCallVarsIf

      public static LocalVariableMap reconcileUpdatedCallVarsIf(LocalVariableMap oldCallVars, LocalVariableMap callVarsIf, LocalVariableMap callVarsElse, StatementBlock sb)

    • reconcileUpdatedCallVarsIf

      public static RecompileStatus reconcileUpdatedCallVarsIf(RecompileStatus oldStatus, RecompileStatus callStatusIf, RecompileStatus callStatusElse, StatementBlock sb)

    • rRecompileProgramBlock2Forced

      public static void rRecompileProgramBlock2Forced(ProgramBlock pb, long tid, Set<String> fnStack, Types.ExecType et)

    • removeUpdatedScalars

      public static void removeUpdatedScalars(LocalVariableMap callVars, StatementBlock sb)
      Remove any scalar variables from the variable map if the variable is updated in this block.
      Parameters:
      callVars - Map of variables eligible for propagation.
      sb - DML statement block.

    • extractDAGOutputStatistics

      public static void extractDAGOutputStatistics(List<Hop> hops, LocalVariableMap vars)

    • extractDAGOutputStatistics

      public static void extractDAGOutputStatistics(List<Hop> hops, LocalVariableMap vars, boolean overwrite)

    • extractDAGOutputStatistics

      public static void extractDAGOutputStatistics(Hop hop, LocalVariableMap vars, boolean overwrite)

    • rClearLops

      public static void rClearLops(Hop hop)
      Clearing lops for a given hops includes to (1) remove the reference to constructed lops and (2) clear the exec type (for consistency). The latter is important for advanced optimizers like parfor; otherwise subtle side-effects of program recompilation and hop-lop rewrites possible (e.g., see indexingop hop-lop rewrite in combination parfor rewrite set exec type that eventuelly might lead to unnecessary remote_parfor jobs).
      Parameters:
      hop - high-level operator

    • rUpdateStatistics

      public static void rUpdateStatistics(Hop hop, LocalVariableMap vars)

    • rReplaceLiterals

      public static void rReplaceLiterals(Hop hop, ExecutionContext ec, boolean scalarsOnly)
      public interface to package local literal replacement
      Parameters:
      hop - high-level operator
      ec - Execution context
      scalarsOnly - if true, replace only scalar variables but no matrix operations; if false, apply full literal replacement

    • rReplaceLiterals

      public static void rReplaceLiterals(Hop hop, LocalVariableMap vars, boolean scalarsOnly)

    • rSetExecType

      public static void rSetExecType(Hop hop, Types.ExecType etype)

    • rGetMaxParallelism

      public static int rGetMaxParallelism(List<Hop> hops)

    • rGetMaxParallelism

      public static int rGetMaxParallelism(Hop hop)

    • rSetMaxParallelism

      public static void rSetMaxParallelism(List<Hop> hops, int k)

    • rSetMaxParallelism

      public static void rSetMaxParallelism(Hop hop, int k)

    • recompileFunctionOnceIfNeeded

      public static void recompileFunctionOnceIfNeeded(boolean recompileOnce, List<ProgramBlock> childBlocks, long tid, boolean inplace, Recompiler.ResetType reset, ExecutionContext ec)

    • checkCPReblock

      public static boolean checkCPReblock(ExecutionContext ec, String varin)
      CP Reblock check for spark instructions; in contrast to MR, we can not rely on the input file sizes because inputs might be passed via rdds.
      Parameters:
      ec - execution context
      varin - variable
      Returns:
      true if CP reblock?

    • checkCPCheckpoint

      public static boolean checkCPCheckpoint(DataCharacteristics dc)

    • executeInMemoryReblock

      public static void executeInMemoryReblock(ExecutionContext ec, String varin, String varout)

    • executeInMemoryReblock

      public static void executeInMemoryReblock(ExecutionContext ec, String varin, String varout, LineageItem litem)