Package org.apache.sysds.runtime.instructions.spark.utils

Class RDDAggregateUtils

java.lang.Object
org.apache.sysds.runtime.instructions.spark.utils.RDDAggregateUtils
public class RDDAggregateUtils extends Object
Collection of utility methods for aggregating binary block rdds. As a general policy always call stable algorithms which maintain corrections over blocks per key. The performance overhead over a simple reducebykey is roughly 7-10% and with that acceptable.

  • Constructor Details

    • RDDAggregateUtils

      public RDDAggregateUtils()

  • Method Details

    • sumStable

      public static MatrixBlock sumStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

    • sumStable

      public static MatrixBlock sumStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in)

    • sumByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

    • sumByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)

    • sumByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)

    • sumCellsByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in)

    • sumCellsByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in, int numParts)

    • aggStable

      public static MatrixBlock aggStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)
      Single block aggregation over pair rdds with corrections for numerical stability.
      Parameters:
      in - matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>
      aop - aggregate operator
      Returns:
      matrix block

    • aggStable

      public static MatrixBlock aggStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in, AggregateOperator aop)
      Single block aggregation over rdds with corrections for numerical stability.
      Parameters:
      in - matrix as JavaRDD<MatrixBlock>
      aop - aggregate operator
      Returns:
      matrix block

    • aggStableTensor

      public static TensorBlock aggStableTensor(org.apache.spark.api.java.JavaPairRDD<TensorIndexes,TensorBlock> in, AggregateOperator aop)
      Single block aggregation over pair rdds with corrections for numerical stability.
      Parameters:
      in - tensor as JavaPairRDD<TensorIndexes, TensorBlock>
      aop - aggregate operator
      Returns:
      tensor block

    • aggStableTensor

      public static TensorBlock aggStableTensor(org.apache.spark.api.java.JavaRDD<TensorBlock> in, AggregateOperator aop)
      Single block aggregation over rdds with corrections for numerical stability.
      Parameters:
      in - tensor as JavaRDD<TensorBlock>
      aop - aggregate operator
      Returns:
      tensor block

    • aggByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)

    • aggByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, boolean deepCopyCombiner)

    • aggByKeyStable

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, int numPartitions, boolean deepCopyCombiner)

    • max

      public static double max(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

    • mergeByKey

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
      Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.
      Parameters:
      in - matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>
      Returns:
      matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>

    • mergeByKey

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)
      Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.
      Parameters:
      in - matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>
      deepCopyCombiner - indicator if the createCombiner functions needs to deep copy the input block
      Returns:
      matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>

    • mergeByKey

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)
      Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.
      Parameters:
      in - matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>
      numPartitions - number of output partitions
      deepCopyCombiner - indicator if the createCombiner functions needs to deep copy the input block
      Returns:
      matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>

    • mergeRowsByKey

      public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeRowsByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,RowMatrixBlock> in)
      Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.
      Parameters:
      in - matrix as JavaPairRDD<MatrixIndexes, RowMatrixBlock>
      Returns:
      matrix as JavaPairRDD<MatrixIndexes, MatrixBlock>