Package org.apache.sysds.runtime.io

Class ReaderWriterFederated

java.lang.Object
org.apache.sysds.runtime.io.ReaderWriterFederated
public class ReaderWriterFederated extends Object
This class serves as the reader for federated objects. To read the files a mdt file is required. The reader is different from the other readers in the since that it does not return a MatrixBlock but a Matrix Object wrapper, containing the federated Mapping. On the Matrix Object the function isFederated() will if called read in the federated locations and instantiate the map. The reading is done through this code. This means in practice that it circumvent the other reading code. See more in: org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS() org.apache.sysds.runtime.controlprogram.caching.CacheableData.isFederated()

  • Constructor Details

    • ReaderWriterFederated

      public ReaderWriterFederated()

  • Method Details

    • read

      public static List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> read(String file, DataCharacteristics mc)
      Read a federated map from disk, It is not initialized before it is used in: org.apache.sysds.runtime.instructions.fed.InitFEDInstruction
      Parameters:
      file - The file to read (defaults to HDFS)
      mc - The data characteristics of the file, that can be read from the mtd file.
      Returns:
      A List of federatedRanges and Federated Data

    • write

      public static void write(String file, FederationMap fedMap)
      TODO add writing to each of the federated locations so that they also save their matrices. Currently this would write the federated matrix to disk only locally.
      Parameters:
      file - The file to save to, (defaults to HDFS paths)
      fedMap - The federated map to save.