Package org.apache.sysds.runtime.io

Class MatrixWriter

java.lang.Object

org.apache.sysds.runtime.io.MatrixWriter

Direct Known Subclasses:

WriterBinaryBlock, WriterCompressed, WriterDelta, WriterHDF5, WriterMatrixMarket, WriterTextCell, WriterTextCSV, WriterTextLIBSVM

public abstract class MatrixWriter
extends Object

Base class for all format-specific matrix writers. Every writer is required to implement the basic write functionality but might provide additional custom functionality. Any non-default parameters (e.g., CSV read properties) should be passed into custom constructors. There is also a factory for creating format-specific writers.

Constructor Summary

Constructors

Constructor	Description
MatrixWriter()

Method Summary

Modifier and Type	Method	Description
static MatrixBlock[]	createMatrixBlocksForReuse(long rlen, long clen, int blen, boolean sparse, long nonZeros)
static MatrixBlock	getMatrixBlockForReuse(MatrixBlock[] blocks, int rows, int cols, int blen)
void	setForcedParallel(boolean par)
abstract void	writeEmptyMatrixToHDFS(String fname, long rlen, long clen, int blen)	Writes a minimal entry to represent an empty matrix on hdfs.
abstract long	writeMatrixFromStream(String fname, OOCStream<IndexedMatrixValue> stream, long rlen, long clen, int blen)	Consumes an out-of-core stream of matrix blocks and writes them to the target output path.
void	writeMatrixToHDFS(MatrixBlock src, String fname, long rlen, long clen, int blen, long nnz)
abstract void	writeMatrixToHDFS(MatrixBlock src, String fname, long rlen, long clen, int blen, long nnz, boolean diag)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MatrixWriter

public MatrixWriter()

Method Details

writeMatrixToHDFS

public void writeMatrixToHDFS(MatrixBlock src,
 String fname,
 long rlen,
 long clen,
 int blen,
 long nnz)
                       throws IOException

Throws:

IOException

writeMatrixToHDFS

public abstract void writeMatrixToHDFS(MatrixBlock src,
 String fname,
 long rlen,
 long clen,
 int blen,
 long nnz,
 boolean diag)
                                throws IOException

Throws:

IOException

writeMatrixFromStream

public abstract long writeMatrixFromStream(String fname,
 OOCStream<IndexedMatrixValue> stream,
 long rlen,
 long clen,
 int blen)
                                    throws IOException

Consumes an out-of-core stream of matrix blocks and writes them to the target output path. Implementations may choose single-file or multipart output depending on format and parallelism.

Parameters:

fname - The target output filename

stream - The OOC stream of matrix blocks to consume

rlen - The total number of rows in the matrix

clen - The total number of columns in the matrix

blen - The block size

Throws:

IOException - if an I/O error occurs

setForcedParallel

public void setForcedParallel(boolean par)

writeEmptyMatrixToHDFS

public abstract void writeEmptyMatrixToHDFS(String fname,
 long rlen,
 long clen,
 int blen)
                                     throws IOException

Writes a minimal entry to represent an empty matrix on hdfs.

Parameters:

fname - file name

rlen - number of rows

clen - number of columns

blen - number of rows/cols in block

Throws:

IOException - if IOException occurs

createMatrixBlocksForReuse

public static MatrixBlock[] createMatrixBlocksForReuse(long rlen,
 long clen,
 int blen,
 boolean sparse,
 long nonZeros)

getMatrixBlockForReuse

public static MatrixBlock getMatrixBlockForReuse(MatrixBlock[] blocks,
 int rows,
 int cols,
 int blen)