Package org.apache.sysds.runtime.io

Class ReaderTextCSVParallel

java.lang.Object

org.apache.sysds.runtime.io.MatrixReader

org.apache.sysds.runtime.io.ReaderTextCSVParallel

public class ReaderTextCSVParallel
extends MatrixReader

Parallel version of ReaderTextCSV.java. To summarize, we do two passes in order to compute row offsets and the actual read. We accordingly create count and read tasks and use fixed-size thread pools to execute these tasks. If the target matrix is dense, the inserts are done lock-free. In contrast to textcell parallel read, we also do lock-free inserts. If the matrix is sparse, because splits contain row partitioned lines and hence there is no danger of lost updates. Note, there is also no sorting of sparse rows required because data comes in sorted order per row.

Constructor Summary

Constructors

Constructor	Description
ReaderTextCSVParallel(FileFormatPropertiesCSV props)

Method Summary

Modifier and Type	Method	Description
MatrixBlock	readMatrixAsStream(OOCStream<IndexedMatrixValue> outStream, String fname, long rlen, long clen, int blen, long estnnz)
MatrixBlock	readMatrixFromHDFS(String fname, long rlen, long clen, int blen, long estnnz)
MatrixBlock	readMatrixFromInputStream(InputStream is, long rlen, long clen, int blen, long estnnz)

Methods inherited from class org.apache.sysds.runtime.io.MatrixReader

checkValidInputFile

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ReaderTextCSVParallel

public ReaderTextCSVParallel(FileFormatPropertiesCSV props)

Method Details

readMatrixFromHDFS

public MatrixBlock readMatrixFromHDFS(String fname,
 long rlen,
 long clen,
 int blen,
 long estnnz)
                               throws IOException,
DMLRuntimeException

Specified by:

readMatrixFromHDFS in class MatrixReader

Throws:

IOException

DMLRuntimeException

readMatrixAsStream

public MatrixBlock readMatrixAsStream(OOCStream<IndexedMatrixValue> outStream,
 String fname,
 long rlen,
 long clen,
 int blen,
 long estnnz)
                               throws IOException,
DMLRuntimeException

Throws:

IOException

DMLRuntimeException

readMatrixFromInputStream

public MatrixBlock readMatrixFromInputStream(InputStream is,
 long rlen,
 long clen,
 int blen,
 long estnnz)
                                      throws IOException,
DMLRuntimeException

Specified by:

readMatrixFromInputStream in class MatrixReader

Throws:

IOException

DMLRuntimeException