Package cz.o2.proxima.direct.io.bulkfs.parquet

Class ParquetFileFormat

java.lang.Object

cz.o2.proxima.direct.io.bulkfs.parquet.ParquetFileFormat

All Implemented Interfaces:

FileFormat, java.io.Serializable

@Internal
public class ParquetFileFormat
extends java.lang.Object
implements FileFormat

Bulk file format which stored data in Parquet format.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	PARQUET_CONFIG_COMPRESSION_KEY_NAME	Config key specified compression used in parquet writer.
static java.lang.String	PARQUET_CONFIG_PAGE_SIZE_KEY_NAME	Config key specified page and row group size in parquet writer.
static java.lang.String	PARQUET_CONFIG_VALUES_PREFIX_KEY_NAME	Config key used for attribute prefix - useful when attribute name collides with proxima fields defined as PARQUET_COLUMN_NAME_*
static int	PARQUET_DEFAULT_PAGE_SIZE	Default page size used for parquet writer
static java.lang.String	PARQUET_DEFAULT_VALUES_NAME_PREFIX	Default attribute prefix.
static java.lang.String	PARQUET_MESSAGE_NAME	Name for top level message in parquet schema.

Constructor Summary

Constructors

Constructor	Description
ParquetFileFormat()

Method Summary

Modifier and Type	Method	Description
java.lang.String	fileSuffix()	Retrieve suffix of file names.
Reader	openReader(Path path, EntityDescriptor entity)	Open reader for data stored at given Path
Writer	openWriter(Path path, EntityDescriptor entity)	Open writer for data on given Path.
void	setup(AttributeFamilyDescriptor family)	Setup FileFormat for given AttributeFamilyDescriptor.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_CONFIG_COMPRESSION_KEY_NAME

public static final java.lang.String PARQUET_CONFIG_COMPRESSION_KEY_NAME

Config key specified compression used in parquet writer.

See Also:

Constant Field Values

PARQUET_CONFIG_PAGE_SIZE_KEY_NAME

public static final java.lang.String PARQUET_CONFIG_PAGE_SIZE_KEY_NAME

Config key specified page and row group size in parquet writer.

See Also:

Constant Field Values

PARQUET_CONFIG_VALUES_PREFIX_KEY_NAME

public static final java.lang.String PARQUET_CONFIG_VALUES_PREFIX_KEY_NAME

Config key used for attribute prefix - useful when attribute name collides with proxima fields defined as PARQUET_COLUMN_NAME_*

See Also:

Constant Field Values

PARQUET_MESSAGE_NAME

public static final java.lang.String PARQUET_MESSAGE_NAME

Name for top level message in parquet schema.

See Also:

Constant Field Values

PARQUET_DEFAULT_PAGE_SIZE

public static final int PARQUET_DEFAULT_PAGE_SIZE

Default page size used for parquet writer

See Also:

Constant Field Values

PARQUET_DEFAULT_VALUES_NAME_PREFIX

public static final java.lang.String PARQUET_DEFAULT_VALUES_NAME_PREFIX

Default attribute prefix. See PARQUET_CONFIG_VALUES_PREFIX_KEY_NAME for details.

See Also:

Constant Field Values

Constructor Detail

ParquetFileFormat

public ParquetFileFormat()

Method Detail

setup

public void setup(AttributeFamilyDescriptor family)

Description copied from interface: FileFormat

Setup FileFormat for given AttributeFamilyDescriptor.

Specified by:

setup in interface FileFormat

Parameters:

family - attribute family

openReader

public Reader openReader(Path path,
                         EntityDescriptor entity)
                  throws java.io.IOException

Description copied from interface: FileFormat

Open reader for data stored at given Path

Specified by:

openReader in interface FileFormat

Parameters:

path - Path on associated FileSystem

entity - descriptor of entity whose data we are going to read

Returns:

reader of the data

Throws:

java.io.IOException - on errors

openWriter

public Writer openWriter(Path path,
                         EntityDescriptor entity)
                  throws java.io.IOException

Description copied from interface: FileFormat

Open writer for data on given Path.

Specified by:

openWriter in interface FileFormat

Parameters:

path - Path on associated FileSystem

entity - descriptor of entity whose data we are going to write

Returns:

writer for the data

Throws:

java.io.IOException - on errors

fileSuffix

public java.lang.String fileSuffix()

Description copied from interface: FileFormat

Retrieve suffix of file names.

Specified by:

fileSuffix in interface FileFormat

Returns:

String suffix