gobblin.util.recordcount

Class CompactionRecordCountProvider

java.lang.Object

gobblin.util.RecordCountProvider

gobblin.util.recordcount.CompactionRecordCountProvider

public class CompactionRecordCountProvider
extends RecordCountProvider

Implementation of RecordCountProvider, which provides record count from file path. The file name should follow the pattern: {Prefix}{RecordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{SUFFIX}. The prefix should be either M_OUTPUT_FILE_PREFIX or MR_OUTPUT_FILE_PREFIX. For example, given a file path: "/a/b/c/part-m-123.1444437036.12345.avro", the record count will be 123.

Field Summary

Fields

Modifier and Type	Field and Description
static String	M_OUTPUT_FILE_PREFIX
static String	MR_OUTPUT_FILE_PREFIX

Constructor Summary

Constructors

Constructor and Description
CompactionRecordCountProvider()

Method Summary

Modifier and Type	Method and Description
static String	constructFileName(String filenamePrefix, long recordCount) Construct the file name as {filenamePrefix}{recordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{SUFFIX}.
Path	convertPath(Path path, RecordCountProvider src) This method currently supports converting the given Path from IngestionRecordCountProvider.
long	getRecordCount(Path filepath) Get the record count through filename.

Methods inherited from class gobblin.util.RecordCountProvider

getNotImplementedException, getRecordCount

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MR_OUTPUT_FILE_PREFIX

public static final String MR_OUTPUT_FILE_PREFIX

See Also:

Constant Field Values

M_OUTPUT_FILE_PREFIX

public static final String M_OUTPUT_FILE_PREFIX

See Also:

Constant Field Values

Constructor Detail

CompactionRecordCountProvider

public CompactionRecordCountProvider()

Method Detail

constructFileName

public static String constructFileName(String filenamePrefix,
                                       long recordCount)

Construct the file name as {filenamePrefix}{recordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{SUFFIX}.

getRecordCount

public long getRecordCount(Path filepath)

Get the record count through filename.

Specified by:

getRecordCount in class RecordCountProvider

convertPath

public Path convertPath(Path path,
                        RecordCountProvider src)

This method currently supports converting the given Path from IngestionRecordCountProvider. The converted Path will start with M_OUTPUT_FILE_PREFIX.

Overrides:

convertPath in class RecordCountProvider