gobblin.util

Class WriterUtils

java.lang.Object

gobblin.util.WriterUtils

public class WriterUtils
extends Object

Utility class for use with the DataWriter class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WriterUtils.WriterFilePathType TABLENAME should be used for jobs that pull from multiple tables/topics and intend to write the records in each table/topic to a separate folder.

Field Summary

Fields

Modifier and Type	Field and Description
static String	WRITER_ENCRYPTED_CONFIG_PATH

Constructor Summary

Constructors

Constructor and Description
WriterUtils()

Method Summary

Modifier and Type	Method and Description
static CodecFactory	getCodecFactory(Optional<String> codecName, Optional<String> deflateLevel) Creates a CodecFactory based on the specified codec name and deflate level.
static Path	getDataPublisherFinalDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given BaseDataPublisher should commits its output data.
static Path	getDefaultWriterFilePath(State state, int numBranches, int branchId) Creates the default Path for the ConfigurationKeys.WRITER_FILE_PATH key.
static Configuration	getFsConfiguration(State state)
static Path	getTableNameWriterFilePath(State state) Creates Path for the ConfigurationKeys.WRITER_FILE_PATH key according to ConfigurationKeys.EXTRACT_TABLE_NAME_KEY.
static String	getWriterFileName(State state, int numBranches, int branchId, String writerId, String formatExtension) Get the value of ConfigurationKeys.WRITER_FILE_NAME for the a given DataWriter.
static Path	getWriterFilePath(State state, int numBranches, int branchId) Get the Path corresponding the the relative file path for a given DataWriter.
static FileSystem	getWriterFs(State state)
static FileSystem	getWriterFS(State state, int numBranches, int branchId)
static Path	getWriterOutputDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given DataWriter should be writing its output data.
static Path	getWriterStagingDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given DataWriter should be writing its staging data.
static Path	getWriterStagingDir(State state, int numBranches, int branchId, String attemptId) Get the staging Path for DataWriter that has attemptId in the path.
static void	mkdirsWithRecursivePermission(FileSystem fs, Path path, FsPermission perm) Create the given dir as well as all missing ancestor dirs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WRITER_ENCRYPTED_CONFIG_PATH

public static final String WRITER_ENCRYPTED_CONFIG_PATH

See Also:

Constant Field Values

Constructor Detail

WriterUtils

public WriterUtils()

Method Detail

getWriterStagingDir

public static Path getWriterStagingDir(State state,
                                       int numBranches,
                                       int branchId)

Get the Path corresponding the to the directory a given DataWriter should be writing its staging data. The staging data directory is determined by combining the ConfigurationKeys.WRITER_STAGING_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getWriterStagingDir

public static Path getWriterStagingDir(State state,
                                       int numBranches,
                                       int branchId,
                                       String attemptId)

Get the staging Path for DataWriter that has attemptId in the path.

getWriterOutputDir

public static Path getWriterOutputDir(State state,
                                      int numBranches,
                                      int branchId)

Get the Path corresponding the to the directory a given DataWriter should be writing its output data. The output data directory is determined by combining the ConfigurationKeys.WRITER_OUTPUT_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getDataPublisherFinalDir

public static Path getDataPublisherFinalDir(State state,
                                            int numBranches,
                                            int branchId)

Get the Path corresponding the to the directory a given BaseDataPublisher should commits its output data. The final output data directory is determined by combining the ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the BaseDataPublisher will publish.

Returns:

a Path specifying the directory where the BaseDataPublisher will publish.

getWriterFilePath

public static Path getWriterFilePath(State state,
                                     int numBranches,
                                     int branchId)

Get the Path corresponding the the relative file path for a given DataWriter. This method retrieves the value of ConfigurationKeys.WRITER_FILE_PATH from the given State. It also constructs the default value of the ConfigurationKeys.WRITER_FILE_PATH if not is not specified in the given State.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the {DataWriter will write to.

Returns:

a Path specifying the relative directory where the DataWriter will write to.

getTableNameWriterFilePath

public static Path getTableNameWriterFilePath(State state)

Creates Path for the ConfigurationKeys.WRITER_FILE_PATH key according to ConfigurationKeys.EXTRACT_TABLE_NAME_KEY.

Parameters:

state -

Returns:

getDefaultWriterFilePath

public static Path getDefaultWriterFilePath(State state,
                                            int numBranches,
                                            int branchId)

Creates the default Path for the ConfigurationKeys.WRITER_FILE_PATH key.

Parameters:

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getWriterFileName

public static String getWriterFileName(State state,
                                       int numBranches,
                                       int branchId,
                                       String writerId,
                                       String formatExtension)

Get the value of ConfigurationKeys.WRITER_FILE_NAME for the a given DataWriter. The method also constructs the default value of the ConfigurationKeys.WRITER_FILE_NAME if it is not set in the State

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the {DataWriter will write to.

writerId - is the id for a specific DataWriter.

formatExtension - is the format extension for the file (e.g. ".avro").

Returns:

a String representation of the file name.

getCodecFactory

public static CodecFactory getCodecFactory(Optional<String> codecName,
                                           Optional<String> deflateLevel)

Creates a CodecFactory based on the specified codec name and deflate level. If codecName is absent, then a CodecFactory.deflateCodec(int) is returned. Otherwise the codecName is converted into a CodecFactory via the CodecFactory.fromString(String) method.

Parameters:

codecName - the name of the codec to use (e.g. deflate, snappy, xz, etc.).

deflateLevel - must be an integer from [0-9], and is only applicable if the codecName is "deflate".

Returns:

a CodecFactory.

mkdirsWithRecursivePermission

public static void mkdirsWithRecursivePermission(FileSystem fs,
                                                 Path path,
                                                 FsPermission perm)
                                          throws IOException

Create the given dir as well as all missing ancestor dirs. All created dirs will have the given permission. This should be used instead of FileSystem.mkdirs(Path, FsPermission), since that method only sets the permission for the given dir, and not recursively for the ancestor dirs.

Parameters:

fs - FileSystem

path - The dir to be created

perm - The permission to be set

Throws:

IOException - if failing to create dir or set permission.

getWriterFS

public static FileSystem getWriterFS(State state,
                                     int numBranches,
                                     int branchId)
                              throws IOException

Throws:

IOException

getWriterFs

public static FileSystem getWriterFs(State state)
                              throws IOException

Throws:

IOException

getFsConfiguration

public static Configuration getFsConfiguration(State state)