gobblin.util

Class AvroUtils

java.lang.Object

gobblin.util.AvroUtils

public class AvroUtils
extends Object

A Utils class for dealing with Avro objects

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroUtils.AvroPathFilter

Field Summary

Fields

Modifier and Type	Field and Description
static String	FIELD_LOCATION_DELIMITER

Constructor Summary

Constructors

Constructor and Description
AvroUtils()

Method Summary

Modifier and Type	Method and Description
static GenericRecord	convertRecordSchema(GenericRecord record, Schema newSchema) Change the schema of an Avro record.
static Schema	getDirectorySchema(Path directory, Configuration conf, boolean latest) Get the latest avro schema for a directory
static Schema	getDirectorySchema(Path directory, FileSystem fs, boolean latest) Get the latest avro schema for a directory
static Optional<Schema.Field>	getField(Schema schema, String fieldLocation) Given a GenericRecord, this method will return the field specified by the path parameter.
static Optional<Schema>	getFieldSchema(Schema schema, String fieldLocation) Given a GenericRecord, this method will return the schema of the field specified by the path parameter.
static Optional<Object>	getFieldValue(GenericRecord record, String fieldLocation) Given a GenericRecord, this method will return the field specified by the path parameter.
static Map<String,Object>	getMultiFieldValue(GenericRecord record, String fieldLocation)
static Schema	getSchemaFromDataFile(Path dataFile, FileSystem fs) Get Avro schema from an Avro data file.
static Schema	nullifyFieldsForSchemaMerge(Schema oldSchema, Schema newSchema) Merge oldSchema and newSchame.
static Schema	parseSchemaFromFile(Path filePath, FileSystem fs) Parse Avro schema from a schema file.
static byte[]	recordToByteArray(GenericRecord record) Convert a GenericRecord to a byte array.
static Optional<Schema>	removeUncomparableFields(Schema schema) Remove map, array, enum fields, as well as union fields that contain map, array or enum, from an Avro schema.
static Path	serializeAsPath(GenericRecord record, boolean includeFieldNames, boolean replacePathSeparators) Serialize a generic record as a relative Path.
static GenericRecord	slowDeserializeGenericRecord(byte[] serializedRecord, Schema schema) Deserialize a GenericRecord from a byte array.
static Schema	switchName(Schema schema, String newName) Copies the input Schema but changes the schema name.
static Map<String,String>	toStringMap(Object map) Given a map: key -> value, return a map: key.toString() -> value.toString().
static void	writeSchemaToFile(Schema schema, Path filePath, FileSystem fs, boolean overwrite)
static void	writeSchemaToFile(Schema schema, Path filePath, FileSystem fs, boolean overwrite, FsPermission perm)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FIELD_LOCATION_DELIMITER

public static final String FIELD_LOCATION_DELIMITER

See Also:

Constant Field Values

Constructor Detail

AvroUtils

public AvroUtils()

Method Detail

getFieldSchema

public static Optional<Schema> getFieldSchema(Schema schema,
                                              String fieldLocation)

Given a GenericRecord, this method will return the schema of the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes the the schema of the field "field1", and retrieves the schema "nestedField1" from it.

Parameters:

schema - is the record to retrieve the schema from

fieldLocation - is the location of the field

Returns:

the schema of the field

getField

public static Optional<Schema.Field> getField(Schema schema,
                                              String fieldLocation)

Given a GenericRecord, this method will return the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes field "field1", and retrieves "nestedField1" from it.

Parameters:

schema - is the record to retrieve the schema from

fieldLocation - is the location of the field

Returns:

the field

getFieldValue

public static Optional<Object> getFieldValue(GenericRecord record,
                                             String fieldLocation)

Given a GenericRecord, this method will return the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes the the value of the field "field1", and retrieves the field "nestedField1" from it.

Parameters:

record - is the record to retrieve the field from

fieldLocation - is the location of the field

Returns:

the value of the field

getMultiFieldValue

public static Map<String,Object> getMultiFieldValue(GenericRecord record,
                                                    String fieldLocation)

toStringMap

public static Map<String,String> toStringMap(Object map)

Given a map: key -> value, return a map: key.toString() -> value.toString(). Avro serializer wraps a String into Utf8. This method helps to restore the original string map object

Parameters:

map - a map object

Returns:

a map of strings

convertRecordSchema

public static GenericRecord convertRecordSchema(GenericRecord record,
                                                Schema newSchema)
                                         throws IOException

Change the schema of an Avro record.

Parameters:

record - The Avro record whose schema is to be changed.

newSchema - The target schema. It must be compatible as reader schema with record.getSchema() as writer schema.

Returns:

a new Avro record with the new schema.

Throws:

IOException - if conversion failed.

recordToByteArray

public static byte[] recordToByteArray(GenericRecord record)
                                throws IOException

Convert a GenericRecord to a byte array.

Throws:

IOException

getSchemaFromDataFile

public static Schema getSchemaFromDataFile(Path dataFile,
                                           FileSystem fs)
                                    throws IOException

Get Avro schema from an Avro data file.

Throws:

IOException

parseSchemaFromFile

public static Schema parseSchemaFromFile(Path filePath,
                                         FileSystem fs)
                                  throws IOException

Parse Avro schema from a schema file.

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(Schema schema,
                                     Path filePath,
                                     FileSystem fs,
                                     boolean overwrite)
                              throws IOException

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(Schema schema,
                                     Path filePath,
                                     FileSystem fs,
                                     boolean overwrite,
                                     FsPermission perm)
                              throws IOException

Throws:

IOException

getDirectorySchema

public static Schema getDirectorySchema(Path directory,
                                        FileSystem fs,
                                        boolean latest)
                                 throws IOException

Get the latest avro schema for a directory

Parameters:

directory - the input dir that contains avro files

fs - the FileSystem for the given directory.

latest - true to return latest schema, false to return oldest schema

Returns:

the latest/oldest schema in the directory

Throws:

IOException

getDirectorySchema

public static Schema getDirectorySchema(Path directory,
                                        Configuration conf,
                                        boolean latest)
                                 throws IOException

Get the latest avro schema for a directory

Parameters:

directory - the input dir that contains avro files

conf - configuration

latest - true to return latest schema, false to return oldest schema

Returns:

the latest/oldest schema in the directory

Throws:

IOException

nullifyFieldsForSchemaMerge

public static Schema nullifyFieldsForSchemaMerge(Schema oldSchema,
                                                 Schema newSchema)

Merge oldSchema and newSchame. Set a field default value to null, if this field exists in the old schema but not in the new schema.

Parameters:

oldSchema -

newSchema -

Returns:

schema that contains all the fields in both old and new schema.

removeUncomparableFields

public static Optional<Schema> removeUncomparableFields(Schema schema)

Remove map, array, enum fields, as well as union fields that contain map, array or enum, from an Avro schema. A schema with these fields cannot be used as Mapper key in a MapReduce job.

switchName

public static Schema switchName(Schema schema,
                                String newName)

Copies the input Schema but changes the schema name.

Parameters:

schema - Schema to copy.

newName - name for the copied Schema.

Returns:

A Schema that is a copy of schema, but has the name newName.

serializeAsPath

public static Path serializeAsPath(GenericRecord record,
                                   boolean includeFieldNames,
                                   boolean replacePathSeparators)

Serialize a generic record as a relative Path. Useful for converting GenericRecord type keys into file system locations. For example {field1=v1, field2=v2} returns field1=v1/field2=v2 if includeFieldNames is true, or v1/v2 if it is false. Illegal HDFS tokens such as ':' and '\\' will be replaced with '_'. Additionally, parameter replacePathSeparators controls whether to replace path separators ('/') with '_'.

Parameters:

record - GenericRecord to serialize.

includeFieldNames - If true, each token in the path will be of the form key=value, otherwise, only the value will be included.

replacePathSeparators - If true, path separators ('/') in each token will be replaced with '_'.

Returns:

A relative path where each level is a field in the input record.

slowDeserializeGenericRecord

public static GenericRecord slowDeserializeGenericRecord(byte[] serializedRecord,
                                                         Schema schema)
                                                  throws IOException

Deserialize a GenericRecord from a byte array. This method is not intended for high performance.

Throws:

IOException