gobblin.util.dataset

Class DatasetUtils

java.lang.Object

gobblin.util.dataset.DatasetUtils

public class DatasetUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATASET
static String	DATASET_SPECIFIC_PROPS A configuration key that allows a user to specify config parameters on a dataset specific level.

Method Summary

Modifier and Type	Method and Description
static Map<String,State>	getDatasetSpecificProps(Iterable<String> datasets, State state) Given a Iterable of dataset identifiers (e.g., name, URN, etc.), return a Map that links each dataset with the extra configuration information specified in the state via DATASET_SPECIFIC_PROPS.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATASET

public static final String DATASET

See Also:

Constant Field Values

DATASET_SPECIFIC_PROPS

public static final String DATASET_SPECIFIC_PROPS

A configuration key that allows a user to specify config parameters on a dataset specific level. The value of this config should be a JSON array. Each entry should be a JsonObject and should contain a JsonPrimitive that identifies the dataset. All configs in each dataset entry will be added to the WorkUnits for that dataset.

An example value could be: "[{"dataset" : "myDataset1", "writer.partition.columns" : "header.memberId"}, {"dataset" : "myDataset2", "writer.partition.columns" : "auditHeader.time"}]".

The "dataset" field also allows regular expressions. For example, one can specify key, value "dataset" : "myDataset.*". In this case all datasets whose name matches the pattern "myDataset.*" will have all the specified config properties added to their WorkUnits. If more a dataset matches multiple "dataset"s then the properties from all the JsonObjects will be added to their WorkUnits.

See Also:

Constant Field Values

Method Detail

getDatasetSpecificProps

public static Map<String,State> getDatasetSpecificProps(Iterable<String> datasets,
                                                        State state)

Given a Iterable of dataset identifiers (e.g., name, URN, etc.), return a Map that links each dataset with the extra configuration information specified in the state via DATASET_SPECIFIC_PROPS.