co.cask.hydrator.common.spark

Class SparkUtils

java.lang.Object

co.cask.hydrator.common.spark.SparkUtils

public final class SparkUtils
extends Object

Spark plugin Utility class. Contains common code to be used in trainer and predictors.

Method Summary

Methods

Modifier and Type	Method and Description
static co.cask.cdap.api.data.format.StructuredRecord.Builder	cloneRecord(co.cask.cdap.api.data.format.StructuredRecord record, co.cask.cdap.api.data.schema.Schema outputSchema, String predictionField) Creates a builder based off the given record.
static Map<Integer,Integer>	getCategoricalFeatureInfo(co.cask.cdap.api.data.schema.Schema inputSchema, String featuresToInclude, String featuresToExclude, String labelField, String cardinalityMapping) Get the feature to cardinality mapping provided by the user.
static Map<String,Integer>	getFeatureList(co.cask.cdap.api.data.schema.Schema inputSchema, String featuresToInclude, String featuresToExclude, String predictionField) Get the feature list of the features that have to be used for training/prediction depending on the featuresToInclude or featuresToInclude list.
static List<String>	getInputFieldValue(co.cask.cdap.api.data.format.StructuredRecord input, String inputField, com.google.common.base.Splitter splitter) Gets the input field for feature generation.
static co.cask.cdap.api.data.schema.Schema	getOutputSchema(co.cask.cdap.api.data.schema.Schema inputSchema, String predictionField)
static void	validateConfigParameters(co.cask.cdap.api.data.schema.Schema inputSchema, String featuresToInclude, String featuresToExclude, String predictionField, String cardinalityMapping) Validate the config parameters for the spark sink and spark compute classes.
static void	validateFeatureGeneratorConfig(co.cask.cdap.api.data.schema.Schema inputSchema, Map<String,String> map, String pattern) Validate config parameters for feature generator classes.
static void	validateLabelFieldForTrainer(co.cask.cdap.api.data.schema.Schema inputSchema, String labelField) Validate label field for trainer.
static void	validateTextField(co.cask.cdap.api.data.schema.Schema inputSchema, String key) Validate the input field to be used for text based feature generation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

validateConfigParameters

public static void validateConfigParameters(co.cask.cdap.api.data.schema.Schema inputSchema,
                            @Nullable
                            String featuresToInclude,
                            @Nullable
                            String featuresToExclude,
                            String predictionField,
                            @Nullable
                            String cardinalityMapping)

Validate the config parameters for the spark sink and spark compute classes.

Parameters:

inputSchema - schema of the received record.

featuresToInclude - features to be used for training/prediction.

featuresToExclude - features to be excluded when training/predicting.

predictionField - field containing the prediction values.

getFeatureList

public static Map<String,Integer> getFeatureList(co.cask.cdap.api.data.schema.Schema inputSchema,
                                 @Nullable
                                 String featuresToInclude,
                                 @Nullable
                                 String featuresToExclude,
                                 String predictionField)

Get the feature list of the features that have to be used for training/prediction depending on the featuresToInclude or featuresToInclude list.

Parameters:

inputSchema - schema of the received record.

featuresToInclude - features to be used for training/prediction.

featuresToExclude - features to be excluded when training/predicting.

predictionField - field containing the prediction values.

Returns:

feature list to be used for training/prediction.

getCategoricalFeatureInfo

public static Map<Integer,Integer> getCategoricalFeatureInfo(co.cask.cdap.api.data.schema.Schema inputSchema,
                                             @Nullable
                                             String featuresToInclude,
                                             @Nullable
                                             String featuresToExclude,
                                             String labelField,
                                             @Nullable
                                             String cardinalityMapping)

Get the feature to cardinality mapping provided by the user.

Parameters:

inputSchema - schema of the received record.

featuresToInclude - features to be used for training/prediction.

featuresToExclude - features to be excluded when training/predicting.

labelField - field containing the prediction values.

cardinalityMapping - feature to cardinality mapping specified for categorical features.

Returns:

categoricalFeatureInfo for categorical features.

validateLabelFieldForTrainer

public static void validateLabelFieldForTrainer(co.cask.cdap.api.data.schema.Schema inputSchema,
                                String labelField)

Validate label field for trainer.

Parameters:

inputSchema - schema of the received record.

labelField - field from which to get the prediction.

cloneRecord

public static co.cask.cdap.api.data.format.StructuredRecord.Builder cloneRecord(co.cask.cdap.api.data.format.StructuredRecord record,
                                                                co.cask.cdap.api.data.schema.Schema outputSchema,
                                                                String predictionField)

Creates a builder based off the given record. The record will be cloned without the prediction field.

getOutputSchema

public static co.cask.cdap.api.data.schema.Schema getOutputSchema(co.cask.cdap.api.data.schema.Schema inputSchema,
                                                  String predictionField)

validateFeatureGeneratorConfig

public static void validateFeatureGeneratorConfig(co.cask.cdap.api.data.schema.Schema inputSchema,
                                  Map<String,String> map,
                                  String pattern)

Validate config parameters for feature generator classes.

Parameters:

inputSchema - input Schema

map - Map of the input fields to map to the transformed output fields.

validateTextField

public static void validateTextField(co.cask.cdap.api.data.schema.Schema inputSchema,
                     String key)

Validate the input field to be used for text based feature generation.

Parameters:

inputSchema - input schema coming in from the previous stage

key - text field on which to perform text based feature generation

getInputFieldValue

public static List<String> getInputFieldValue(co.cask.cdap.api.data.format.StructuredRecord input,
                              String inputField,
                              com.google.common.base.Splitter splitter)

Gets the input field for feature generation. If the field is an array, returns the field. Otherwise, splits the field based on the given pattern. Returns an empty list if value is null. The method assumes the input field is an array of strings or a string. Validation for the field type should be performed at configure time. The method will throw an exception if the input field is not of required type.

Parameters:

input - input Structured Record

inputField - field to use for feature generation

splitter - Splitter object to be used for splitting the input string

Returns:

text to be used for feature generation