Build Training Data Jobs

Configuration Properties

Use this job to build training data for query classification by joining signals data with catalog data. The output of this job can be used as input for the Classification job.

For detailed configuration steps, see Classify New Queries.

Configuration Properties

Property Description, Type

analyzerConfig

Lucene Text Analyzer

required

The style of text analyzer you would like to use.

type: string

default value: '{ "analyzers": [{ "name": "StdTokLowerStop","charFilters": [ { "type": "htmlstrip" } ],"tokenizer": { "type": "standard" },"filters": [{ "type": "lowercase" }] }],"fields": [{ "regex": ".+", "analyzer": "StdTokLowerStop" } ]}'

catalogFormat

Catalog Format

required

Spark-compatible format that contains catalog data (like 'solr', 'parquet', 'orc' etc)

type: string

catalogIdField

Item Id Field in Catalog

required

Item Id field in catalog, which will be used to join with signals

type: string

catalogPath

Catalog Path

required

Catalog collection or cloud storage path which contains item categories.

type: string

categoryField

Category Field in Catalog

required

Item category field in catalog.

type: string

countField

Count Field in Signals

required

Count Field in raw or aggregated signals.

type: string

default value: 'aggr_count_i'

dataFormat

Signals Format

Spark-compatible format that contains training data (like 'solr', 'parquet', 'orc' etc)

type: string

default value: 'solr'

minLength: 1

dataOutputFormat

Data output format

Spark-compatible output format (like 'solr', 'parquet', etc)

type: string

default value: 'solr'

minLength: 1

fieldToVectorize

Query Field

required

Field containing query strings.

type: string

default value: 'query_s'

minLength: 1

Spark Job ID

required

The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_). Maximum length: 63 characters.

type: string

maxLength: 63

pattern: [a-zA-Z][_\-a-zA-Z0-9]*[a-zA-Z0-9]?

itemIdField

Item Id Field in Signals

required

Item Id field in signals, which will be used to join with catalog.

type: string

default value: 'doc_id_s'

outputCollection

Output Collection

Solr Collection to store model-labeled data to

type: string

outputPath

Output Path

required

Output collection or cloud storage path which contains item categories.

type: string

overwriteOutput

Overwrite Output

Overwrite output collection

type: boolean

default value: 'true'

partitionCols

Partition fields

If writing to non-Solr sources, this field will accept a comma-delimited list of column names for partitioning the dataframe before writing to the external output

type: string

randomSeed

Random seed

For any deterministic pseudorandom number generation

type: integer

default value: '1234'

readOptions

Read Options

Options used when reading input from Solr or other sources.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

signalsPath

Signals Path

required

Signals collection or cloud storage path which contains item categories.

type: string

sourceFields

Fields to Load

Solr fields to load (comma-delimited). Leave empty to allow the job to select the required fields to load at runtime.

type: string

sparkConfig

Spark Settings

Spark configuration settings.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

sparkSQL

Spark SQL filter query

Use this field to create a Spark SQL query for filtering your input data. The input data will be registered as spark_input

type: string

default value: 'SELECT * from spark_input'

topCategoryProportion

Top Category Proportion

Proportion of the top category has to be among all categories.

type: number

default value: '0.5'

topCategoryThreshold

Minimum Count

Minimum number of query,category pair counts.

type: integer

default value: '1'

exclusiveMinimum: false

minimum: 1

trainingCollection

Training Collection

Solr Collection containing labeled training data

type: string

minLength: 1

trainingDataFilterQuery

Signal Data Filter Query

Solr query to additionally filter signals. For non-solr data source use SPARK SQL FILTER QUERY under Advanced to filter results

type: string

default value: '*:*'

trainingDataFrameConfigOptions

Dataframe Config Options

Additional spark dataframe loading configuration options

type: object

object attributes: {
}

trainingDataSamplingFraction

Training data sampling fraction

Fraction of the training data to use

type: number

default value: '1.0'

exclusiveMaximum: false

maximum: 1.0

type

Spark Job Type

required

type: string

default value: 'build-training'

enum: { build-training }

writeOptions

Write Options

Options used when writing output to Solr or other sources

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }