TextFeaturizer

class TextFeaturizer.TextFeaturizer(binary=False, caseSensitiveStopWords=False, defaultStopWordLanguage='english', inputCol=None, minDocFreq=1, minTokenLength=0, nGramLength=2, numFeatures=262144, outputCol=None, stopWords=None, toLowercase=True, tokenizerGaps=True, tokenizerPattern='\s+', useIDF=True, useNGram=False, useStopWordsRemover=False, useTokenizer=True)

Bases: mmlspark.Utils.ComplexParamsMixin, pyspark.ml.util.JavaMLReadable, pyspark.ml.util.JavaMLWritable, pyspark.ml.wrapper.JavaEstimator

Featurize text.

Parameters:

• binary (bool) – If true, all nonegative word counts are set to 1 (default: false)
• caseSensitiveStopWords (bool) – Whether to do a case sensitive comparison over the stop words (default: false)
• defaultStopWordLanguage (str) – Which language to use for the stop word remover, set this to custom to use the stopWords input (default: english)
• inputCol (str) – The name of the input column
• minDocFreq (int) – The minimum number of documents in which a term should appear. (default: 1)
• minTokenLength (int) – Minimum token length, >= 0. (default: 0)
• nGramLength (int) – The size of the Ngrams (default: 2)
• numFeatures (int) – Set the number of features to hash each document to (default: 262144)
• outputCol (str) – The name of the output column (default: [self.uid]_output)
• stopWords (str) – The words to be filtered out.
• toLowercase (bool) – Indicates whether to convert all characters to lowercase before tokenizing. (default: true)
• tokenizerGaps (bool) – Indicates whether regex splits on gaps (true) or matches tokens (false). (default: true)
• tokenizerPattern (str) – Regex pattern used to match delimiters if gaps is true or tokens if gaps is false. (default: s+)
• useIDF (bool) – Whether to scale the Term Frequencies by IDF (default: true)
• useNGram (bool) – Whether to enumerate N grams (default: false)
• useStopWordsRemover (bool) – Whether to remove stop words from tokenized data (default: false)
• useTokenizer (bool) – Whether to tokenize the input (default: true)

getBinary()

Returns:If true, all nonegative word counts are set to 1 (default: false) Return type:bool

getCaseSensitiveStopWords()

Returns:Whether to do a case sensitive comparison over the stop words (default: false) Return type:bool

getDefaultStopWordLanguage()

Returns:Which language to use for the stop word remover, set this to custom to use the stopWords input (default: english) Return type:str

getInputCol()

Returns:The name of the input column Return type:str

static getJavaPackage()

Returns package name String.

getMinDocFreq()

Returns:The minimum number of documents in which a term should appear. (default: 1) Return type:int

getMinTokenLength()

Returns:Minimum token length, >= 0. (default: 0) Return type:int

getNGramLength()

Returns:The size of the Ngrams (default: 2) Return type:int

getNumFeatures()

Returns:Set the number of features to hash each document to (default: 262144) Return type:int

getOutputCol()

Returns:The name of the output column (default: [self.uid]_output) Return type:str

getStopWords()

Returns:The words to be filtered out. Return type:str

getToLowercase()

Returns:Indicates whether to convert all characters to lowercase before tokenizing. (default: true) Return type:bool

getTokenizerGaps()

Returns:Indicates whether regex splits on gaps (true) or matches tokens (false). (default: true) Return type:bool

getTokenizerPattern()

Returns:Regex pattern used to match delimiters if gaps is true or tokens if gaps is false. (default: s+) Return type:str

getUseIDF()

Returns:Whether to scale the Term Frequencies by IDF (default: true) Return type:bool

getUseNGram()

Returns:Whether to enumerate N grams (default: false) Return type:bool

getUseStopWordsRemover()

Returns:Whether to remove stop words from tokenized data (default: false) Return type:bool

getUseTokenizer()

Returns:Whether to tokenize the input (default: true) Return type:bool

classmethod read()

Returns an MLReader instance for this class.

setBinary(value)

Parameters:binary (bool) – If true, all nonegative word counts are set to 1 (default: false)

setCaseSensitiveStopWords(value)

Parameters:caseSensitiveStopWords (bool) – Whether to do a case sensitive comparison over the stop words (default: false)

setDefaultStopWordLanguage(value)

Parameters:defaultStopWordLanguage (str) – Which language to use for the stop word remover, set this to custom to use the stopWords input (default: english)

setInputCol(value)

Parameters:inputCol (str) – The name of the input column

setMinDocFreq(value)

Parameters:minDocFreq (int) – The minimum number of documents in which a term should appear. (default: 1)

setMinTokenLength(value)

Parameters:minTokenLength (int) – Minimum token length, >= 0. (default: 0)

setNGramLength(value)

Parameters:nGramLength (int) – The size of the Ngrams (default: 2)

setNumFeatures(value)

Parameters:numFeatures (int) – Set the number of features to hash each document to (default: 262144)

setOutputCol(value)

Parameters:outputCol (str) – The name of the output column (default: [self.uid]_output)

setParams(binary=False, caseSensitiveStopWords=False, defaultStopWordLanguage='english', inputCol=None, minDocFreq=1, minTokenLength=0, nGramLength=2, numFeatures=262144, outputCol=None, stopWords=None, toLowercase=True, tokenizerGaps=True, tokenizerPattern='\\s+', useIDF=True, useNGram=False, useStopWordsRemover=False, useTokenizer=True)

Set the (keyword only) parameters

Parameters:

• binary (bool) – If true, all nonegative word counts are set to 1 (default: false)
• caseSensitiveStopWords (bool) – Whether to do a case sensitive comparison over the stop words (default: false)
• defaultStopWordLanguage (str) – Which language to use for the stop word remover, set this to custom to use the stopWords input (default: english)
• inputCol (str) – The name of the input column
• minDocFreq (int) – The minimum number of documents in which a term should appear. (default: 1)
• minTokenLength (int) – Minimum token length, >= 0. (default: 0)
• nGramLength (int) – The size of the Ngrams (default: 2)
• numFeatures (int) – Set the number of features to hash each document to (default: 262144)
• outputCol (str) – The name of the output column (default: [self.uid]_output)
• stopWords (str) – The words to be filtered out.
• toLowercase (bool) – Indicates whether to convert all characters to lowercase before tokenizing. (default: true)
• tokenizerGaps (bool) – Indicates whether regex splits on gaps (true) or matches tokens (false). (default: true)
• tokenizerPattern (str) – Regex pattern used to match delimiters if gaps is true or tokens if gaps is false. (default: s+)
• useIDF (bool) – Whether to scale the Term Frequencies by IDF (default: true)
• useNGram (bool) – Whether to enumerate N grams (default: false)
• useStopWordsRemover (bool) – Whether to remove stop words from tokenized data (default: false)
• useTokenizer (bool) – Whether to tokenize the input (default: true)

setStopWords(value)

Parameters:stopWords (str) – The words to be filtered out.

setToLowercase(value)

Parameters:toLowercase (bool) – Indicates whether to convert all characters to lowercase before tokenizing. (default: true)

setTokenizerGaps(value)

Parameters:tokenizerGaps (bool) – Indicates whether regex splits on gaps (true) or matches tokens (false). (default: true)

setTokenizerPattern(value)

Parameters:tokenizerPattern (str) – Regex pattern used to match delimiters if gaps is true or tokens if gaps is false. (default: s+)

setUseIDF(value)

Parameters:useIDF (bool) – Whether to scale the Term Frequencies by IDF (default: true)

setUseNGram(value)

Parameters:useNGram (bool) – Whether to enumerate N grams (default: false)

setUseStopWordsRemover(value)

Parameters:useStopWordsRemover (bool) – Whether to remove stop words from tokenized data (default: false)

setUseTokenizer(value)

Parameters:useTokenizer (bool) – Whether to tokenize the input (default: true)

class TextFeaturizer.TextFeaturizerModel(java_model=None)

Bases: mmlspark.Utils.ComplexParamsMixin, pyspark.ml.wrapper.JavaModel, pyspark.ml.util.JavaMLWritable, pyspark.ml.util.JavaMLReadable

Model fitted by TextFeaturizer.

This class is left empty on purpose. All necessary methods are exposed through inheritance.

static getJavaPackage()

Returns package name String.

classmethod read()

Returns an MLReader instance for this class.