Wie kann man das RandomForest Modell validieren?

Möchte ich bewerten, ein random forest ausgebildet, auf einige Daten. Gibt es eine utility, die in Apache Spark, das gleiche zu tun oder habe ich zu erledigen-Kreuzvalidierung manuell?

InformationsquelleAutor der Frage ashishsjsu | 2015-09-24

  1. 32

    ML bietet CrossValidator Klasse, die kann verwendet werden, um cross-Validierung und-parameter suchen. Vorausgesetzt, Ihre Daten bereits vorverarbeitet, die Sie hinzufügen können, die Kreuz-Validierung wie folgt:

    import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.tuning.{ParamGridBuilder, CrossValidator}
import org.apache.spark.ml.classification.RandomForestClassifier
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator

//[label: double, features: vector]
trainingData org.apache.spark.sql.DataFrame = ??? 
val nFolds: Int = ???
val NumTrees: Int = ???
val metric: String = ???

val rf = new RandomForestClassifier()
  .setLabelCol("label")
  .setFeaturesCol("features")
  .setNumTrees(NumTrees)

val pipeline = new Pipeline().setStages(Array(rf)) 

val paramGrid = new ParamGridBuilder().build() //No parameter search

val evaluator = new MulticlassClassificationEvaluator()
  .setLabelCol("label")
  .setPredictionCol("prediction")
  //"f1" (default), "weightedPrecision", "weightedRecall", "accuracy"
  .setMetricName(metric) 

val cv = new CrossValidator()
  //ml.Pipeline with ml.classification.RandomForestClassifier
  .setEstimator(pipeline)
  //ml.evaluation.MulticlassClassificationEvaluator
  .setEvaluator(evaluator) 
  .setEstimatorParamMaps(paramGrid)
  .setNumFolds(nFolds)

val model = cv.fit(trainingData) //trainingData: DataFrame

    Mit PySpark:

    from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

trainingData = ... # DataFrame[label: double, features: vector]
numFolds = ... # Integer

rf = RandomForestClassifier(labelCol="label", featuresCol="features")
evaluator = MulticlassClassificationEvaluator() # + other params as in Scala    

pipeline = Pipeline(stages=[rf])

crossval = CrossValidator(
    estimator=pipeline,
    estimatorParamMaps=paramGrid,
    evaluator=evaluator,
    numFolds=numFolds)

model = crossval.fit(trainingData)

    InformationsquelleAutor der Antwort zero323

  2. 1

    Bauen auf zero323 tolle Antwort mit Random Forest Klassifikator, hier ist ein ähnliches Beispiel für Random Forest Regressor:

    import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.tuning.{ParamGridBuilder, CrossValidator}
import org.apache.spark.ml.regression.RandomForestRegressor //CHANGED
import org.apache.spark.ml.evaluation.RegressionEvaluator //CHANGED
import org.apache.spark.ml.feature.{VectorAssembler, VectorIndexer}

val numFolds = ??? //Integer
val data = ??? //DataFrame

//Training (80%) and test data (20%)
val Array(train, test) = data.randomSplit(Array(0.8,0.2))
val featuresCols = data.columns
val va = new VectorAssembler()
va.setInputCols(featuresCols)
va.setOutputCol("rawFeatures")
val vi = new VectorIndexer()
vi.setInputCol("rawFeatures")
vi.setOutputCol("features")
vi.setMaxCategories(5)
val regressor = new RandomForestRegressor()
regressor.setLabelCol("events")

val metric = "rmse"
val evaluator = new RegressionEvaluator()
  .setLabelCol("events")
  .setPredictionCol("prediction")
  //    "rmse" (default): root mean squared error
  //    "mse": mean squared error
  //    "r2": R2 metric
  //    "mae": mean absolute error 
  .setMetricName(metric) 

val paramGrid = new ParamGridBuilder().build()
val cv = new CrossValidator()
  .setEstimator(regressor)
  .setEvaluator(evaluator) 
  .setEstimatorParamMaps(paramGrid)
  .setNumFolds(numFolds)

val model = cv.fit(train) //train: DataFrame
val predictions = model.transform(test)
predictions.show
val rmse = evaluator.evaluate(predictions)
println(rmse)

    Evaluator-Metrik Quelle:
    https://spark.apache.org/docs/latest/api/scala/#org.apache.spark.ml.evaluation.RegressionEvaluator

    InformationsquelleAutor der Antwort Garren

Schreibe einen Kommentar