Executing ML Jobs in Azure Databricks From StreamSets

In my previous post, I demonstrated how to achieve low-latency inference using Databricks ML models in StreamSets. Now let's say you have a dataflow pipeline that is ingesting data, enriching it, and performing transformations, and based on certain condition(s), you'd like to (re)train the Databricks ML model. For instance, using a different value for hyperparameter n_estimators (“number of trees” in the forest), which is one of the most important parameters of Random forest machine learning method.

In this post, you will learn how to execute such machine learning jobs in Azure Databricks using StreamSets Databricks Executor.

from DZone.com Feed https://ift.tt/2HmH1O6