Boosting Vs. Bagging — how they work (fun way)

Explained boosting and bagging algorithm in simple layman terms and explained their differences in plain English


Before explaining the techniques, let us consider a scenario:

Manager 1

Manager 1 is a strong believer of democratic process. He decided to assign the problem to one and all of the 100 employees in his department. He asked the employees to randomly sample data from the full dataset and develop a model and come up with a predicted score and verdict.

Manager 2

Manager 2, on the other hand, believes that there is always a scope of improvement and every time a new employee looks at the model, new insights and value get added. So he asked his employees to work in sequence: first a randomly picked employee with generate a candidate model. Once the model is built, it will be passed to the next employee (randomly picked excluding the first employee). But while developing the model, the second employee will try to reduce the errors committed by employee 1.

Difference between the bagging and boosting algorithm

As evident from the above two examples, there are some basic differences between these two algorithms. Let us list down these differences:

  • Bagging can operate in parallel: In the above example no one employee is dependent on other for generating their own candidate model.
    Boosting works in sequence: in the above example, an employee has to wait for the previous employee to finish his job and then he can leverage the model to find out errors and perform his steps. Hence this type of algorithm takes longer to solve.
  • Bagging do not leverage the information of other employees. As a result the performance of bagging algorithm is lower a compared to Boosting algorithm with put a lot a focus on error part and try to reduce it.
  • Bagging is considered stable solution due to the democratic approach it takes. Boosting, on the other hand, often suffers from overfitting problem. Selection of learning rate is important in Boosting algorithm to attain the global optimum.

 by the author.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store