Comparing Different Characteristics of Deterministic and Stochastic Optimization Methods

In this video, I’m going to compare different characteristics of deterministic and stochastic optimization methods. This comparison could help you choose the right optimization method to solve various optimization problems in your fields.

Optimisation is generally referred to as finding the best solution to an optimisation problem. Optimisation is always desirable in many fields such as engineering design, computer science, operations research, economics, computational chemistry, biomedicine, etc.

Optimisation methods can be broadly classified into two categories: deterministic methods and stochastic methods. Each category has advantages and disadvantages.

This comparison can help you select the right optimization method to solve your optimization problems. Let’s get started.

For more videos like this, check my YouTube channel here.

OPTIMIZATION METHODS

Optimisation is generally referred to as finding the best solution to an optimisation problem. Optimisation is always desirable in many fields such as engineering design, computer science, operations research, economics, computational chemistry, biomedicine, etc. (Coelho, Ayala & Mariani 2014; Ng & Li 2014; Wang et al. 2013).

Optimisation methods can be broadly classified into two categories: deterministic methods and stochastic methods (Hanagandi & Nikolaou 1998). Each category has advantages and disadvantages.

Deterministic methods are capable of guaranteeing the global optimal solutions for certain problems thanks to exploiting some helpful features of the problem structure. However, they may fail when tackling black-box problems, extremely ill-behaved functions or complex large-scale problems, due to the combinatorial explosion issue.

Stochastic methods can work with any kind of optimisation problems but they are of weak capability of guaranteeing the global optimal solutions. Stochastic methods only provide the global optimal solutions with probabilistic guarantee and this probability will become 1 in infinite computing time (Liberti & Kucherenko 2005; Moles, Mendes & Banga 2003).

Nevertheless, there is no algorithm capable of solving general optimisation problem with certainty in finite computing time (Boender & Romeijn 1995).

1. Deterministic Optimisation Methods

Deterministic optimisation methods are capable of guaranteeing the global optimal solutions for certain problems thanks to exploiting some helpful features of the problem structure. There are a number of deterministic optimisation methods such as Branch-and-Bound methods, Cutting Plane methods, Primal-Dual Decomposition methods, Outer Approximation methods, Inner Approximation methods, Difference of Convex methods, Reverse Convex methods, Reformulation-Linearization methods, Lipschitzian methods, Trajectory and Homotopy methods, Interval Analysis methods, etc. (Floudas 2000).

2. Stochastic Optimisation Methods

Stochastic optimisation methods can work with any kind of problems but they have a weak capability of guaranteeing the global optimal solution. Stochastic methods provide the global optimal solution with probabilistic guarantee only and this probability will become 1 in infinite computing time (Liberti & Kucherenko 2005; Moles, Mendes & Banga 2003). Nevertheless, there is no algorithm capable of finding the global optimal solution with certainty to a general optimisation problem in finite computing time (Boender & Romeijn 1995).

Literature survey shows that stochastic optimisation methods are more popular than deterministic ones in real-life applications. This could be due to a number of reasons. First, stochastic methods do not require sophisticated mathematical analysis to solve the problems. Second, stochastic methods could handle practical and large-scale problems better than deterministic ones. Finally, nowadays the advanced computational technology allows stochastic methods to increase the probability of finding the global optimal solution because a large number of solutions can be generated and evaluated in a relatively short computing time (Dao, Abhary & Marian 2016).

A number of stochastic optimisation methods have been developed to solve many complex large-scale problems such as Genetic Algorithm, Tabu Search, Particle Swarm Optimisation, Cuckoo Search, Ant Colony Optimisation, Simulated Annealing, Hill Climbing, Downhill Simplex, Artificial Bee Colony Algorithm, Swarm Intelligence, Differential Evolution Algorithm, etc. Each method has advantages and disadvantages.

My favourite optimization algorithm is Genetic Algorithm. How about you? What is your favourite optimization algorithm?

P/s: If you find the post useful, share it to remember and to help other people as well.