In-Depth Case Study on Artificial Neural Network Weights Optimization Using Meta-Heuristic and Heuristic Algorithmic Approach

Abstract

The Meta-heuristic and Heuristic algorithms have been
introduced for deep neural network optimization in this paper.
Artificial Intelligence and the most used Deep Learning methods
are getting popularity in these days, thus we need faster
optimization strategies for finding more accurate results in the
future. Neural Network Optimization with Particle Swarm
Optimization, Backpropagation (BP), Resilient Propagation
(Rprop), and Genetic Algorithm (GA) have been used for
numerical analysis of different datasets and compared with each
other to find out which algorithms work better for finding optimal
solutions by reducing training loss. Meta-heuristic algorithms GA
and PSO are higher-level formulas and problem-independent
techniques that may be used for a diverse number of challenges.
The characteristic of heuristic algorithms has extremely specific
features that vary depending on the problem. The conventional
backpropagation (BP) based optimization, genetic algorithm,
particle swarm optimization, and resilient propagation (Rprop)
are all fully presented, and how to apply these procedures in
artificial deep neural networks optimization is also thoroughly
described. Applied numerical simulation over several datasets
shows that the Meta-heuristic algorithm particle swarm
optimization and also the genetic algorithm performed better than
the conventional heuristic algorithm like backpropagation and
resilient propagation over these datasets. Evaluation of these
algorithms was done based on training epoch and their error
convergence.