\subsection{TensorFlow backend}
\label{sec:implementation}
Deep learning has recently gained a lot of attention as it has been quite
successful
as a tool in big data analysis and predictive statistics. In its core, the idea
is to extract correlations from data by training a neural network on
them in order to make
accurate predictions on unknown samples. This can be the classification of
images, the prediction of stock markets etc. More interestingly, the typical
deep learning workflow
can be summarised also by Fig.
\ref{fig:fit_workflow} by replacing ``model" with ``neural network",
\footnote{Neural networks in deep learning are also called models. The term
``model" will here solely be used for ``classical" model fitting as in
\zfit{}.}
``minimisation" with
``training"
and removing the last block ``Result \& Error". Moreover, deep learning and HEP
model fitting both use large data samples and build complex models. This
similarity inspired the implementation of \zfit{} with a deep learning
framework as the backend.
While we have just shown how the two workflows look incredibly similar at first
glance, there are
some hidden, crucial differences.
Knowing them is essential in order to understand the advantages but
also limitations of this approach. In the following we will have a simplified
at the core of deep learning and compare then to model fitting.
\begin{itemize}
\item A Deep Neural Network (DNN) is simply a function $f:
\mathbb{R}^n \rightarrow \mathbb{R}^m$. The input space is the number of
observables of a single event. The output of the function, the prediction,
is
notably $m$ dimensional. Contrary to this, a model with PDFs outputs into
the one dimensional space of a normalised probability. DNN outputs, if used
for classification,
correspond to pseudo-probabilities. While they are not normalised, there
exists a monotonic transformation to a probability.
\textit{Consequence}: normalisation over a certain range is specific to
model fitting and no explicit tool, e.g. for numerical integration, exists
in the deep learning frameworks.
\item
In model fitting, the composition of shapes is motivated by previous
knowledge
and an underlying theory is built. There is often a meaning behind each
part and the shape of the model is restricted to a specific problem. This
specific shape, coming from assumptions and
previous knowledge, is what keeps the number of parameters low: typically,
no more than a dozen for simple fits and a maximum of a few hundreds for
the most complicated fits are used.
\begin{figure}[tbp]
\centering
\includegraphics[width=0.5\textwidth]{images/neural_network_scheme.png}
\caption{A schematic view of a DNN function. Input takes the data and
has
the dimension of an event. Each end-point of a line is multiplied
by a
weight, a free parameter, and added with the other lines. At a
node, a
non-linear function is then applied.}
\label{fig:dnn}
\end{figure}
Contrary to that, the structure of a DNN is basically agnostic to the
problem and depends mostly on its complexity. It therefore incorporates a
minimum of pre-knowledge and assumptions about the correlations in the
data. This huge
versatility is what makes deep learning such a successful field but comes
at a price: a great number of free parameters is needed, starting at
thousands for really simple DNNs, typically being around hundreds of
thousands and going to tens of millions. DNNs are a structure consisting of
layers with nodes as shown in Fig. \ref{fig:dnn}. Each of the nodes adds an
additional parameter for every incoming connection, resulting in a large
number of parameters.
\textit{Consequence}: While dozens of DNN building libraries like
Keras\footnote{Keras is an API specification only, a
reference implementation exists.}, PyTorch and more offer great
capabilities in building DNNs,
their abstractions into layers are of no use to model fitting.
\item A DNN is essentially matrix
multiplications scaled by the free parameters, with an additional simple,
non-linear activation function applied. In model fitting, the shape can
have an arbitrary complexity and contain a whole range of elementary
functions. Furthermore control flow elements and complex number are often
used as well. Not only is the function itself more complicated, also the
dependency of parameters can be highly non-trivial.
There is an additional difference in the precision of the
floating point operations. In model fitting, the
precision required is
higher, because the values of likelihoods and the changes are larger
compared with neural networks often having values varying between $-1$ and
$1$. Also the Quasi-Newton methods as described below build an approximate
second order derivatives which needs a high enough precision.
\textit{Consequence}: While both fields do heavy computations, the focus of
the optimizations in a fitting library is slightly different and requires
for
example to always explicitly specify float64 as data type.
\item Minimising a loss is a non-trivial task.
Algorithms usually start at a certain point and use local information to
make forward steps. The gradient and sometimes higher order derivatives,
usually up to the second order, are used to help finding the
minimum. In particular, which order is usable strongly depends on the
number of
parameters: the
Hessian matrix of $n$ parameters has $n^2$ entries rendering its
calculation unfeasible for more than a few hundred of parameters;
this restricts the minimisation of DNNs to only use the first order
derivatives at the cost of more required steps.
\textit{Consequence}: On one side, minimisers designed for DNNs and
optimised to work with the framework are not suitable for model fitting. On
the other side the analytic gradients that are provided by th frameworks
for their minimisers can be easily extended to higher orders and come in
very handy for model fitting minimisers.
\item Fitting a model has the goal to find the
parameters for which the model matches the data best. In terms of a loss
function, this is equivalent to finding its \textit{global} minimum. Being
stuck in a local minimum is a problem and requires careful treatment.
Contrary in
DNNs the global minimum is not found but also not desired. The DNN
has to
approximate an arbitrary data sample \textit{good enough} and a local
minimum is usually found, it is in fact preferred over the global
minimum since, due to the high degrees of freedom of a DNN, this typically
entails
a huge over-fit\footnote{Basically remembering every noise in the
data.} and a bad generalization.
\textit{Consequence}: While finding the global minimum is crucial in model
fitting, deep learning is interested to find a local minimum.
\item
Finding the global minimum in the model fitting case involves the
evaluation of the loss over the whole data sample at every step. For the
training
of a DNN, variations of a technique called stochastic gradient descent
(SGD) are used: they work by only
evaluating a small mini-batch of the data, typically around 32 events, and
then taking a step towards
the negative gradient direction.
\textit{Consequence}: Both fields are pushing the limits of handling big
data and samples too large for the memory are common. Deep learning is
optimized to loop through a data set with very small batch sizes to take a
minimisation step but to do that millions of times. Model fitting needs to
loop
through the whole data sample \textit{once} for a single step but no more
then a few thousand times.
\end{itemize}
As seen, while there are differences, the
core of the problem is still the same: build a complicated model, use data to
get a value from it and tune parameters to optimize the loss.
To accomplish this task efficiently, deep learning frameworks use a declarative
paradigm by building a computation graph as seen in Fig.
\ref{fig:graph_example}.
This allows to perform optimisations and define the parallelisation
\textit{before} the actual execution of the computation. Furthermore, it also
allows to get an analytic expression for the gradient by consecutively applying
the chain rule within the graph.
\begin{figure}[tbp]
\centering
\includegraphics[width=0.35\textwidth]{figs/graph_example.png}
\caption{Example of a graph representing $result = (5 + 3) *
(7 + 2)$.}
\label{fig:graph_example}
\end{figure}
To implement model fitting in \zfit{}, the TensorFlow library was chosen,
restricting the implementation to static graphs as explained in more detail in
Appendix \ref{appendix:tensorflow}. The main motivation for this decision comes
from the fact
that models in model fitting \textit{usually} don't change their logic but
are rather built once and then minimised. While the same is true for most DNNs,
more
advanced fields in deep learning like reinforcement learning can heavily rely
on dynamic models. The main advantages of a static graph are the additional,
potential speedup and the immutability, which leads to less unexpected
behaviour. The
restriction that \textit{anything built will remain like that and not change}
allows for obviously more efficient optimizations compared to a graph where any
part could change anytime and a re-analysis of the graph is required.
Working with graphs leads to difficulties and unexpected behaviours
in comparison with more traditional, non-graph based code, such as the one used
in \roofit.
In \zfit{}, most of these complications are hidden from the user and the
library offers a
similar user experience as the one found in other model fitting libraries. This
requires
some extra care
taking behind the scenes, as explained in Appendix \ref{appendix:tensorflow}.
