Understand the Simple Power of Backpropagation but also the Dangers

So states Lex Fridman in his lecture on Recurrent Neural Networks (from the course on Deep Learning for Self-Driving Cars).

pip install tensorflow

This will install the current stable release for TensorFlow.

All Eyes on Keras - Layer = IO TRANSFORMATION

What is it and Why Use it

Keras is the high-level API for TensorFlow, covering all aspect of workflow, from data processing to (hyperparameter) tuning to deployment.  It is the API to be used by default.

 

Layers and Models, Layers and Models

The core data structs of Keras are layers and models.

• A layer is a simple input/output transformation
• A model is a directed acyclic graph (DAG) of layers (production flow of layers, and thus a production flow of transformations)

A model is thus a "special" series of input-output transformations i.e. a "series" of layers.

Sidebar: what are hyperparameters

Hyperparameters are parameters you set before training a model. They can be set by a user or by a tuning algorithm. Example parameters could include learning rate (how fast the model learns), or number of layers in the neural network (more layers the more complex patterns the neural net can learn).

Dude, FP16, really? Why not FP32? Ask Voltaire.

With the rise of AI, including hardware acceleration of AI, comes a renewed interested in efficient data types.  

In this spirit, we raise a toast to FP16 or float16, also known as half-precision floating point format, which can be a more appropriate format in some circumstances and some algorithms than what is known as single precision floating point which occupies 32 bits (and hence also called FP32 or float32).

But why use a less precise data type at all, when more precision options are available?

Half precision values are useful in applications where perfect precision is not required, such applications include image processing and neural networks.

FP16 is not to be confused with bfloat16 (Brain float16) which is a different format developed for Google Brain (now Google AI) with the explicit intent of accelerating machine learning and is used a variety of AI processors (including Google Cloud TPUs).