Networks Library

This module provides utilities to build neural networks with Knet, along with a library of standard architectures.

KNetwork{Game} <: AbstractNetwork{Game}

Abstract type for neural networks implemented using the Knet framework.

• Subtypes are expected to be expressed as the composition of Flux-like layers that implement a functor interface through functions children and mapchildren.
• A custom implementation of regularized_params_ must also be implemented for layers containing parameters that are subject to regularization.

Provided that the above holds, KNetwork implements the full network interface with the following exceptions: Network.HyperParams, Network.hyperparams, Network.forward and Network.on_gpu.

TwoHeadNetwork{Game} <: KNetwork{G}

An abstract type for two-head neural networks implemented with Knet.

Subtypes are assumed to have the following fields: hyper, common, vhead and phead. Based on those, an implementation is provided for Network.hyperparams, Network.forward and Network.on_gpu, leaving only Network.HyperParams to be implemented.

ResNet{Game} <: TwoHeadNetwork{Game}

The convolutional residual network architecture that is used in the original AlphaGo Zero paper.

ResNetHP

Hyperparameters for the convolutional resnet architecture.

The trunk of the two-head network consists of num_blocks consecutive blocks. Each block features two convolutional layers with num_filters filters and with kernel size conv_kernel_size. Note that both kernel dimensions must be odd.

During training, the network is evaluated in training mode on the whole dataset to compute the loss before it is switched to test model, using big batches. Therefore, it makes sense to use a high batch norm momentum (put a lot of weight on the latest measurement).

AlphaGo Zero Parameters

The network in the original paper from Deepmind features 20 blocks with 256 filters per convolutional layer.

SimpleNet{Game} <: TwoHeadNetwork{Game}

A simple two-headed architecture with only dense layers.

SimpleNetHP

Hyperparameters for the simplenet architecture.