Sparsifier
A sparse vector, as opposed to a dense one, is a vector which contains a lot of zeroes. When we speak about making a neural network sparse, we thus mean that the network’s weight are mostly zeroes.
With fasterai, you can do that thanks to the Sparsifier class.
Sparsifier
Sparsifier (model, granularity, context, criteria, layer_type=<class 'torch.nn.modules.conv.Conv2d'>)
Class providing sparsifying capabilities
The Sparsifier class allows us to remove some weights, that are considered to be less useful than others. This can be done by first creating an instance of the class, specifying:
- The
granularity, i.e. the part of filters that you want to remove. Typically, we usually remove weights, vectors, kernels or even complete filters. - The
context, i.e. if you want to consider each layer independently (local), or compare the parameters to remove across the whole network (global). - The
criteria, i.e. the way to assess the usefulness of a parameter. Common methods compare parameters using their magnitude, the lowest magnitude ones considered to be less useful.
User can pass a single layer to prune by using the Sparsifier.sparsify_layer method.
Sparsifier.sparsify_layer
Sparsifier.sparsify_layer (m, sparsity, round_to=None)
Most of the time, we may want to prune the whole model at once, using the Sparsifier.prune_model method, indicating the percentage of sparsity to you want to apply.
Sparsifier.sparsify_model
Sparsifier.sparsify_model (sparsity, round_to=None)
In some case, you may want to impose the remaining amount of parameters to be a multiple of a given number (e.g. 8), this can be done by passing the round_to parameter.
Also, instead of passing a single value of sparsity, a list of sparsities can also be provided. In that case, each value in the list is the sparsity that will be applied to all layers.
Example: I have a 4-layer network and want to remove half of the parameters from the layers 2 and 3, I can provide the list: sparsity = [0, 50, 50, 0]