one_shot.plot(50)Schedules
When should I prune my network ?
Schedule
Schedule (sched_func, start_pct=0.0, end_pct=None, start_sparsity=0.0)
Base class to create schedules
The Schedule class allows you to create any schedule function according to 4 parameters:
- the sched_func: the function according to which the sparsity will evolve (e.g. linear, cos, …)
- the start_pct: the percentage of training steps at which the sparsification process will start
- the end_pct: the percentage of training steps at which the sparsification process will end
- the start_sparsity: the percentage of sparsity at which the model starts
One-Shot
The easiest schedule is the one-shot pruning, i.e. prune the network once. This can be done by simply returning the desired sparsity value. The moment when you want to prune will be controlled by the start_epoch argument in the SparsifyCallback.
sched_oneshot
sched_oneshot (start, end, pos)
Iterative
Instead of pruning the network to desired sparsity in one step, you can do it iteratively. In fasterai, you can change the amount of iterations
sched_iterative
sched_iterative (start, end, pos, n_steps=3)
Perform iterative pruning, and pruning in n_steps steps
iterative.plot(50)
To modify the default n_steps, you can use the partial function.
iterative = Schedule(partial(sched_iterative, n_steps=5), start_pct=0.2)iterative.plot(50)
Automated Gradual Pruning
Some researchers have come up with more sophisticated schedules, such as the Automated Gradual Pruning.
sched_agp
sched_agp (start, end, pos)
agp.plot(50)
One-Cycle Pruning
sched_onecycle
sched_onecycle (start, end, pos, α=14, β=6)
one_cycle.plot(50)
On top of that, all of the schedules available in fastai by default are also available: - sched_cos - sched_linear
Dense-Sparse-Dense
You can also create even more interesting behaviours such as the DSD method, where you prune the model in the first place, then re-grow it to its initial amount of parameter.
sched_dsd
sched_dsd (start, end, pos)
dsd.plot(50)