Fully-Connected Layers Decomposer

Factorize heavy FC layers into smaller ones

We can factorize our big fully-connected layers and replace them by an approximation of two smaller layers. The idea is to make an SVD decomposition of the weight matrix, which will express the original matrix in a product of 3 matrices: \(U \Sigma V^T\) With \(\Sigma\) being a diagonal matrix with non-negative values along its diagonal (the singular values). We then define a value \(k\) of singular values to keep and modify matrices \(U\) and \(V^T\) accordingly. The resulting will be an approximation of the initial matrix.

---

FC_Decomposer.decompose

 FC_Decomposer.decompose (model, percent_removed=0.5)

A tutorial about how to use the FC_Decomposer functionalities can be found here