High demand for computational power significantly limits the possibility of using modern deep learning methods in the environments where one has to deal with devices limited by the performance and the energy constraints. To address this issue, this paper proposes a novel method of combining the pruning and the transfer learning techniques for the purpose of delivering solid accuracy while simultaneously lowering the demand for energy and computing power. This method is referred to as \emph{Finicky Transfer Learning} (FTL) as it is finicky during the process of selecting filters from a pre-trained feature extractor to compose a sparser architecture. The proposed filter selection process is based on an original approach utilizing the Jaccard similarity coefficient calculated between the activation maps and the masks obtained by semantic segmentation. This enables the use of CNN, trained previously on a large generic dataset, in a crack classification task. The presented method significantly lowers the inference time while maintaining or even slightly increasing the classification accuracy, enabling real-time operation on single-board computers.