QPLaBSE: Quantized and Pruned Language-Agnostic BERT Sentence Embedding Model : Production-ready compression for multilingual transformers
Abstract: Transformer models perform well on Natural Language Processing and Natural Language Understanding tasks. Training and fine-tuning of these models consume a large amount of data and computing resources. Fast inference also requires high-end hardware for user-facing products. While distillation, quantization, and head-pruning for transformer models are well- explored domains in academia, the practical application is not straightforward. Currently, for good accuracy of the optimized models, it is necessary to fine-tune them for a particular task. This makes the generalization of the model difficult. If the same model has to be used for multiple downstream tasks, then it would require applying the process of optimization with fine-tuning for each task. This thesis explores the techniques of quantization and pruning for optimization of the Language-Agnostic BERT Sentence Embedding (LaBSE) model without fine-tuning for a downstream task. This should enable the model to be generalized enough for any downstream task. The techniques explored in this thesis are dynamic quantization, static quantization, quantize-aware training quantization, and head-pruning. The downstream performance is evaluated using sentiment classification, intent classification, and language-agnostic classification tasks. The results show that LaBSE can be accelerated on the CPU to 2.6x its original inference time without any loss of accuracy. Head-pruning 50% of the heads from each layer leads to 1.2x speedup while removing all heads but one leads to 1.32x speedup. A speedup of almost 9x is achieved by combining quantization with head-pruning with average 8% drop in accuracy on downstream evaluation tasks.
AT THIS PAGE YOU CAN DOWNLOAD THE WHOLE ESSAY. (follow the link to the next page)