Title: Classifying stable and unstable videos with deep convolutional networks

Abstract: Since the invention of cameras, video shooting has become a passion for human. However, the quality of videos recorded with devices such as handheld cameras, head cameras, and vehicle cameras may be low due to shaking, jittering and unwanted periodic movements. Although the issue of video stabilization has been studied for decades, there is no consensus on how to measure the performance of a video stabilization method. In many studies in the literature, different metrics have been used for comparison of different methods. In this study, deep convolutional neural networks are used as a decision maker for video stabilization. VGG networks with different number of layers are used to determine the stability status of the videos. It was observed that VGG networks showed a classification performance up to 96.537% using only two consecutive scenes. These results show that deep learning networks can be utilized as a metric for video stabilization.

Keywords: Video stabilization; deep learning; convolutional neural networks

Başlık: Stabil ve stabil olmayan videoların derin evrişimli ağlarla sınıflandırılması

Özet: Kameraların icadından bu yana, video çekmek insan için bir tutku haline gelmiştir. Ancak el kameraları, baş kameraları ve araç kameraları gibi cihazlarla kaydedilen videoların kalitesi sallantı, titreme ve istenmeyen periyodik hareketler nedeniyle düşük olabilmektedir. Video stabilizasyonu konusu onlarca yıldır çalışılmış olsa da, bir video stabilizasyon yönteminin performansının nasıl ölçüleceği konusunda bir fikir birliği yoktur. Literatürdeki birçok çalışmada, farklı yöntemlerin karşılaştırılması için farklı ölçütler kullanılmıştır. Bu çalışmada, derin evrişimli sinir ağları video stabilizasyonu için karar verici olarak kullanılmaktadır. Videoların kararlılık durumunu belirlemek için farklı sayıda katmana sahip VGG ağları kullanılmıştır. VGG ağlarının sadece iki ardışık sahne kullanarak %96,537'ye varan bir sınıflandırma performansı gösterdiği görülmüştür. Bu sonuçlar, derin öğrenme ağlarının video stabilizasyonu için bir ölçü olarak kullanılabileceğini göstermektedir.

Anahtar kelimeler: Video stabilizasyonu; derin öğrenme; evrişimli sinir ağları

• Matsushita Y, Ofek E, Ge W, Tang X, Shum HY. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006; 28(7): 1150-1163.
• Battiato S, Gallo G, Puglisi G, Scellato S. SIFT features tracking for video stabilization. In 14th International Conference on Image Analysis and Processing (ICIAP 2007), September 10-14, 2007, Modena, Italy, pp. 825-830.
• Liu S, Yuan L, Tan P, Sun J. Steadyflow: Spatially smooth optical flow for video stabilization. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 23-28, 2014, Columbus, OH, USA, pp. 4209-4216.
• Liu S, Tan P, Yuan L, Sun J, Zeng B. Meshflow: Minimum latency online video stabilization. In European Conference on Computer Vision (ECCV 2016), Lecture Notes in Computer Science 2016; 9910: 800–815.
• Walha A, Wali A, Alimi AM. Video stabilization for aerial video surveillance. AASRI Procedia 2013; 4: 72-77.
• Xu SZ, Hu J, Wang M, Mu TJ, Hu SM. Deep video stabilization using adversarial networks. Computer Graphics Forum 2018; 37(7): 267-276.
• Wang M, Yang GY, Lin JK, Zhang SH, Shamir A, Lu SP, Hu SM. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing 2018; 28(5): 2283-2292.
• Wang M, Yang GY, Lin JK, Shamir A, Zhang SH, Lu SP, Hu SM. Deep online video stabilization. arXiv, 2018.
• Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS 2012), December 2012, pp. 1097–1105.
• Lawrence S, Giles CL, Tsoi AC, Back AD. Face recognition: A convolutional neural-network approach. IEEE Transactions on Neural Networks 1997; 8(1): 98-113.
• Zhang X, Zhou X, Lin M, Sun J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 6848-6856.
• Dong C, Loy CC, Tang X. Accelerating the super-resolution convolutional neural network. In Computer Vision – ECCV 2016. Lecture Notes in Computer Science 2016; 9906: 391-407.
• Xu L, Ren JS, Liu C, Jia J. Deep convolutional neural network for image deconvolution. Advances in Neural Information Processing Systems 2014; 27: 1790-1798.
• Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint, 2014.
• Wang M, Yang GY, Lin JK, Shamir A, Zhang SH, Lu SP, Hu SM. Deep online video stabilization. arXiv preprint, 2018.