Title: Interpolation-Based Smart Video Stabilization

Abstract: Video stabilization is the process of eliminating unwanted camera movements and shaking in a recorded video. Recently, learning-based video stabilization methods have become very popular. Supervised learning-based approaches need labeled data. For the video stabilization problem, recording both stable and unstable versions of the same video is quite troublesome and requires special hardware. In order to overcome this situation, learning-based interpolation methods that do not need such data have been proposed. In this paper, we review recent learning-based interpolation methods for video stabilization and discuss the shortcomings and potential improvements of them.

Keywords: video stabilization; deep learning; unsupervised learning; interpolation methods

Başlık: Enterpolasyon Tabanlı Akıllı Video Stabilizasyonu

Özet: Video stabilizasyonu, kaydedilen bir videoda istenmeyen kamera hareketlerini ve titremeyi ortadan kaldırma işlemidir. Son zamanlarda, öğrenme tabanlı video sabitleme yöntemleri oldukça popüler hale geldi. Denetimli öğrenme temelli yaklaşımların etiketlenmiş verilere ihtiyacı vardır. Video stabilizasyon problemi için aynı videonun hem stabil hem de stabil olmayan versiyonlarını kaydetmek oldukça zahmetlidir ve özel donanım gerektirir. Bu durumun üstesinden gelebilmek için bu tür verilere ihtiyaç duymayan öğrenme tabanlı enterpolasyon yöntemleri önerilmiştir. Bu yazıda, video sabitleme için en son öğrenmeye dayalı enterpolasyon yöntemlerini gözden geçiriyoruz ve bunların eksikliklerini ve potansiyel iyileştirmelerini tartışıyoruz.

Anahtar kelimeler: video stabilizasyon; derin öğrenme; öğreticisiz öğrenme; enterpolasyon yöntemleri

• Kir Savas B, Becerikli Y. Development of driver fatigue detection system by using video images. Journal of Intelligent Systems with Applications 2019; 2(1): 26-29.
• Chiu LC, Chang TS, Chen JY, Chang NYC. Fast SIFT design for realtime visual feature extraction. IEEE Transactions on Image Processing 2013; 22(8): 3158-3167.
• Lowe DG. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 2004; 60(2): 91-110.
• Bay H, Tuytelaars T, Van Gool L. SURF: Speeded up robust features. In European Conference on Computer Vision (ECCV), May 7-13, 2006, Graz, Austria, pp. 404-417.
• Rublee E, Rabaud V, Konolige K, Bradski G. ORB: An efficient alternative to SIFT or SURF. In 2011 International Conference on Computer Vision, November 6-13, 2011, Barcelona, Spain, pp. 2564-2571.
• Lee YC, Tseng KW, Chen YT, Chen CC, Chen CS, Hung YP. 3D video stabilization with depth estimation by CNN-based optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 20-25, 2021, Nashville, TN, USA, pp. 10621-10630.
• Xu Y, Zhang J, Maybank SJ, Tao D. DUT: Learning video stabilization by simply watching unstable videos. arXiv preprint, 2011.14574, 2020.
• Choi M, Kim H, Han B, Xu N, Lee KM. Channel attention is all you need for video frame interpolation. Proceedings of the AAAI Conference on Artificial Intelligence 2020; 34(7): 10663-10671.
• Han D. Comparison of commonly used image interpolation methods. In Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE), 2013, pp. 1556-1559.
• Mahajan D, Huang FC, Matusik W, Ramamoorthi R, Belhumeur P. Moving gradients: A path-based method for plausible image interpolation. ACM Transactions on Graphics (TOG) 2009; 28(3): 1-11.
• Tran LT, Ly NQ. Learning video stabilization using optical flow. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 13-19, 2020, Seattle, WA, USA, pp. 8159-8167.
• Werlberger M, Pock T, Unger M, Bischof H. Optical flow guided TV-L1 video interpolation and restoration. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, July 25-27, 2011, St. Petersburg, Russia, pp. 273-286.
• Yu Z, Li H, Wang Z, Hu Z, Chen CW. Multi-level video frame interpolation: Exploiting the interaction among different levels. IEEE Transactions on Circuits and Systems for Video Technology 2013; 23(7): 1235-1248.
• Meyer S, Wang O, Zimmer H, Grosse M, Sorkine-Hornung A. Phase-based frame interpolation for video. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7-12, 2015, Boston, MA, USA, pp. 1410-1418.
• Niklaus S, Mai L, Liu F. Video frame interpolation via adaptive convolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA, pp. 670-679.
• Niklaus S, Mai L, Liu F. Video frame interpolation via adaptive separable convolution. In 2017 IEEE International Conference on Computer Vision (ICCV), October 22-29, 2017, Venice, Italy, pp. 261-270.
• Liu Z, Yeh RA, Tang X, Liu Y, Agarwala A. Video frame synthesis using deep voxel flow. In 2017 IEEE International Conference on Computer Vision (ICCV), October 22-29, 2017, Venice, Italy, pp. 4463-4471.
• Niklaus S, Liu F. Context-aware synthesis for video frame interpolation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA, pp. 1701-1710.
• Wang M, Yang GY, Lin JK, Zhang SH, Shamir A, Lu SP, Hu SM. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing 2018; 28(5): 2283–2292.
• Xu SZ, Hu J, Wang M, Mu TJ, Hu SM. Deep video stabilization using adversarial networks. Computer Graphics Forum 2018; 37(7): 267-276.
• Yu J, Ramamoorthi R. Robust video stabilization by optimization in CNN weight space. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 15-20, 2019, Long Beach, CA, USA, pp. 3800-3808.
• Choi J, Kweon IS. Deep iterative frame interpolation for full-frame video stabilization. ACM Transactions on Graphics (TOG) 2020; 39(1): 1-9.
• Sarigul M, Karacan L. Classifying stable and unstable videos with deep convolutional networks. Journal of Intelligent Systems with Applications 2020; 3(2): 90-92.
• Guilluy W, Oudre L, Beghdadi A. Video stabilization: Overview, challenges and perspectives. Signal Processing: Image Communication 2021; 90: 116015.
• Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA, pp. 1874-1883.
• Ali MK, Yu S, Kim TH. Learning deep video stabilization without optical flow. arXiv preprint, 2011.09697, 2020.