Newer
Older
The repository for NI-MVI semestral work on video resolution upscaling.
Vezměte krátké 5-10 sekundové video a vytvořte generátor, který bude generovat
video s vyšším rozlišením. Používejte obě architektury: GAN a U-Net.
## Report
## SR of the precipitation video
## Milestone
The objective of this work is to 4x upscale the resolution of `data/target.mp4` video. The video contains 24 hours of weather radar precipitation data with a resolution of 480x270 pixels. Weather radar data is generally noisy, which poses a secondary challenge of denoising the data during SR.
### Planned approach to the semestral work
The upscaling of video resolution can be decomposed into upscaling of individual frames. Thus, I will focus on image super-resolution (SR) ML models and use the best performing one to generate target video.
I plan to build and train the following models:
* U-net, initially motivated by my bachelor's thesis [1]. From my experience, this architecture is able to pick the low hanging fruits in various computer vision tasks. The U-net described in [2] won second place at NTIRE2019 challenge [4], which supports this claim. I plan to utilize the findings from [2] for the training of the U-net.
* SRGAN [5], which is the first utilization of the GAN framework for the SR task.
I will use the DIV2K dataset [6] for training. For validation, I will use the Set14 benchmark [7], and weather radar validation set created for this work. I will use both qualitative evaluation and quantitative evaluation with PSNR and SSIM metrics.
I am posing the following questions:
* Can SR model trained on camera images generate weather radar images?
* Does evaluation on Set14 benchmark correlate with the weather radar validation set?
* Can training or finetuning on weather radar data improve the performance?
>All of the weather radar data was provided by the Czech company [Meteopress](https://www.meteopress.cz/).
### Literature
* [[1]](https://dspace.cvut.cz/bitstream/handle/10467/83134/F8-BP-2019-Choma-Matej-thesis.pdf) Choma, Matej. *Interpolation and Extrapolation of Subsequent Weather Radar Images.*
* [[2]](https://openaccess.thecvf.com/content_CVPRW_2019/papers/NTIRE/Feng_Suppressing_Model_Overfitting_for_Image_Super-Resolution_Networks_CVPRW_2019_paper.pdf) Feng, Ruicheng, et al. *Suppressing model overfitting for image super-resolution networks.*
* [[3]](http://de.arxiv.org/pdf/2005.01996) Lugmayr, Andreas, Martin Danelljan, and Radu Timofte. *Ntire 2020 challenge on real-world image super-resolution: Methods and results.*
* [[4]](https://openaccess.thecvf.com/content_CVPRW_2019/papers/NTIRE/Cai_NTIRE_2019_Challenge_on_Real_Image_Super-Resolution_Methods_and_Results_CVPRW_2019_paper.pdf) Cai, Jianrui, et al. *Ntire 2019 challenge on real image super-resolution: Methods and results.*
* [[5]](https://arxiv.org/pdf/1609.04802.pdf) Ledig, Christian, et al. *Photo-realistic single image super-resolution using a generative adversarial network.*
* [[6]](https://data.vision.ee.ethz.ch/cvl/DIV2K/) Agustsson, Eirikur and Timofte, Radu. *NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study.*
* [[7]](http://www.cs.technion.ac.il/users/wwwb/cgi-bin/tr-get.cgi/2010/CS/CS-2010-12.pdf) Zeyde, Roman, Michael Elad, and Matan Protter. *On single image scale-up using sparse-representations.*