Commit 246339aa authored by Matej Choma's avatar Matej Choma

milestone commit

parent fc90d1e5
data/benchmark
data/radar
notes.md
**/.ipynb_checkpoints/
\ No newline at end of file
# NI-MVI-sp-2020-chomamat # Upscaling of weather radar video resolution
**NI-MVI-sp-2020-chomamat**
The repository for NI-MVI semestral work on video resolution upscaling. The repository for NI-MVI semestral work on video resolution upscaling.
## Assignment ### Assignment
Vezměte krátké 5-10 sekundové video a vytvořte generátor, který bude generovat Vezměte krátké 5-10 sekundové video a vytvořte generátor, který bude generovat
video s vyšším rozlišením. Používejte obě architektury: GAN a U-Net. video s vyšším rozlišením. Používejte obě architektury: GAN a U-Net.
Porovnejte výsledky. Porovnejte výsledky.
\ No newline at end of file
## Milestone
The objective of this work is to 4x upscale the resolution of `data/target.mp4` video. The video contains 24 hours of weather radar precipitation data with a resolution of 480x270 pixels. Weather radar data is generally noisy, which poses a secondary challenge of denoising the data during SR.
![Target Video](data/target.mp4)
### Planned approach to the semestral work
The upscaling of video resolution can be decomposed into upscaling of individual frames. Thus, I will focus on image super-resolution (SR) ML models and use the best performing one to generate target video.
I plan to build and train the following models:
* U-net, initially motivated by my bachelor's thesis [1]. From my experience, this architecture is able to pick the low hanging fruits in various computer vision tasks. The U-net described in [2] won second place at NTIRE2019 challenge [4], which supports this claim. I plan to utilize the findings from [2] for the training of the U-net.
* SRGAN [5], which is the first utilization of the GAN framework for the SR task.
I will use the DIV2K dataset [6] for training. For validation, I will use the Set14 benchmark [7], and weather radar validation set created for this work. I will use both qualitative evaluation and quantitative evaluation with PSNR and SSIM metrics.
![weather radar image](data/examples/radar.png "1920x1080 weather radar image")
I am posing the following questions:
* Can SR model trained on camera images generate weather radar images?
* Does evaluation on Set14 benchmark correlate with the weather radar validation set?
* Can training or finetuning on weather radar data improve the performance?
>All of the weather radar data was provided by the Czech company [Meteopress](https://www.meteopress.cz/).
### Literature
* [[1]](https://dspace.cvut.cz/bitstream/handle/10467/83134/F8-BP-2019-Choma-Matej-thesis.pdf) Choma, Matej. *Interpolation and Extrapolation of Subsequent Weather Radar Images.*
* [[2]](https://openaccess.thecvf.com/content_CVPRW_2019/papers/NTIRE/Feng_Suppressing_Model_Overfitting_for_Image_Super-Resolution_Networks_CVPRW_2019_paper.pdf) Feng, Ruicheng, et al. *Suppressing model overfitting for image super-resolution networks.*
* [[3]](http://de.arxiv.org/pdf/2005.01996) Lugmayr, Andreas, Martin Danelljan, and Radu Timofte. *Ntire 2020 challenge on real-world image super-resolution: Methods and results.*
* [[4]](https://openaccess.thecvf.com/content_CVPRW_2019/papers/NTIRE/Cai_NTIRE_2019_Challenge_on_Real_Image_Super-Resolution_Methods_and_Results_CVPRW_2019_paper.pdf) Cai, Jianrui, et al. *Ntire 2019 challenge on real image super-resolution: Methods and results.*
* [[5]](https://arxiv.org/pdf/1609.04802.pdf) Ledig, Christian, et al. *Photo-realistic single image super-resolution using a generative adversarial network.*
* [[6]](https://data.vision.ee.ethz.ch/cvl/DIV2K/) Agustsson, Eirikur and Timofte, Radu. *NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study.*
* [[7]](http://www.cs.technion.ac.il/users/wwwb/cgi-bin/tr-get.cgi/2010/CS/CS-2010-12.pdf) Zeyde, Roman, Michael Elad, and Matan Protter. *On single image scale-up using sparse-representations.*
import math
import matplotlib.pyplot as plt
import numpy as np
def _squeeze(x):
_x = np.squeeze(x)
while _x.ndim < 3:
_x = _x[None, ...]
return _x
def grid(X, cols=None, size=(15, 10), title=None, vmax=None, cmap=plt.cm.nipy_spectral):
x = _squeeze(np.array(X))
rows = math.ceil(len(x) / cols)
fig, axes = plt.subplots(nrows=rows, ncols=cols,
figsize=(cols*size[0], rows*size[1]),
sharex=True, sharey=True)
ax = axes.ravel()
for i in range(rows):
for j in range(cols):
if i*cols + j < len(x):
ax[i*cols + j].imshow(x[i*cols+j], cmap=cmap, vmax=vmax)
else:
break
if title is not None:
fig.suptitle(title, fontsize=30)
fig.tight_layout(rect=[0, 0.03, 1, 0.93])
else:
fig.tight_layout()
def show(x, size=(15, 10), title=None, vmax=None, cmap=plt.cm.nipy_spectral, **kwargs):
plt.figure(figsize=size)
if title is not None:
plt.title(title)
if (x.ndim == 2):
return plt.imshow(x, cmap=cmap, vmax=vmax, **kwargs)
else:
return plt.imshow(x, **kwargs)
plt.tight_layout()
\ No newline at end of file
File added
This diff is collapsed.
%% LaTeX template class for technical report
%% for subject MVI in Czech Technical University in Prague (CTU)
%%
%% (c) 2008 Vit Zyka
%%
%% History:
%% 0.1 2008-09-21 new for NRP
%% 0.2 2008-11-12 href for email, unicode in hyperref; tolerance; microtype; cmap
%% \texorpdfstring
%% 0.3 2018-02-12 update for purposes of MI-MVI
\typeout{Class mvi-report v0.3, 2018-02-12, Martin Slapak, Vit Zyka}
%% Options...
\newif\ifCzech
\DeclareOption{czech}{\Czechtrue}
\ProcessOptions
\LoadClass[a4paper,normalheadings]{scrartcl}
%% Dimensions...
\setlength{\columnsep}{10mm}
\renewcommand{\topfraction}{0.9}
\renewcommand{\bottomfraction}{0.9}
\renewcommand{\dbltopfraction}{0.9}
\renewcommand{\textfraction}{0}
\setcounter{topnumber}{4}
\setcounter{bottomnumber}{4}
\setcounter{totalnumber}{4}
\setcounter{dbltopnumber}{4}
\tolerance=300
%% Packages...
\RequirePackage{ifpdf}
\ifCzech
\RequirePackage[czech]{babel}
\fi
\RequirePackage{lmodern}
\RequirePackage{cmap}
\usepackage[T1]{fontenc}
%\usepackage{ae,aecompl}
\usepackage[left=15mm,right=15mm,top=18mm,bottom=20mm,footskip=10mm]{geometry}
\ifpdf
\RequirePackage[pdftex]{graphicx}
\RequirePackage[pdftex]{color}
\else
\RequirePackage{graphicx}
\RequirePackage{color}
\fi
\RequirePackage[pdfpagemode=UseNone,pdfborder=0 0 0,unicode]{hyperref}
\RequirePackage{mflogo}
\RequirePackage{natbib}
\RequirePackage{microtype}
%% Title...
\newtoks\MVIemail \def\email#1{\MVIemail={#1}}
\newtoks\MVIaffiliation \def\affiliation#1{\MVIaffiliation={#1}}
\def\affiliationFont{\normalfont\fontsize{12pt}{14pt}\selectfont}
\AtBeginDocument{%
\expandafter\author\expandafter{%
\@author\\[4pt]\affiliationFont
\def\\{\crcr\affiliationFont}\the\MVIaffiliation\\
\href{mailto:\the\MVIemail}{\affiliationFont\the\MVIemail}}%
\expandafter\date\expandafter{\expandafter\affiliationFont\@date}%
}
\let\maketitleOrig=\maketitle
\def\maketitle{\@ifnextchar[{\domaketitle}{\domaketitle[]}}
\def\domaketitle[#1]{%
\twocolumn[\vskip-2.0cm\maketitleOrig\begin{quotation}#1\end{quotation}\vskip-1.0cm]%
}
%\bigskip
% pokus o usetreni mista v \paragraph
\let\paragraphOrig=\paragraph
\renewcommand\paragraph{
\@startsection
{paragraph} % the name
{4} % the level
{\z@} % the indent
{1.5ex \@plus1ex \@minus.2ex} % the before skip
{-1em} % the after skip
{\normalfont\normalsize\bfseries} % the style
}
\renewcommand{\baselinestretch}{1}
\setlength{\parskip}{-0.3ex}
%% Bibliography...
% citace dle abecedy
\bibliographystyle{plain}
% citace dle poradi, ale nevejde se to
%\bibliographystyle{is-unsrt}
% citace dle poradi, ale vejde se to ;-)
%\bibliographystyle{unsrt}
%% Misc...
\def\CONTEXT{\texorpdfstring{Con\kern-.1em\TeX{}t}{ConTeXt}}
\def\MetaPost{\texorpdfstring{\MP}{MetaPost}}
\endinput
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\
% pokud nechcem aby nam to zmensovalo pismenka v nazvech titulu,
% tak to obklopit chlupatejma zavorkama
@misc{ zizka,
title = {Some interesting source},
author = { Jan {\v Z}i{\v z}ka},
year = {1415},
howpublished = {online},
note = {[cit.~2018--10--11] \url{http://zizka.trocnov/husiti/conspiration.pdf} }
}
\relax
\providecommand\hyper@newdestlabel[2]{}
\providecommand*\new@tpo@label[2]{}
\providecommand\HyperFirstAtBeginDocument{\AtBeginDocument}
\HyperFirstAtBeginDocument{\ifx\hyper@anchor\@undefined
\global\let\oldcontentsline\contentsline
\gdef\contentsline#1#2#3#4{\oldcontentsline{#1}{#2}{#3}}
\global\let\oldnewlabel\newlabel
\gdef\newlabel#1#2{\newlabelxx{#1}#2}
\gdef\newlabelxx#1#2#3#4#5#6{\oldnewlabel{#1}{{#2}{#3}}}
\AtEndDocument{\ifx\hyper@anchor\@undefined
\let\contentsline\oldcontentsline
\let\newlabel\oldnewlabel
\fi}
\fi}
\global\let\hyper@last\relax
\gdef\HyperFirstAtBeginDocument#1{#1}
\providecommand\HyField@AuxAddToFields[1]{}
\providecommand\HyField@AuxAddToCoFields[2]{}
\bibstyle{plain}
\citation{zizka}
\bibdata{reference}
\@writefile{toc}{\contentsline {section}{\numberline {1}\IeC {\'U}vod}{1}{section.1}}
\@writefile{toc}{\contentsline {section}{\numberline {2}Vstupn\IeC {\'\i } data}{1}{section.2}}
\@writefile{toc}{\contentsline {section}{\numberline {3}Metody}{1}{section.3}}
\@writefile{toc}{\contentsline {section}{\numberline {4}V\IeC {\'y}sledky}{1}{section.4}}
\@writefile{lof}{\contentsline {figure}{\numberline {1}{\ignorespaces Vliv parametru \emph {Y}}}{1}{figure.1}}
\newlabel{fig:par-y}{{1}{1}{Vliv parametru \emph {Y}}{figure.1}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {2}{\ignorespaces Vliv parametru \emph {X}}}{1}{figure.2}}
\newlabel{fig:par-x}{{2}{1}{Vliv parametru \emph {X}}{figure.2}{}}
\@writefile{toc}{\contentsline {section}{\numberline {5}Z\IeC {\'a}v\IeC {\v e}r}{1}{section.5}}
This diff is collapsed.
\BOOKMARK [1][-]{section.1}{\376\377\000\332\000v\000o\000d}{}% 1
\BOOKMARK [1][-]{section.2}{\376\377\000V\000s\000t\000u\000p\000n\000\355\000\040\000d\000a\000t\000a}{}% 2
\BOOKMARK [1][-]{section.3}{\376\377\000M\000e\000t\000o\000d\000y}{}% 3
\BOOKMARK [1][-]{section.4}{\376\377\000V\000\375\000s\000l\000e\000d\000k\000y}{}% 4
\BOOKMARK [1][-]{section.5}{\376\377\000Z\000\341\000v\001\033\000r}{}% 5
%%
%% Created in 2018 by Martin Slapak
%%
%% Based on file for NRP report LaTeX class by Vit Zyka (2008)
%%
%% Compilation:
%% >pdflatex report
%% >bibtex report
%% >pdflatex report
%% >pdflatex report
\documentclass{mvi-report}
\usepackage[utf8]{inputenc}
\title{Precipitation Video Resolution Upscaling}
\author{Matej Choma}
\affiliation{ČVUT - FIT}
\email{chomamat@fit.cvut.cz}
\def\file#1{{\tt#1}}
\begin{document}
\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Úvod}
Definice problému/úkolu, který práce řeší\ldots
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Vstupní data}
Původ, proces získání, předzpracování, \ldots
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Metody}
Použité metody, jejich přizpůsobení, aplikace\ldots
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% --- VYSLEDKY
\section{Výsledky}
Jakých výsledků bylo dosaženo, co na ně melo vliv. Srovnání s očekáváním, \emph{diskuze nad výsledky} -- zvláště důležitá v případě, že něco vyšlo \emph{divně}.
\begin{figure}[h]
\centering\leavevmode
\includegraphics[width=.45\linewidth]{img/fit-logo-cz.pdf}\vskip-0.5cm
\caption{Vliv parametru \emph{Y}}
\label{fig:par-y}
\end{figure}
\begin{figure}[h]
\centering\leavevmode
\includegraphics[width=.45\linewidth]{img/fit-logo-en.pdf}\vskip-0.5cm
\caption{Vliv parametru \emph{X}}
\label{fig:par-x}
\end{figure}
Jak je z ilustrací \ref{fig:par-y} a \ref{fig:par-x} patrné, není to totéž, protože\ldots
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% --- ZAVER
\section{Závěr}
K čemu to bylo/je dobré, jak to půjde využít dále, co by šlo ještě vylepšit\ldots
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% --- Bibliography
\nocite{zizka}
%\bibliographystyle{plain-cz-online}
\bibliography{reference}
\end{document}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment