Difference between revisions of "Real-Time Optical Flow Using Neural Networks"
From iis-projects
(→Practical Details) |
m |
||
(25 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[[File:OptFlowBus.jpg|500px|thumb]] | [[File:OptFlowBus.jpg|500px|thumb]] | ||
− | [[File:OptFlowTennis.png|400px|thumb]] | + | [[File:OptFlowTennis.png|400px|thumb|source: [[http://lmb.informatik.uni-freiburg.de/Publications/2011/Bro11a/ link]]]] |
+ | [[File:OptFlowCar.gif|300px|thumb|source: [[http://people.csail.mit.edu/celiu/OpticalFlow/ link]]]] | ||
+ | [[File:OptFlowCarOut.jpg|300px|thumb|source: [[http://people.csail.mit.edu/celiu/OpticalFlow/ link]]]] | ||
==Short Description== | ==Short Description== | ||
− | Optical flow | + | Optical flow is an essential ingredient to many complex computer vision systems. There are already quite a few algorithms to determine the optical flow between two frames very accurately. |
Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) in both images to find correspondences and then solve complex optimization problems to find reasonable solutions between these key points. | Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) in both images to find correspondences and then solve complex optimization problems to find reasonable solutions between these key points. | ||
Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach where computer vision is most interesting: on low-power and mobile platforms, in cars, ... | Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach where computer vision is most interesting: on low-power and mobile platforms, in cars, ... | ||
− | We would like to take an unconventional approach to improve on this, training deep convolutional neural networks to calculate optical flow. The biggest issue of neural networks is usually to get a large enough, (hand-)labeled dataset. This is not a problem here, since we can take a high-quality algorithm's output as our reference | + | We would like to take an unconventional approach to improve on this, training deep convolutional neural networks to calculate optical flow. The biggest issue of neural networks is usually to get a large enough, (hand-)labeled dataset. This is not a problem here, since we can take a high-quality algorithm's output as our reference result and require only some raw video data. |
As part of this project, the student(s) will: | As part of this project, the student(s) will: | ||
Line 14: | Line 16: | ||
* Implement the most promissing candidate(s) (or make use of available implementations) | * Implement the most promissing candidate(s) (or make use of available implementations) | ||
* Learn the fundamentals of convolutional neural networks (ConvNets) | * Learn the fundamentals of convolutional neural networks (ConvNets) | ||
− | * Get to know and use the Torch framework to train and adapt | + | * Get to know and use the Torch framework to train and adapt a ConvNet to this problem |
− | * Evaluate the results (accuracy, performance) | + | * Evaluate the results thoroughly (accuracy, performance) |
* If there is time left: make a real-time demonstrator (camera interface, working platform, ... is already available) | * If there is time left: make a real-time demonstrator (camera interface, working platform, ... is already available) | ||
− | The project can be adapted to the preferences of the student(s). Just drop me an email or stop by my office if you .. | + | The project can be adapted to the preferences of the student(s). Just drop me an email or stop by my office if you ... |
* ... want to know more, | * ... want to know more, | ||
* ... are uncertain whether this project suits you, or | * ... are uncertain whether this project suits you, or | ||
* ... if you have any other question. | * ... if you have any other question. | ||
− | + | ===Status: Completed=== | |
− | ===Status: Available === | + | : Christoph Weilenmann |
− | : Looking for 1 Master or 2 semester project students | + | : Autumn Semester 2015 |
− | : Contact: [[:User:Lukasc | Lukas Cavigelli]] | + | : Contact/Supervision: [[:User:Lukasc | Lukas Cavigelli]] |
− | + | [[Category:Digital]] [[Category:Software]] [[Category:Completed]] [[Category:Semester Thesis]] [[Category:Lukasc]] [[Category:2015]] | |
+ | <!-- | ||
+ | ===Status: {Available, Reserved, In Progress, Completed}=== | ||
+ | : Looking for 1 Master or (2 or 1) semester project students (work load will be adjusted) | ||
+ | : Fall Semester 2014 (sem13h2) | ||
+ | : Contact/Supervision: [[:User:Lukasc | Lukas Cavigelli]] | ||
+ | [[Category:Digital]] [[Category:System Design]] | ||
+ | [[Category:Available]] [[Category:In progress]] [[Category:Completed]] [[Category:Hot]] | ||
+ | [[Category:Semester Thesis]] [[Category:Master Thesis]] [[Category:PhD Thesis]] [[Category:Research]] | ||
+ | [[Category:2014]] | ||
+ | [[Category:Lukasc]] | ||
+ | ---> | ||
===Prerequisites=== | ===Prerequisites=== | ||
: Knowledge of Matlab and/or C/C++ | : Knowledge of Matlab and/or C/C++ | ||
: Interest in video processing and machine learning | : Interest in video processing and machine learning | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
===Character=== | ===Character=== | ||
: 30% Theory / Literature Research | : 30% Theory / Literature Research | ||
− | : 70% | + | : 70% Software Prototyping, Experimentation |
===Professor=== | ===Professor=== | ||
Line 56: | Line 59: | ||
* '''[[Project Plan]]''' | * '''[[Project Plan]]''' | ||
* '''[[Project Meetings]]''' | * '''[[Project Meetings]]''' | ||
− | * '''[[Design Review]]''' (ASIC | + | * '''[[Design Review]]''' (ASIC designs only) |
− | * '''[[Coding Guidelines]]''' (ASIC & FPGA | + | * '''[[Coding Guidelines]]''' (ASIC & FPGA designs only) |
* '''[[Final Report]]''' | * '''[[Final Report]]''' | ||
* '''[[Final Presentation]]''' | * '''[[Final Presentation]]''' | ||
Line 65: | Line 68: | ||
==Links== | ==Links== | ||
If you have a look at these, please don't worry too much about the math involved. The underlying concept is often much simpler, and I will help you with understand whatever you need. | If you have a look at these, please don't worry too much about the math involved. The underlying concept is often much simpler, and I will help you with understand whatever you need. | ||
+ | * Comparison of various optical flow algorithms with the corresponding benchmark dataset [[http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=flow website]] | ||
+ | * Artificial optical flow dataset/movie: [[http://sintel.is.tue.mpg.de/ website]], natural stereo dataset: [[http://vision.middlebury.edu/flow/data/ website]], [[http://www.cvc.uab.es/adas/site/?q=node/36 website]], artificial optical flow dataset: [[http://www.cvc.uab.es/adas/site/?q=node/59 website]] | ||
* Motion tracking/optical flow using keypoint matching [[http://lmb.informatik.uni-freiburg.de/Publications/2011/Bro11a/brox_tpami10_ldof.pdf paper]] | * Motion tracking/optical flow using keypoint matching [[http://lmb.informatik.uni-freiburg.de/Publications/2011/Bro11a/brox_tpami10_ldof.pdf paper]] | ||
* PhD thesis focusing on motion tracking/optical flow using keypoint matching (SIFT-Flow): [[http://people.csail.mit.edu/celiu/OpticalFlow/ website]], [[http://people.csail.mit.edu/celiu/Thesis/CePhDThesis.pdf thesis]] | * PhD thesis focusing on motion tracking/optical flow using keypoint matching (SIFT-Flow): [[http://people.csail.mit.edu/celiu/OpticalFlow/ website]], [[http://people.csail.mit.edu/celiu/Thesis/CePhDThesis.pdf thesis]] | ||
+ | * A nice starting point: [[https://hal.inria.fr/file/index/docid/873592/filename/DeepFlow_iccv2013.pdf paper]] | ||
* Correlation-based, variational optical flow: [[http://www.cs.bgu.ac.il/~rba/R_Ben-Ari_Aligned_OF_JMIV09.pdf paper]] | * Correlation-based, variational optical flow: [[http://www.cs.bgu.ac.il/~rba/R_Ben-Ari_Aligned_OF_JMIV09.pdf paper]] | ||
* The Torch framework: [[http://torch.ch/ website]] | * The Torch framework: [[http://torch.ch/ website]] | ||
Line 72: | Line 78: | ||
[[#top|↑ top]] | [[#top|↑ top]] | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Latest revision as of 11:22, 5 February 2016
Contents
Short Description
Optical flow is an essential ingredient to many complex computer vision systems. There are already quite a few algorithms to determine the optical flow between two frames very accurately. Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) in both images to find correspondences and then solve complex optimization problems to find reasonable solutions between these key points.
Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach where computer vision is most interesting: on low-power and mobile platforms, in cars, ...
We would like to take an unconventional approach to improve on this, training deep convolutional neural networks to calculate optical flow. The biggest issue of neural networks is usually to get a large enough, (hand-)labeled dataset. This is not a problem here, since we can take a high-quality algorithm's output as our reference result and require only some raw video data.
As part of this project, the student(s) will:
- Explore existing optical flow algorithms
- Implement the most promissing candidate(s) (or make use of available implementations)
- Learn the fundamentals of convolutional neural networks (ConvNets)
- Get to know and use the Torch framework to train and adapt a ConvNet to this problem
- Evaluate the results thoroughly (accuracy, performance)
- If there is time left: make a real-time demonstrator (camera interface, working platform, ... is already available)
The project can be adapted to the preferences of the student(s). Just drop me an email or stop by my office if you ...
- ... want to know more,
- ... are uncertain whether this project suits you, or
- ... if you have any other question.
Status: Completed
- Christoph Weilenmann
- Autumn Semester 2015
- Contact/Supervision: Lukas Cavigelli
Prerequisites
- Knowledge of Matlab and/or C/C++
- Interest in video processing and machine learning
Character
- 30% Theory / Literature Research
- 70% Software Prototyping, Experimentation
Professor
Detailed Task Description
This will be written after talking to the interested students, taking into account their interests and existing knowledge.
Goals
Practical Details
- Project Plan
- Project Meetings
- Design Review (ASIC designs only)
- Coding Guidelines (ASIC & FPGA designs only)
- Final Report
- Final Presentation
Results
Links
If you have a look at these, please don't worry too much about the math involved. The underlying concept is often much simpler, and I will help you with understand whatever you need.
- Comparison of various optical flow algorithms with the corresponding benchmark dataset [website]
- Artificial optical flow dataset/movie: [website], natural stereo dataset: [website], [website], artificial optical flow dataset: [website]
- Motion tracking/optical flow using keypoint matching [paper]
- PhD thesis focusing on motion tracking/optical flow using keypoint matching (SIFT-Flow): [website], [thesis]
- A nice starting point: [paper]
- Correlation-based, variational optical flow: [paper]
- The Torch framework: [website]