Difference between revisions of "Real-Time Optical Flow Using Neural Networks"
From iis-projects
(Created page with "500px|thumb 400px|thumb ==Short Description== Optical flow data is an ingredient to many complex computer vision systems. T...") |
|||
| Line 3: | Line 3: | ||
==Short Description== | ==Short Description== | ||
| − | Optical flow data is an ingredient to many complex computer vision systems. There are already quite a few algorithms to | + | Optical flow data is an ingredient to many complex computer vision systems. There are already quite a few algorithms to determine the optical flow between two frames very well. |
| − | Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) and then solve complex optimization problems to find reasonable solutions between these key points. | + | Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) in both images to find correspondences and then solve complex optimization problems to find reasonable solutions between these key points. |
Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach, particularly on low-power and mobile platforms, where computer vision is most interesting. | Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach, particularly on low-power and mobile platforms, where computer vision is most interesting. | ||
| − | We would like to take an unconventional approach to | + | We would like to take an unconventional approach to improve on this, training deep convolutional neural networks to calculate optical flow. The biggest issue of neural networks is usually to get a large enough, (hand-)labeled dataset. This is not a problem here, since we can take a high-quality algorithm's output as our reference input and getting raw video data without a lot of constraints is not really an issue. |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | As part of this project, the student(s) will: | ||
| + | * Explore existing optical flow algorithms | ||
| + | * Implement the most promissing candidate(s) (or make use of available implementations) | ||
| + | * Learn the fundamentals of convolutional neural networks (ConvNets) | ||
| + | * Look for a suitable video data source | ||
| + | * Use the Torch framework to train and adapt his ConvNet | ||
| + | * If there is time left: make a real-time demonstrator (camera interface, working platform, ... is already available) | ||
| + | The project can be adapted to the preferences of the students. Just drop me an email or stop by my office if you .. | ||
| + | * ... want to know more, | ||
| + | * ... are uncertain whether this project suits you, or | ||
| + | * ... if you have any other question. | ||
===Status: Available === | ===Status: Available === | ||
| − | : Looking for 1 Master or 2 semester project students | + | : Looking for 1 Master or 2 semester project students |
: Contact: [[:User:Lukasc | Lukas Cavigelli]] | : Contact: [[:User:Lukasc | Lukas Cavigelli]] | ||
| Line 24: | Line 30: | ||
: Knowledge of Matlab and/or C/C++ | : Knowledge of Matlab and/or C/C++ | ||
: Interest in video processing and machine learning | : Interest in video processing and machine learning | ||
| − | |||
<!-- | <!-- | ||
===Status: Completed === | ===Status: Completed === | ||
| Line 38: | Line 43: | ||
===Character=== | ===Character=== | ||
: 20% Theory / Literature Research | : 20% Theory / Literature Research | ||
| − | : | + | : 80% software development |
| − | |||
===Professor=== | ===Professor=== | ||
| Line 59: | Line 63: | ||
==Links== | ==Links== | ||
| + | [[http://lmb.informatik.uni-freiburg.de/Publications/2011/Bro11a/brox_tpami10_ldof.pdf Motion tracking/optical flow using keypoint matching]] | ||
| + | [[http://people.csail.mit.edu/celiu/Thesis/CePhDThesis.pdf PhD thesis focusing on motion tracking/optical flow using keypoint matching (SIFT-Flow)]] | ||
| + | [[http://www.cs.bgu.ac.il/~rba/R_Ben-Ari_Aligned_OF_JMIV09.pdf Correlation-based, variational optical flow (don't worry about the math!)]] | ||
[[#top|↑ top]] | [[#top|↑ top]] | ||
Revision as of 01:21, 9 February 2015
Contents
Short Description
Optical flow data is an ingredient to many complex computer vision systems. There are already quite a few algorithms to determine the optical flow between two frames very well. Some of these approaches use spatial correlation of small image patches, others work by looking at few well-trackable points (corners, edges, ...) in both images to find correspondences and then solve complex optimization problems to find reasonable solutions between these key points.
Evaluation of these algorithms is very time-consuming, often taking many seconds per frame on powerful workstations. This puts real-time applications out of reach, particularly on low-power and mobile platforms, where computer vision is most interesting.
We would like to take an unconventional approach to improve on this, training deep convolutional neural networks to calculate optical flow. The biggest issue of neural networks is usually to get a large enough, (hand-)labeled dataset. This is not a problem here, since we can take a high-quality algorithm's output as our reference input and getting raw video data without a lot of constraints is not really an issue.
As part of this project, the student(s) will:
- Explore existing optical flow algorithms
- Implement the most promissing candidate(s) (or make use of available implementations)
- Learn the fundamentals of convolutional neural networks (ConvNets)
- Look for a suitable video data source
- Use the Torch framework to train and adapt his ConvNet
- If there is time left: make a real-time demonstrator (camera interface, working platform, ... is already available)
The project can be adapted to the preferences of the students. Just drop me an email or stop by my office if you ..
- ... want to know more,
- ... are uncertain whether this project suits you, or
- ... if you have any other question.
Status: Available
- Looking for 1 Master or 2 semester project students
- Contact: Lukas Cavigelli
Prerequisites
- Knowledge of Matlab and/or C/C++
- Interest in video processing and machine learning
Character
- 20% Theory / Literature Research
- 80% software development
Professor
Detailed Task Description
Goals
Practical Details
Results
Links
[Motion tracking/optical flow using keypoint matching] [PhD thesis focusing on motion tracking/optical flow using keypoint matching (SIFT-Flow)] [Correlation-based, variational optical flow (don't worry about the math!)]
