| Title:
|
The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences (English) |
| Author:
|
Häming, Klaus |
| Author:
|
Peters, Gabriele |
| Language:
|
English |
| Journal:
|
Kybernetika |
| ISSN:
|
0023-5954 |
| Volume:
|
46 |
| Issue:
|
5 |
| Year:
|
2010 |
| Pages:
|
926-937 |
| Summary lang:
|
English |
| . |
| Category:
|
math |
| . |
| Summary:
|
The problem addressed in this paper is the reconstruction of an object in the form of a realistically textured 3D model from images taken with an uncalibrated camera. We especially focus on reconstructions from short image sequences. By means of a description of an easy to use system, which is able to accomplish this in a fast and reliable way, we give a survey of all steps of the reconstruction pipeline. For the purpose of developing a coherent reconstruction system it is necessary to integrate a number of different techniques such as feature detection, algorithms of the RANSAC-family, and methods for auto-calibration. We describe and review recent developments of distinct strands of these techniques. While developing our system the necessity of improvements of several steps of the state-of-the-art reconstruction pipeline emerged. Two of these innovations are introduced in detail in this paper: an advanced SIFT-based feature detector and a two-stage RANSAC process facilitating a faster selection of relevant object points. In addition, we give a recommendation regarding auto-calibration for short image sequences. (English) |
| Keyword:
|
structure from motion |
| Keyword:
|
feature detection |
| Keyword:
|
RANSAC |
| Keyword:
|
auto-calibration |
| MSC:
|
68U05 |
| MSC:
|
68U10 |
| idZBL:
|
Zbl 1211.94006 |
| idMR:
|
MR2778920 |
| . |
| Date available:
|
2010-12-20T16:32:08Z |
| Last updated:
|
2013-09-22 |
| Stable URL:
|
http://hdl.handle.net/10338.dmlcz/141400 |
| . |
| Reference:
|
[1] Baumberg, A.: Reliable feature matching across widely separated views.In: IEEE Conf. on Computer Vision and Pattern Recognition 2000, Vol. 01, pp. 1774–1781. |
| Reference:
|
[2] Bay, H., Tuytelaars, T., Gool, L. Van: Surf: Speeded up robust features.In: 9th European Conference on Computer Vision, Graz 2006. |
| Reference:
|
[3] Beardsley, P. A., Torr, P. H. S., Zisserman, A.: 3d model acquisition from extended image sequences.In: ECCV ’96: Proc. 4th European Conference on Computer Vision-Volume II, Springer, London 1996, pp. 683–695. |
| Reference:
|
[4] Beis, J. S., Lowe, D. G.: Shape indexing using approximate nearest-neighbour search in high-dimensional spaces.In: Proc. IEEE Conf. Comp. Vision Patt. Recog 1997, pp. 1000–1006. |
| Reference:
|
[5] Birchfield, S., Tomasi, C.: Depth discontinuities by pixel-to-pixel stereo.Internat. J. Comput. Vision 3 (1999), 269–293. 10.1023/A:1008160311296 |
| Reference:
|
[6] Canny, J.: A computational approach to edge detection.IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986), 6, 679–698. 10.1109/TPAMI.1986.4767851 |
| Reference:
|
[7] Chum, O., Matas, J.: Matching with PROSAC – progressive sample consensus.In: Proc. Conference on Computer Vision and Pattern Recognition (C. Schmid, S. Soatto, and C. Tomasi, eds.), Vol. 1, Los Alamitos 2005, IEEE Computer Society, pp. 220–226. |
| Reference:
|
[8] Chum, O., Matas, J., Kittler, J.: Locally optimized ransac.In: DAGM-Symposium 2003, pp. 236–243. |
| Reference:
|
[9] Chum, O., Matas, J., Obdržálek, Š.: Enhancing RANSAC by generalized model optimization.In: Proc. Asian Conference on Computer Vision (ACCV) (K.-S. Hong and Z. Zhang, eds.), Vol. 2, Seoul 2004, Asian Federation of Computer Vision Societies, pp. 812–817. |
| Reference:
|
[10] Cox, I. J., Hingorani, S. L., Rao, S. B., Maggs, B. M.: A maximum likelihood stereo algorithm.Comput. Vis. Image Underst. 63 (1996), 3, 542–567. 10.1006/cviu.1996.0040 |
| Reference:
|
[11] Dellaert, F., Seitz, S. M., Thorpe, Ch. E., Thrun, S.: Structure from motion without correspondence.In: IEEE Conf. on Computer Vision and Pattern Recognition 2000, pp. 557–564. |
| Reference:
|
[12] Dempster, A. P., Laird, N. M., Rubin, D. B.: Maximum likelihood from incomplete data via the em algorithm.J. Roy. Statist. Soc. Ser. B 39 (1977), 1, 1–38. Zbl 0364.62022, MR 0501537 |
| Reference:
|
[13] Fischler, M. A., Bolles, R. C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Commun. ACM 24 (1981), 6, 381–395. MR 0618158, 10.1145/358669.358692 |
| Reference:
|
[14] Fitzgibbon, A. W., Zisserman, A.: Automatic 3D model acquisition and generation of new images from video sequences.In: Proc.European Signal Processing Conference (EUSIPCO ’98), Rhodes 1998, pp. 1261–1269. |
| Reference:
|
[15] Fitzgibbon, A. W., Zisserman, A.: Automatic camera recovery for closed or open image sequences.In: Proc. European Conference on Computer Vision 1998, pp. 311–326. |
| Reference:
|
[16] Frahm, J.-M., Pollefeys, M.: Ransac for (quasi-)degenerate data (qdegsac).In: CVPR ’06: Proc. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington 2006, IEEE Computer Society, pp. 453–460. |
| Reference:
|
[17] Friedman, J. H., Bentley, J. L., Finkel, R. A.: An algorithm for finding best matches in logarithmic expected time.ACM Trans. Math. Software 3 (1997), 3, 209–226. |
| Reference:
|
[18] Pollefeys, M., Gool, L J. Van, Meerbergen, G. Van, Vergauwen, M.: A hierarchical symmetric stereo algorithm using dynamic programming.Internat. J. Comput. Vision 47 (2002), 275–285. 10.1023/A:1014562312225 |
| Reference:
|
[19] Häming, K., Peters, G.: Extension of the generalized image rectification – Catching the infinity cases.In: Proc. 4th International Conference on Informatics in Control, Automation, and Robotics (ICINCO 2007) (J. Zaytoon, J.-L. Ferrier, J. A. Cetto, and J. Filipe, eds.), Vol. RA-2, Angers 2007, Institute for Systems and Technologies of Information, Control and Communication, pp. 275–279. |
| Reference:
|
[20] Harris, Ch., Stephens, M.: A combined Corner and Edge detector.In: 4th ALVEY Vision Conference 1988, pp. 147–151. |
| Reference:
|
[21] Hartley, R. I., Zisserman, A.: Multiple View Geometry in Computer Vision.Second edition. Cambridge University Press 2004. Zbl 1072.68104, MR 2059248 |
| Reference:
|
[22] Koch, R., Pollefeys, M., Gool, L. J. Van: Realistic surface reconstruction of 3d scenes from uncalibrated image sequences.J. Visualization and Computer Animation 11 (2000), 3, 115–127. 10.1002/1099-1778(200007)11:3<115::AID-VIS228>3.0.CO;2-2 |
| Reference:
|
[23] Lhuillier, M., Quan, L.: A quasi-dense approach to surface reconstruction from uncalibrated images.IEEE Trans. Pattern Analysis and Machine Intelligence 27 (2005), 3, 418–433. 10.1109/TPAMI.2005.44 |
| Reference:
|
[24] Lindeberg, T.: Feature detection with automatic scale selection.Internat. J. Comput. Vision 30 (1998), 2, 77–116. |
| Reference:
|
[25] Lindeberg, T., Bretzner, L.: Real-time scale selection in hybrid multi-scale representations.In: Proc. Scale-Space, Lect. Notes in Comput. Sci. 2695, Springer 2003, pp. 148–163. Zbl 1067.68753 |
| Reference:
|
[26] Lowe, D. G.: Distinctive image features from scale-invariant keypoints.Internat. J. Comput. Vision 60 (2004), 2, 91–110. 10.1023/B:VISI.0000029664.99615.94 |
| Reference:
|
[27] Lucas, B. D., Kanade, T.: An iterative image registration technique with an application to stereo vision.In: IJCAI81, pp. 674–679. |
| Reference:
|
[28] Mikolajczyk, K., Schmid, C.: Scale and affine invariant interest point detectors.Internat. J. Comput. Vision 60 (2004), 1, 63–86. 10.1023/B:VISI.0000027790.02288.f2 |
| Reference:
|
[29] Peters, G., Häming, K.: Fast freehand acquisition of 3d objects and their visualization.J. Commun. Comput. 7 (2010), 2–3. |
| Reference:
|
[30] Pollefeys, M., Gool, L. Van, Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koch, R.: Visual modeling with a hand-held camera.Internat. J. Comput. Vision 59 (2004), 3, 207–232. 10.1023/B:VISI.0000025798.50602.3a |
| Reference:
|
[31] Pollefeys, M., Koch, R., Gool, L. J. van: Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters.In: ICCV 1998, pp. 90–95. |
| Reference:
|
[32] Pollefeys, M., Koch, R., Gool, L. J. van: A simple and efficient rectification method for general motion.In: Proc. Internat. Conference on Computer Vision (ICCV 1999), pp. 496–501. |
| Reference:
|
[33] Pollefeys, M., Verbiest, F., Gool, L. Van: Surviving dominant planes in uncalibrated structure and motion recovery.In: Computer Vision – ECCV 2002, 7th European Conference on Computer Vision (Johansen, ed.). Lect. Notes Comput. Sci. 2351, Springer-Verlag 2002, pp. 837–851. |
| Reference:
|
[34] Pollefeys, M., Vergauwen, M., Cornelis, K., Tops, J., Verbiest, F., Structure, L. Van Gool., In, motion from image sequences.: Proc.Conference on Optical 3-D Measurement Techniques V (K. Gruen, ed.), Vienna 2001. pp. 251–258. |
| Reference:
|
[35] Ponce, J., Papadopoulo, T., Teillaud, M., Triggs, B.: On the absolute quadratic complex and its application to autocalibration.In: IEEE Conference on Computer Vision & Pattern Recognition 2005, Vol. I., pp. 780–787. |
| Reference:
|
[36] Prasad, M., Fitzgibbon, A. W.: Single view reconstruction of curved surfaces.In: IEEE Conf. on Computer Vision and Pattern Recognition 2006, Vol. 02, pp. ,1345–1354. |
| Reference:
|
[37] Saxena, A., Sun, M., Ng, A. Y.: Make3d: Depth perception from a single still image.In: AAAI (D. Fox and C. P. Gomes, eds.), AAAI Press 2008, pp. 1571–1576. |
| Reference:
|
[38] Schaffalitzky, F., Zisserman, A.: Multi-view matching for unordered image sets, or “How do I organize my holiday snaps?”.In: Proc. 7th European Conference on Computer Vision, Copenhagen 2002, Springer, Vol. 1, pp. 414–431. Zbl 1034.68662 |
| Reference:
|
[39] Schaffalitzky, F., Zisserman, A., Hartley, R. I., Torr, P. H. S.: A six point solution for structure and motion.In: ECCV ’00: Proc. 6th European Conference on Computer Vision, Vol. I, London 2000, Springer, pp. 632–648. |
| Reference:
|
[40] Shen, F., Wang, H.: A local edge detector used for finding corners.Proc. ICICS, 2001. |
| Reference:
|
[41] Shi, J., Tomasi, C.: Good features to track.In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), Seattle 1994. |
| Reference:
|
[42] Snavely, N., Seitz, S. M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d.ACM Trans. on Graphics (SIGGRAPH Proc.), 25 (2006), 3, 835–846. 10.1145/1141911.1141964 |
| Reference:
|
[43] Torr, P. H. S., Zisserman, A.: Mlesac: a new robust estimator with application to estimating image geometry.Comput. Vis. Image Underst. 78 (2000), 1, 138–156. 10.1006/cviu.1999.0832 |
| Reference:
|
[44] Triggs, B.: Autocalibration and the absolute quadric.In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico 1977, IEEE Computer Society Press, pp. 609–614. |
| Reference:
|
[45] Tsai, R. Y.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses.In: Radiometry (L. B. Wolff, S. A. Shafer, and G. Healey, eds.), Jones and Bartlett Publishers, Inc., pp. 221–244, 1992. |
| Reference:
|
[46] Vergauwen, M., Gool, L. Van: Web-based 3d reconstruction service.Mach. Vision Appl. 17 (2006), 6, 411–426. 10.1007/s00138-006-0027-1 |
| Reference:
|
[47] Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features.IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2001, Vol. 1, 511. |
| Reference:
|
[48] Woodford, O. J., Torr, P. H. S., Reid, I. D., Fitzgibbon, A. W.: Global stereo reconstruction under second order smoothness priors.In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Anchorage 2008. |
| Reference:
|
[49] Zhang, Z.: A flexible new technique for camera calibration.IEEE Trans. Pattern Analysis and Machine Intelligence 22 (1998), 1330–1334. 10.1109/34.888718 |
| . |
