Image Stitching in EmguCV

How to Stitch Images Together In EmguCV

Advertisements

Since the first camera was invented, humans have taken millions and millions of photos. With all these images floating around, some people have wanted to create a way to combine their photos of relevant objects into one long and big image, better known as a panorama. Although there are multiple ways to achieve this, for this article, we will be using EmguCV, and more specifically, the stitcher namespace and class.

The Example project for this article can be found on Github.

The four images I want to stitch can be found below. Across the four images, I made sure that there was enough overlap, as it will be needed to successfully stitch the images together.

The first step in stitching images together is reading in the desired photos from a folder and storing them in a list. In the code below, I read in all the images from the desired folder, and store them in the output list.

            //Store all paths to each individual file 
            //in the directory in an array
            string[] files = Directory.GetFiles(path);

            //Declare a list of Mat to store all images loaded in
            List<Mat> outputList = new List<Mat>();

            //For each item in the files array, 
            //read in the image and store it in the list
            foreach(var image in files)
            {
                //Read in the image and store it as a mat object
                Mat img = CvInvoke.Imread(image, ImreadModes.AnyColor);

                //Add the mat object to the list
                outputList.Add(img);
            }

Once the images have been successfully read in and stored in a list of Mat objects, we can now begin the image stitching process.

The first step in stitching images together in EmguCV, is declaring a new stitcher from the Stitching class. The reason for this is that in order to stitch images together, the process must be done using the Stitcher class in EmguCV, where we can better control how the process works and how our final image will be generated. This is done using the code below.

            //Declare a new stitcher
            Stitcher stitcher = new Stitcher();

The basis for image stitching is that in order to combine all the relevant images into a single photo, is the reliance of keypoints. If you want to dive deep into how keypoints work, you can find some examples online, but to simplify it, keypoints are used to find similar features across multiple images. Keypoitns are found by converting an image to gray scale, then using that gray scale image’s histogram to find relevant areas and creating keypoints at those locations.

In order to find keypoints on an image, we must first declare a detector. There are a verity of detectors available, both open source and patented, but for today’s project, we will use the Brisk detector. Although the Binary Robust Invariant Scalable Keypoints detector is fairly new, Brisk is an open source detector that I have found that plays well with multiple types of images, while still generating a solid final image without much warping or distortion. To declare the detector, you can use the code below. After the declaration, I also set the features finder of the new stitcher that we created above to be the Brisk detector that we just declared.

            //Declare the type of detector that will be used to detect keypoints
            Brisk detector = new Brisk();

            //Set the stitcher class to use the specified detector declared above
            stitcher.SetFeaturesFinder(detector);

Before we get to the final stage of image stitching, we must first convert all the images in our list to a vector. Although somewhat of an annoyance, the process is fairly easy, and can be done with a couple of lines of code.

            //Declare a vector to store all images from the list
            VectorOfMat matVector = new VectorOfMat();

            //Push all images in the list into a vector
            foreach (Mat img in images)
            {
                matVector.Push(img);
            }

Once we have pushed all of the images in our list to our new vector, we can now finally actually stitch our images together. In order to accomplish this, we must first declare the output variable that will hold our final stitched image. Once the new Mat object has been declared, we then pass it along with our vector to the stitcher. Doing so can be done with the code below.

            Mat output = new Mat();   
         
            //Stitch the images together
            stitcher.Stitch(matVector, output);

In order to view your result, you can output your image to c#’s WPF image source, or you can write the mat object to a folder. If you want to do the first option, you must first convert the output mat object into a bitmap, then convert that bitmap to an image source. Although fairly complicated and complex, I found some code online that will convert a bitmap to an image source.

        //Convert a bitmap to imagesource
        [DllImport("gdi32.dll", EntryPoint = "DeleteObject")]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool DeleteObject([In] IntPtr hObject);

        public ImageSource ImageSourceFromBitmap(Bitmap bmp)
        {
            var handle = bmp.GetHbitmap();
            try
            {
                return Imaging.CreateBitmapSourceFromHBitmap(handle, IntPtr.Zero, Int32Rect.Empty, 
                    BitmapSizeOptions.FromEmptyOptions());
            }
            finally { DeleteObject(handle); }
        }

Once this block of code has been become funcitonal, you then just have to convert your mat object to a bitmap, then run that bitmap through this above code.

                //Convert the Mat object to a bitmap
                Bitmap img = output.ToBitmap();

                //Using the method below, convert the bitmap to an imagesource
                imgOutput.Source = ImageSourceFromBitmap(img);

Another, and more simpler way, would just be writing the image to your current project directory. This can be done using the following code.

            //Write the stitched image
            CvInvoke.Imwrite("Output.png", output);

Once the images have been stitched, we get our result.

The complete code for my stitching class can be found below.

    /// <summary>
    /// Contains all methods to stitch images together
    /// </summary>
    class ImageStitching
    {
        /// <summary>
        /// Pass a path to a folder of images and read in all those images and store them in a list
        /// </summary>
        /// <param name="path">The path to where the folder where the images are kept</param>
        /// <returns>A list of Mat objects</returns>
        public static List<Mat> GetImages(string path)
        {
            //Store all paths to each individual file in the directory in an array
            string[] files = Directory.GetFiles(path);

            //Declare a list of Mat to store all images loaded in
            List<Mat> outputList = new List<Mat>();

            //For each item in the files array, read in the image and store it in the list
            foreach(var image in files)
            {
                //Read in the image and store it as a mat object
                Mat img = CvInvoke.Imread(image, ImreadModes.AnyColor);

                //Add the mat object to the list
                outputList.Add(img);
            }

            //Return the list of read in mat objects
            return outputList;
        }

        /// <summary>
        /// Stitch images together
        /// </summary>
        /// <param name="images">The list of images to stitch</param>
        /// <returns>A final stitched image</returns>
        public static Mat StichImages(List<Mat> images)
        {
            //Declare the Mat object that will store the final output
            Mat output = new Mat();

            //Declare a vector to store all images from the list
            VectorOfMat matVector = new VectorOfMat();

            //Push all images in the list into a vector
            foreach (Mat img in images)
            {
                matVector.Push(img);
            }

            //Declare a new stitcher
            Stitcher stitcher = new Stitcher();

            //Declare the type of detector that will be used to detect keypoints
            Brisk detector = new Brisk();

            //Here are some other detectors that you can try
            //ORBDetector detector = new ORBDetector();
            //KAZE detector = new KAZE();
            //AKAZE detector = new AKAZE();

            //Set the stitcher class to use the specified detector declared above
            stitcher.SetFeaturesFinder(detector);

            //Stitch the images together
            stitcher.Stitch(matVector, output);

            //Return the final stiched image
            return output;
        }
    }

Although initially it sounded like a difficult task, image stitching in EmguCV is fairly easy, and can be done in less then a hundred lines of code. This article as just a starting example, but if you want to learn more about other methods within the image stitching class and its namespace, you can read up on its documentation here.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: