Creating A Credit Card OCR Application Part 2

With the second portion of the credit card ocr application, I wanted to focus on sprucing up the GUI and making the whole application more presentable. Most of my past developer experience has been related to back end and deep learning work, so I wanted to get better at front end and making applications more appealing to the eye. The contents of this repo can be found here.

Since the original application was coded in c#, the logical choice for my front end code is Xaml. Although it is very different to the other, more popular front end languages like JavaScript or CSS, coupling it with c# was my best option.

My early versions of the GUI were very minimal, with only the default textures in the xaml toolbox being used.

My first step in cleaning up my GUI was splitting up some of my functions and navigation into multiple pages. I separated two main functions into their own separate pages, the capture method relating to relaying a live camera feed to the GUI and capture an image when needed, and the output page where the detected numbers would go.

Below are the whole Xaml code for both the camera page and the output page.

<Page x:Class="Credit_Card_OCR.Pages.CameraPage"
      xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
      xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
      xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" 
      xmlns:d="http://schemas.microsoft.com/expression/blend/2008" 
      xmlns:local="clr-namespace:Credit_Card_OCR.Pages"
      mc:Ignorable="d" 
      d:DesignHeight="400" d:DesignWidth="700"
      Title="CameraPage">

    <Grid Background="DarkTurquoise">
        <Grid.RowDefinitions>
            <RowDefinition Height ="1*"/>
            <RowDefinition Height ="50"/>
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="1*"/>
        </Grid.ColumnDefinitions>

        <Image x:Name="webcamOutput" Grid.Column="0" HorizontalAlignment="Center" VerticalAlignment="Center" Grid.Row="0" Stretch="Fill"/>
        <Button x:Name="btnCapture" Style="{StaticResource MetroButtonGreen}" Content="Start Camera Stream" Grid.Column="0" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Grid.Row="1" Click="btnCapture_Click"/>

    </Grid>
</Page>

<Page x:Class="Credit_Card_OCR.Pages.ImageListOutputPage"
      xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
      xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
      xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" 
      xmlns:d="http://schemas.microsoft.com/expression/blend/2008" 
      xmlns:local="clr-namespace:Credit_Card_OCR.Pages"
      mc:Ignorable="d" 
      d:DesignHeight="450" d:DesignWidth="800"
      Title="ImageListOutputPage">

    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height ="1*"/>
            <RowDefinition Height="50"/>
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="1*"/>
        </Grid.ColumnDefinitions>

        <Image x:Name="imgOutput" HorizontalAlignment="Center" VerticalAlignment="Center" Grid.Column="1" Stretch="Fill"/>
        <ListBox x:Name="lstOutput" VerticalContentAlignment="Stretch" Grid.Column="1" Grid.Row="6" HorizontalContentAlignment="Stretch"/>
        
    </Grid>
</Page>

Most of the code above is just simple boilerplate that Microsoft provides when creating a page, the items on the page that I created are marked with the tags < and />. The only component of the Xaml code above that is not implemented in Microsoft’s default Xaml code with the button style “metrogreen”, which is defined in the App.xaml.

<Application x:Class="Credit_Card_OCR.App"
             xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:local="clr-namespace:Credit_Card_OCR"
             StartupUri="MainWindow.xaml">
    <Application.Resources>
        <Style x:Key="MetroButtonAllert" TargetType="{x:Type Button}" >
            <Setter Property="Background" Value="OrangeRed" />
            <Setter Property="Foreground" Value="White" />
            <Setter Property="HorizontalContentAlignment" Value="Center" />
            <Setter Property="VerticalContentAlignment" Value="Center" />
            <Setter Property="Padding" Value="10 5" />
            <Setter Property="FontFamily" Value="Tahoma" />
            <Setter Property="FontSize" Value="14" />
            <Setter Property="BorderThickness" Value="2" />
            <Setter Property="Template">
                <Setter.Value>
                    <ControlTemplate TargetType="{x:Type Button}">
                        <Grid>
                            <Border
                                x:Name="Border"
                                Background="{TemplateBinding Background}"
                                BorderBrush="{TemplateBinding BorderBrush}"
                                BorderThickness="{TemplateBinding BorderThickness}"/>
                            <ContentPresenter
                                HorizontalAlignment="{TemplateBinding HorizontalContentAlignment}"
                                Margin="{TemplateBinding Padding}"
                                VerticalAlignment="{TemplateBinding VerticalContentAlignment}"
                                RecognizesAccessKey="True"/>
                        </Grid>
                        <ControlTemplate.Triggers>
                            <Trigger Property="IsPressed" Value="True">
                                <Setter Property="OpacityMask" Value="#AA888888"/>
                            </Trigger>
                            <Trigger Property="IsMouseOver" Value="True">
                                <Setter Property="Background" Value="#cf2a0e"/>
                            </Trigger>
                            <Trigger Property="IsEnabled" Value="False">
                                <Setter Property="Foreground" Value="#ADADAD" />
                            </Trigger>
                        </ControlTemplate.Triggers>
                    </ControlTemplate>
                </Setter.Value>
            </Setter>
        </Style>
        <Style x:Key="MetroButtonGreen" BasedOn="{StaticResource MetroButtonAllert}" TargetType="{x:Type Button}">
            <Setter Property="Background" Value="OliveDrab"/>
            <Style.Triggers>
                <Trigger Property="IsMouseOver" Value="True">
                    <Setter Property="Background" Value="#368a55"/>
                </Trigger>
            </Style.Triggers>
        </Style>
    </Application.Resources>
</Application>

The point of the code above is to create a style, a distinct set of parameters and events that can be assigned to different Xaml features to make it look better, similar to JavaScript and CSS. Here I have to styles, one for a green button and one for a red button, both of witch are used more in the next step in our Xaml code.

To control the navigation between these pages, we use multiple buttons on a stack panel, which allows a static control panel to exist while the rest of the GUI changes whenever needed.

<Window x:Class="Credit_Card_OCR.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:Credit_Card_OCR"
        mc:Ignorable="d"
        Title="Credit Card Detection" Height="450" Width="800" Background="DarkTurquoise">

    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height ="1*"/>
            <RowDefinition Height ="50"/>
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="100"/>
            <ColumnDefinition Width="1*"/>
        </Grid.ColumnDefinitions>

        <StackPanel Grid.Row="0" Grid.RowSpan="2" Grid.Column="0" Grid.ColumnSpan="2">
            <Grid>
                <Grid.RowDefinitions>
                    <RowDefinition Height ="50"/>
                    <RowDefinition Height ="50"/>
                    <RowDefinition Height ="50"/>
                    <RowDefinition Height ="50"/>
                    <RowDefinition Height ="50"/>
                    <RowDefinition Height ="110"/>
                    <RowDefinition Height ="50"/>
                </Grid.RowDefinitions>
                <Grid.ColumnDefinitions>
                    <ColumnDefinition Width="100"/>
                    <ColumnDefinition Width="1*"/>
                </Grid.ColumnDefinitions>

                <Button x:Name="btnOCR" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" Grid.Row="1" Click="btnOCR_Click">
                    <StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
                        <Image Source="Resources\ocrIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
                        <TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Capture Image"/></TextBlock>
                    </StackPanel>
                </Button>

                <Button x:Name="btnStart" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="2" VerticalAlignment="Stretch" Click="btnStart_Click">
                    <StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
                        <Image Source="Resources\startIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
                        <TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Start Camera"/></TextBlock>
                    </StackPanel>
                </Button>
                
                <Button x:Name="btnOpen" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="3" VerticalAlignment="Stretch" Click="btnOpen_Click">
                    <StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
                        <Image Source="Resources\folderIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
                        <TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Open Image"/></TextBlock>
                    </StackPanel>
                </Button>
                
                <Button x:Name="btnCapture" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="4" VerticalAlignment="Stretch" Click="btnCapture_Click" IsEnabled="False">
                    <StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
                        <Image Source="Resources\cameraIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
                        <TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Capture"/></TextBlock>
                    </StackPanel>
                </Button>
                
                <Button x:Name="btnClear" Grid.Column="0" Style="{StaticResource MetroButtonAllert}" HorizontalAlignment="Stretch" Grid.Row="6" VerticalAlignment="Stretch" Click="btnClear_Click">
                    <StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
                        <Image Source="Resources\clearIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
                        <TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Clear"/></TextBlock>
                    </StackPanel>
                </Button>
            </Grid>
        </StackPanel>

        <Frame x:Name="frame" Grid.Column="1" Grid.Row="0" Grid.RowSpan="2" NavigationUIVisibility="Hidden"/>

    </Grid>
</Window>

For my stack panel, I have five buttons, one for running ocr with the color green, one for starting the camera with the color green, one for opening an image with the color green, one for capturing an image with the color green, and one for clearing all necessary objects with the color red.

The images on the buttons are from Icon8, which I choose them as they were simple and accomplished what the button was trying to do.

The OCR button is only used when an image is present, which launches the ocr detection method, of which we created back in the part one of this post. The start camera button initializes the camera page and creates a camera stream, which is created in the cs of the page.

using Emgu.CV;
using Emgu.CV.Structure;
using System;
using System.Drawing;
using System.Windows;
using System.Windows.Controls;

namespace Credit_Card_OCR.Pages
{
    /// <summary>
    /// Interaction logic for CameraPage.xaml
    /// </summary>
    public partial class CameraPage : Page
    {
        //The video capture stream that holds the video from the camera
        public VideoCapture stream = new VideoCapture(0);

        public CameraPage()
        {
            InitializeComponent();
        }

        public Bitmap GenerateTakenImage()
        {
            //Get the taken picture from the video stream
            Image<Bgr, byte> inputImg = CameraConfig.TakePicture(stream);

            //Convert the image to a bitmap
            Bitmap bit = inputImg.ToBitmap();

            return bit;
        }

        private void Capture_ImageGrabbed1(object sender, EventArgs e)
        {
            //Get the frame
            Bitmap frame = CameraConfig.GetFrame(stream);

            //Check if the frame taken is null
            //if it is, that means that no camera is connected,
            //so output that to the user and end the subscription
            if (frame == null)
            {
                MessageBox.Show("There is no camera connected");
                stream.ImageGrabbed -= Capture_ImageGrabbed1;
            }
            else
            {
                try
                {
                    //Pass each frame from the video capture to the output image
                    this.Dispatcher.Invoke(() =>
                    {
                        webcamOutput.Source = ImageUtils.ImageSourceFromBitmap(frame);
                    });
                }
                catch (System.Threading.Tasks.TaskCanceledException)
                {
                    //End the thread if its exited early
                    this.Dispatcher.InvokeShutdown();
                }
            }
        }

        private void btnCapture_Click(object sender, RoutedEventArgs e)
        {
            //Start grabbing frames from the webcam input
            stream.ImageGrabbed += Capture_ImageGrabbed1;
            stream.Start();
        }
    }
}

When the capture button is clicked on the capture page and start a subscription to constantly stream frames to the camera page. When the capture button on the MAINWINDOW is clicked, it grabs the latest frame being streamed and outputs it as a single still image ready for the OCR method. I put the streaming of the camera inside its own method, as if it was coupled inside the main thread, lag and loss of frames causes issues. The entire code for the main window can be found below.

using Credit_Card_OCR.Pages;
using Emgu.CV;
using Emgu.CV.CvEnum;
using System.Collections.Generic;
using System.Drawing;
using System.Windows;
using System.Windows.Forms;

namespace Credit_Card_OCR
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        ImageListOutputPage outputPage = new ImageListOutputPage();

        public MainWindow()
        {
            InitializeComponent();

            frame.Content = outputPage;
        }

        //Mat object to hold image that is taken
        public static Mat img = new Mat();

        //Bool variables to help with the flow of the algorithm
        bool isStream = false;
        bool isImageLoaded = false;

        private void btnStart_Click(object sender, RoutedEventArgs e)
        {
            //Set bool variable to use camera image
            isStream = true;

            //Create a new object for the page
            CameraPage camera = new CameraPage();

            btnCapture.IsEnabled = true;
            btnOCR.IsEnabled = false;

            //Navigate to the new page
            frame.NavigationService.Navigate(camera);
        }

        private void btnOCR_Click(object sender, RoutedEventArgs e)
        {
            if (isImageLoaded == true)
            {
                //If a video stream is currently happening, do the following, else do the last
                if (isStream == true)
                {
                    //Align the video capture image
                    img = AutoAlignment.Align(img);

                    Bitmap bit = img.ToBitmap();

                    //Pass the bitmap image to be displayed
                    outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);
                }
                else
                {
                    //Read in the desired image
                    img = OCR.ReadInImage();

                    //Convert the image to a bitmap, then to an image source
                    Bitmap bit = img.ToBitmap();
                    outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);
                }

                //Detect the text from the image
                string detectedText = OCR.RecognizeText(img);

                //Create a new list for the output
                List<string> output = new List<string>();

                //Add the detected string to the list
                output.Add(detectedText);

                //Display the list
                outputPage.lstOutput.ItemsSource = output;
            }
            else
            {
                System.Windows.MessageBox.Show("Please take or import an image");
            }
        }

        private void btnOpen_Click(object sender, RoutedEventArgs e)
        {
            //Create a string to hold the file location
            string fileLocation = string.Empty;

            //Create an openfiledialog object to select a photo
            OpenFileDialog dialog = new OpenFileDialog
            {
                InitialDirectory = @"C:\",
                Title = "Browse Images",

                CheckFileExists = true,
                CheckPathExists = true,

                FilterIndex = 2,
                RestoreDirectory = true,

                ReadOnlyChecked = true,
                ShowReadOnly = true
            };

            //If a photo is selected, do the following
            if (dialog.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                //Get the file location of the photo selected
                fileLocation = dialog.FileName;

                //If the length is greater than one, then use it to read in the image
                if (fileLocation.Length > 1)
                {
                    //Read in the image
                    img = CvInvoke.Imread(fileLocation, ImreadModes.AnyColor);
                }
            }

            //If the file location is greater than one, pass the loaded in image to be displayed
            if (fileLocation.Length > 1)
            {
                //Convert the image to a bitmap, then to an image source
                Bitmap bit = img.ToBitmap();
                outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);

                //Declare the bool variable that an image has been loaded
                isImageLoaded = true;
            }
        }

        private void btnCapture_Click(object sender, RoutedEventArgs e)
        {
            CameraPage cameraPage = new CameraPage();

            Bitmap taken = cameraPage.GenerateTakenImage();

            if (taken != null)
            {
                outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(taken);
            }

            //Declare that image is used
            isImageLoaded = true;

            //Enable the detect button
            btnOCR.IsEnabled = true;

            //Clear the frame
            frame.Content = outputPage;
        }

        private void btnClear_Click(object sender, RoutedEventArgs e)
        {
            //Clear the image
            outputPage.imgOutput.Source = null;

            //Clear the frame
            frame.Content = outputPage;

            //Empty the list box
            if (outputPage.lstOutput.Items.Count > 0)
            {
                outputPage.lstOutput.ItemsSource = null;
            }

            //Set both bool variables to false
            isImageLoaded = false;
            isStream = false;

            //Disable capture button
            btnCapture.IsEnabled = false;
            btnOCR.IsEnabled = true;
        }
    }
}

Although this is pretty boiler plate code, and I must admit the front end code does look a little sloppy, but Its a step in a better direction then what the GUI was earlier. I am still working to add more features to the GUI, and I hope to change the color scheme and other parts of the GUI.