Creating A Credit Card OCR Application Part 2
Credit Card OCR II
With the second portion of the credit card ocr application, I wanted to focus on sprucing up the GUI and making the whole application more presentable. Most of my past developer experience has been related to back end and deep learning work, so I wanted to get better at front end and making applications more appealing to the eye. The contents of this repo can be found here.
Since the original application was coded in c#, the logical choice for my front end code is Xaml. Although it is very different to the other, more popular front end languages like JavaScript or CSS, coupling it with c# was my best option.
My early versions of the GUI were very minimal, with only the default textures in the xaml toolbox being used.
My first step in cleaning up my GUI was splitting up some of my functions and navigation into multiple pages. I separated two main functions into their own separate pages, the capture method relating to relaying a live camera feed to the GUI and capture an image when needed, and the output page where the detected numbers would go.
Below are the whole Xaml code for both the camera page and the output page.
<Page x:Class="Credit_Card_OCR.Pages.CameraPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:local="clr-namespace:Credit_Card_OCR.Pages"
mc:Ignorable="d"
d:DesignHeight="400" d:DesignWidth="700"
Title="CameraPage">
<Grid Background="DarkTurquoise">
<Grid.RowDefinitions>
<RowDefinition Height ="1*"/>
<RowDefinition Height ="50"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="1*"/>
</Grid.ColumnDefinitions>
<Image x:Name="webcamOutput" Grid.Column="0" HorizontalAlignment="Center" VerticalAlignment="Center" Grid.Row="0" Stretch="Fill"/>
<Button x:Name="btnCapture" Style="{StaticResource MetroButtonGreen}" Content="Start Camera Stream" Grid.Column="0" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Grid.Row="1" Click="btnCapture_Click"/>
</Grid>
</Page>
<Page x:Class="Credit_Card_OCR.Pages.ImageListOutputPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:local="clr-namespace:Credit_Card_OCR.Pages"
mc:Ignorable="d"
d:DesignHeight="450" d:DesignWidth="800"
Title="ImageListOutputPage">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height ="1*"/>
<RowDefinition Height="50"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="1*"/>
</Grid.ColumnDefinitions>
<Image x:Name="imgOutput" HorizontalAlignment="Center" VerticalAlignment="Center" Grid.Column="1" Stretch="Fill"/>
<ListBox x:Name="lstOutput" VerticalContentAlignment="Stretch" Grid.Column="1" Grid.Row="6" HorizontalContentAlignment="Stretch"/>
</Grid>
</Page>
Most of the code above is just simple boilerplate that Microsoft provides when creating a page, the items on the page that I created are marked with the tags < and />. The only component of the Xaml code above that is not implemented in Microsoft’s default Xaml code with the button style “metrogreen”, which is defined in the App.xaml.
<Application x:Class="Credit_Card_OCR.App"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="clr-namespace:Credit_Card_OCR"
StartupUri="MainWindow.xaml">
<Application.Resources>
<Style x:Key="MetroButtonAllert" TargetType="{x:Type Button}" >
<Setter Property="Background" Value="OrangeRed" />
<Setter Property="Foreground" Value="White" />
<Setter Property="HorizontalContentAlignment" Value="Center" />
<Setter Property="VerticalContentAlignment" Value="Center" />
<Setter Property="Padding" Value="10 5" />
<Setter Property="FontFamily" Value="Tahoma" />
<Setter Property="FontSize" Value="14" />
<Setter Property="BorderThickness" Value="2" />
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type Button}">
<Grid>
<Border
x:Name="Border"
Background="{TemplateBinding Background}"
BorderBrush="{TemplateBinding BorderBrush}"
BorderThickness="{TemplateBinding BorderThickness}"/>
<ContentPresenter
HorizontalAlignment="{TemplateBinding HorizontalContentAlignment}"
Margin="{TemplateBinding Padding}"
VerticalAlignment="{TemplateBinding VerticalContentAlignment}"
RecognizesAccessKey="True"/>
</Grid>
<ControlTemplate.Triggers>
<Trigger Property="IsPressed" Value="True">
<Setter Property="OpacityMask" Value="#AA888888"/>
</Trigger>
<Trigger Property="IsMouseOver" Value="True">
<Setter Property="Background" Value="#cf2a0e"/>
</Trigger>
<Trigger Property="IsEnabled" Value="False">
<Setter Property="Foreground" Value="#ADADAD" />
</Trigger>
</ControlTemplate.Triggers>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
<Style x:Key="MetroButtonGreen" BasedOn="{StaticResource MetroButtonAllert}" TargetType="{x:Type Button}">
<Setter Property="Background" Value="OliveDrab"/>
<Style.Triggers>
<Trigger Property="IsMouseOver" Value="True">
<Setter Property="Background" Value="#368a55"/>
</Trigger>
</Style.Triggers>
</Style>
</Application.Resources>
</Application>
The point of the code above is to create a style, a distinct set of parameters and events that can be assigned to different Xaml features to make it look better, similar to JavaScript and CSS. Here I have to styles, one for a green button and one for a red button, both of witch are used more in the next step in our Xaml code.
To control the navigation between these pages, we use multiple buttons on a stack panel, which allows a static control panel to exist while the rest of the GUI changes whenever needed.
<Window x:Class="Credit_Card_OCR.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:local="clr-namespace:Credit_Card_OCR"
mc:Ignorable="d"
Title="Credit Card Detection" Height="450" Width="800" Background="DarkTurquoise">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height ="1*"/>
<RowDefinition Height ="50"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="100"/>
<ColumnDefinition Width="1*"/>
</Grid.ColumnDefinitions>
<StackPanel Grid.Row="0" Grid.RowSpan="2" Grid.Column="0" Grid.ColumnSpan="2">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height ="50"/>
<RowDefinition Height ="50"/>
<RowDefinition Height ="50"/>
<RowDefinition Height ="50"/>
<RowDefinition Height ="50"/>
<RowDefinition Height ="110"/>
<RowDefinition Height ="50"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="100"/>
<ColumnDefinition Width="1*"/>
</Grid.ColumnDefinitions>
<Button x:Name="btnOCR" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" Grid.Row="1" Click="btnOCR_Click">
<StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
<Image Source="Resources\ocrIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
<TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Capture Image"/></TextBlock>
</StackPanel>
</Button>
<Button x:Name="btnStart" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="2" VerticalAlignment="Stretch" Click="btnStart_Click">
<StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
<Image Source="Resources\startIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
<TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Start Camera"/></TextBlock>
</StackPanel>
</Button>
<Button x:Name="btnOpen" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="3" VerticalAlignment="Stretch" Click="btnOpen_Click">
<StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
<Image Source="Resources\folderIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
<TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Open Image"/></TextBlock>
</StackPanel>
</Button>
<Button x:Name="btnCapture" Grid.Column="0" Style="{StaticResource MetroButtonGreen}" HorizontalAlignment="Stretch" Grid.Row="4" VerticalAlignment="Stretch" Click="btnCapture_Click" IsEnabled="False">
<StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
<Image Source="Resources\cameraIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
<TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Capture"/></TextBlock>
</StackPanel>
</Button>
<Button x:Name="btnClear" Grid.Column="0" Style="{StaticResource MetroButtonAllert}" HorizontalAlignment="Stretch" Grid.Row="6" VerticalAlignment="Stretch" Click="btnClear_Click">
<StackPanel HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Visibility="Visible" Grid.Column="0">
<Image Source="Resources\clearIcon.png" HorizontalAlignment="Center" Stretch="Uniform" Height="30" Width="50"/>
<TextBlock HorizontalAlignment="Center" FontSize="10"><Run Text="Clear"/></TextBlock>
</StackPanel>
</Button>
</Grid>
</StackPanel>
<Frame x:Name="frame" Grid.Column="1" Grid.Row="0" Grid.RowSpan="2" NavigationUIVisibility="Hidden"/>
</Grid>
</Window>
For my stack panel, I have five buttons, one for running ocr with the color green, one for starting the camera with the color green, one for opening an image with the color green, one for capturing an image with the color green, and one for clearing all necessary objects with the color red.
The images on the buttons are from Icon8, which I choose them as they were simple and accomplished what the button was trying to do.
The OCR button is only used when an image is present, which launches the ocr detection method, of which we created back in the part one of this post. The start camera button initializes the camera page and creates a camera stream, which is created in the cs of the page.
using Emgu.CV;
using Emgu.CV.Structure;
using System;
using System.Drawing;
using System.Windows;
using System.Windows.Controls;
namespace Credit_Card_OCR.Pages
{
/// <summary>
/// Interaction logic for CameraPage.xaml
/// </summary>
public partial class CameraPage : Page
{
//The video capture stream that holds the video from the camera
public VideoCapture stream = new VideoCapture(0);
public CameraPage()
{
InitializeComponent();
}
public Bitmap GenerateTakenImage()
{
//Get the taken picture from the video stream
Image<Bgr, byte> inputImg = CameraConfig.TakePicture(stream);
//Convert the image to a bitmap
Bitmap bit = inputImg.ToBitmap();
return bit;
}
private void Capture_ImageGrabbed1(object sender, EventArgs e)
{
//Get the frame
Bitmap frame = CameraConfig.GetFrame(stream);
//Check if the frame taken is null
//if it is, that means that no camera is connected,
//so output that to the user and end the subscription
if (frame == null)
{
MessageBox.Show("There is no camera connected");
stream.ImageGrabbed -= Capture_ImageGrabbed1;
}
else
{
try
{
//Pass each frame from the video capture to the output image
this.Dispatcher.Invoke(() =>
{
webcamOutput.Source = ImageUtils.ImageSourceFromBitmap(frame);
});
}
catch (System.Threading.Tasks.TaskCanceledException)
{
//End the thread if its exited early
this.Dispatcher.InvokeShutdown();
}
}
}
private void btnCapture_Click(object sender, RoutedEventArgs e)
{
//Start grabbing frames from the webcam input
stream.ImageGrabbed += Capture_ImageGrabbed1;
stream.Start();
}
}
}
When the capture button is clicked on the capture page and start a subscription to constantly stream frames to the camera page. When the capture button on the MAINWINDOW is clicked, it grabs the latest frame being streamed and outputs it as a single still image ready for the OCR method. I put the streaming of the camera inside its own method, as if it was coupled inside the main thread, lag and loss of frames causes issues. The entire code for the main window can be found below.
using Credit_Card_OCR.Pages;
using Emgu.CV;
using Emgu.CV.CvEnum;
using System.Collections.Generic;
using System.Drawing;
using System.Windows;
using System.Windows.Forms;
namespace Credit_Card_OCR
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
ImageListOutputPage outputPage = new ImageListOutputPage();
public MainWindow()
{
InitializeComponent();
frame.Content = outputPage;
}
//Mat object to hold image that is taken
public static Mat img = new Mat();
//Bool variables to help with the flow of the algorithm
bool isStream = false;
bool isImageLoaded = false;
private void btnStart_Click(object sender, RoutedEventArgs e)
{
//Set bool variable to use camera image
isStream = true;
//Create a new object for the page
CameraPage camera = new CameraPage();
btnCapture.IsEnabled = true;
btnOCR.IsEnabled = false;
//Navigate to the new page
frame.NavigationService.Navigate(camera);
}
private void btnOCR_Click(object sender, RoutedEventArgs e)
{
if (isImageLoaded == true)
{
//If a video stream is currently happening, do the following, else do the last
if (isStream == true)
{
//Align the video capture image
img = AutoAlignment.Align(img);
Bitmap bit = img.ToBitmap();
//Pass the bitmap image to be displayed
outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);
}
else
{
//Read in the desired image
img = OCR.ReadInImage();
//Convert the image to a bitmap, then to an image source
Bitmap bit = img.ToBitmap();
outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);
}
//Detect the text from the image
string detectedText = OCR.RecognizeText(img);
//Create a new list for the output
List<string> output = new List<string>();
//Add the detected string to the list
output.Add(detectedText);
//Display the list
outputPage.lstOutput.ItemsSource = output;
}
else
{
System.Windows.MessageBox.Show("Please take or import an image");
}
}
private void btnOpen_Click(object sender, RoutedEventArgs e)
{
//Create a string to hold the file location
string fileLocation = string.Empty;
//Create an openfiledialog object to select a photo
OpenFileDialog dialog = new OpenFileDialog
{
InitialDirectory = @"C:\",
Title = "Browse Images",
CheckFileExists = true,
CheckPathExists = true,
FilterIndex = 2,
RestoreDirectory = true,
ReadOnlyChecked = true,
ShowReadOnly = true
};
//If a photo is selected, do the following
if (dialog.ShowDialog() == System.Windows.Forms.DialogResult.OK)
{
//Get the file location of the photo selected
fileLocation = dialog.FileName;
//If the length is greater than one, then use it to read in the image
if (fileLocation.Length > 1)
{
//Read in the image
img = CvInvoke.Imread(fileLocation, ImreadModes.AnyColor);
}
}
//If the file location is greater than one, pass the loaded in image to be displayed
if (fileLocation.Length > 1)
{
//Convert the image to a bitmap, then to an image source
Bitmap bit = img.ToBitmap();
outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(bit);
//Declare the bool variable that an image has been loaded
isImageLoaded = true;
}
}
private void btnCapture_Click(object sender, RoutedEventArgs e)
{
CameraPage cameraPage = new CameraPage();
Bitmap taken = cameraPage.GenerateTakenImage();
if (taken != null)
{
outputPage.imgOutput.Source = ImageUtils.ImageSourceFromBitmap(taken);
}
//Declare that image is used
isImageLoaded = true;
//Enable the detect button
btnOCR.IsEnabled = true;
//Clear the frame
frame.Content = outputPage;
}
private void btnClear_Click(object sender, RoutedEventArgs e)
{
//Clear the image
outputPage.imgOutput.Source = null;
//Clear the frame
frame.Content = outputPage;
//Empty the list box
if (outputPage.lstOutput.Items.Count > 0)
{
outputPage.lstOutput.ItemsSource = null;
}
//Set both bool variables to false
isImageLoaded = false;
isStream = false;
//Disable capture button
btnCapture.IsEnabled = false;
btnOCR.IsEnabled = true;
}
}
}
Although this is pretty boiler plate code, and I must admit the front end code does look a little sloppy, but Its a step in a better direction then what the GUI was earlier. I am still working to add more features to the GUI, and I hope to change the color scheme and other parts of the GUI.
