Advanced
Non-constraining Online Signature Reconstruction System for Persons with Handwriting Problems
Non-constraining Online Signature Reconstruction System for Persons with Handwriting Problems
ETRI Journal. 2015. Feb, 37(1): 138-146
Copyright © 2015, Electronics and Telecommunications Research Institute(ETRI)
  • Received : March 01, 2014
  • Accepted : September 29, 2014
  • Published : February 01, 2015
Download
PDF
e-PUB
PubReader
PPT
Export by style
Share
Article
Author
Metrics
Cited by
TagCloud
About the Authors
Belkacem Abbadi
Messaoud Mostefai
Adel Oulefki

Abstract
This paper presents a new non-constraining online optical handwritten signature reconstruction system that, in the main, makes use of a transparent glass pad placed in front of a color camera. The reconstruction approach allows efficient exploitation of hand activity during a signing process; thus, the system as a whole can be seen as a viable alternative to other similar acquisition tools. This proposed system allows people with physical or emotional problems to carry out their own signatures without having to use a pen or sophisticated acquisition system. Moreover, the developed reconstruction signature algorithms have low computational complexity and are therefore well suited for a hardware implementation on a dedicated smart system.
Keywords
I. Introduction
The use of the signature as an authentication tool is a common practice adopted by most public and private institutions [1] [4] . To deal with the requirements of automated processes, online signature approaches seem to be pretty useful [3] . The effectiveness of such approaches depends on both the acquisition system and the adopted authentication method [4] [6] . However, this effectiveness will be considerably decreased in cases where the signer suffers from a physical or emotional handicap; in particular, an elderly person who has a hand tremor or who is no longer able to reproduce their specimen signature. Such physical or emotional handicaps hinder a person’s ability to authenticate their own signature. This can cause complications with certain institutions, such as banks, whereby an institution can be very uncompromising about the authenticity of signatures [4] , [7] . Different solutions to avoid this impediment have been adopted. For instance, we cite the following possible alternative solutions to the issue:
  • ▪ Getting help from a third party (tutor). However, this would evoke the issue of confidentiality.
  • ▪ Using special bank ID cards or signature stamps. However, these are liable to be lost or stolen, especially from disabled or elderly persons, which leads to illegal use.
  • ▪ The use of other modalities, such as a fingerprint, which would come at the cost of the advantages of a signature.
Besides the aforementioned alternative solutions, it should be noted that there has been no published research works focused on tackling the issue directly.
In this paper, we propose an original approach for the reconstruction of signatures, based on the movement of a person’s hand operating on a glass support. This approach allows persons with handwriting problems to more successfully reproduce a signature that is close to their specimen signature.
Aiming to benefit from the advantages of vision-based methods, Munich and Perona [1] proposed a camera-based signature tracking, which merely involved a paper support and a simple pen. Reasonable classification results were achieved through this technique since it allowed an efficient parameterization of acquired signatures. Likewise in [2] , pen-grasping information was included to enhance the reliability of the verification process, especially in cases where there was a suspicion of forgery. In a recent work [8] , we presented a new online signature acquisition system that is able to rebuild transparent signatures from a recorded pen motion. An initial laboratory prototype has been developed for this purpose ( Fig. 1 ). It was initially composed of a high-resolution camera placed in front of a transparent signing glass. Signers perform signatures on the glass by moving their index finger. The acquired movements are then used to generate the corresponding signature features x ( t ), y ( t ), and ( x , y ). This method captures the index finger’s trajectory, including the intervals where the index finger is off-glass. This additional information represents the specific and non-visible dynamic behavior of the signatory [4] , [9] .
PPT Slide
Lager Image
Initial laboratory prototype.
II. System Performance Improvement
Reconstructed signatures reproduce accurately the hand movement but remain slightly different from the offline signatures (obtained on a paper support). This is due to the fact that the signers are not familiarized with this signing method and that the glass disposition does not allow for precise handwriting [10] .
Important modifications were made to the prototype (see Fig. 2 ) to make the online signing process similar to the offline process. These are as follows:
  • ▪ The signing glass is placed horizontally with a camera underneath it directed toward the signing glass.
  • ▪ The pen to be used has a tip with a well-selected shape and color.
  • ▪ To reduce the effects of lighting variations, the signing pane is protected by an opaque lid.
PPT Slide
Lager Image
Improved online-signature acquisition system: (a) global view, (b) laboratory prototype, and (c) signing operation view.
Figure 3 presents some reconstructed signatures obtained with our improved acquisition system. The reconstructed signatures are very similar to their offline counterparts. The success of the improved acquisition system means it is open to a wide range of applications that are based on both online and offline signatures.
PPT Slide
Lager Image
Example of obtained signatures.
III. Hand Dynamics Exploitation
By exploring the recorded videos, we noticed that it was possible to extend the dynamics of the pen toward that of the hand. Indeed, if the pen moves, then it is because the hand moves. Thus, these two phenomena are strongly dependent and from which it is then possible to associate them for a better signature characterization.
In what follows, we present the basic principle of our signature reconstruction approach and show some preliminary experiments to confirm the efficiency of the proposed method.
- 1. Basic Principle
When a person signs, he carries out specific movements with his hand. The images presented in Fig. 4 show the various positions of the hand during a signing operation.
PPT Slide
Lager Image
Example of successive frames.
By a simple differentiation between successive binary images, it is possible to describe the signature dynamics with a moving-hand surface function, denoted by MHS( t ), which varies over time. However, this function does not give an exact idea of the signature course, and consequently, it does not allow us to reproduce the pen signature. By separating MHS( t ) into the two directions x and y , one can extract the required information for a precise signature reconstruction. This separation is illustrated in Fig. 5 .
PPT Slide
Lager Image
Moving-hand dynamics extraction.
First of all, each frame I , of size W × H , needs to be binarized; that is, the pixels around the hand region are to be represented by 1s and those that make up the background by 0s. Then, we compute the corresponding vertical and horizontal projection profiles, VPP x and HPP y , respectively, which describe the distribution of the hand surface along the x - and y -axis, respectively, as follows:
VPP x = y=1 W I(x, y) ,
HPP y = x=1 H I(x, y) .
These two vectors will be used to compute the moving hand surface between any two successive frames ( In and I n+1 ) in both the x and y directions. Thus, we have
MHS x = VPP x ( I n ) VPP x ( I n+1 )   for x= 1,  ,W, 
MHS y = HPP y ( I n ) HPP y ( I n+1 )   for y= 1,  ,H. 
The hand movement direction is deduced according to (Ext x , Ext y ), where Ext x and Ext y are the first non-zero local extrema of the vectors MHS x and MHS y , respectively (see Table 1 ).
Moving-hand directions extraction.
Extx Exty Evolution along X Evolution along Y Direction
= 0 < 0 0 Dir0←
< 0 < 0 Dir1↖
< 0 = 0 0 Dir2↑
< 0 > 0 + Dir3↗
= 0 > 0 0 + Dir4→
> 0 > 0 + + Dir5↘
> 0 = 0 + 0 Dir6↓
> 0 < 0 + Dir7↙
For the example presented in Fig. 5 , Ext x = −4 and Ext y = +3. The video object (hand) was displaced from the down to the top (−) and from the left to the right (+). According to Table 1 , the hand is moving in the direction labelled Dir3; that is, in a north-easterly direction.
A real example of the coding process is shown in Fig. 6 , in which we have just taken two different samples of a hand movement between two successive frames along the video sequence. Then, we have explained how to get the movement directions using the differences in the hand surfaces, by referring to Table 1 .
PPT Slide
Lager Image
Directions extraction scheme.
After the coding operation, signature reconstruction is accomplished through a simple manipulation of the obtained direction vector. Table 2 presents some examples of reconstructions of real online signatures based on MHS x and MHS y curves.
Examples of reconstructions of real online signatures.
Offline signature MHSx(100, 101) curve MHSy(100, 101) curve Reconstructed signature
It is important to note that this reconstruction method is completely independent of the use of a pen and allows signatures to be performed by the simple displacement of a closed hand (see Fig. 7 ). In what follows, we show (through real testing) that the success of using this approach to minimize problems in signature reconstruction is dependent upon the physical or emotional status of the signatory.
PPT Slide
Lager Image
New signature reconstruction method: (a) moving closed hand and (b) selected frame.
- 2. Preliminary Experimental Results
A detailed study on the writing of a signature and related problems showed that the emotional or physical state of the person doing the writing plays a vital role in both the quality of writing and the signature itself [10] [11] . Thus, among all psychophysical states, the three most relevant cases that meet our concern are as follows:
  • ▪ A normal person in an abnormal emotional state caused by a panic, stress, disease, and so on.
  • ▪ A person suffering from a hand tremor problem.
  • ▪ A person with a malformed hand, which would mean they would not be able to use a pen.
For all of the above cases, real signature acquisition tests were performed both offline (scanned paper signature) and online (with our system). Online signatures are reconstructed using pen-position tracking and hand tracking. The principle cases are presented in Table 3 . In the case of a normal person (first case), the obtained signatures were nearly identical across the three approaches. For the elderly person with a tremor problem (second case), only the hand tracking approach gave good signatures.
Preliminary results.
Signatory Offline signature acquisition Online signature acquisition
Pen-tip position tracking Closed-hand tracking
Normal person
Old person with tremor problem
Person with malformed hand Not possible
- 3. Test and Evaluation of Acquisition Quality
Like all biometric authentication systems, the verification process is the step that follows that of data acquisition [3] . Although our primary concern is to improve the acquisition quality, verification tests should be presented to evaluate our proposed approach.
We have already mentioned that the effectiveness of such an authentication system depends on both the acquisition system and the adopted verification algorithms. Thus, to demonstrate our contribution toward helping persons with physical or emotional difficulties to reproduce a near-perfect replica of their own signature, we have performed a test on 1,000 signatures acquired from 10 persons who were suffering from a tremor disease. Each signature needed to be carried out with a pen at first and then with the hand. The following describes the signature types: 10 reference signatures, 20 similar to the reference ones to test for false rejections, and 20 that are totally different from the reference signatures to test for false acceptances. Each person’s test signature would be compared against all ten of their reference signatures. Then, an average of the similarity rates would be retained as the score for the verification process for this signature. So, it would be accepted if this score is above a certain threshold, and vice versa. By varying the threshold from 0% to 100%, one obtains the receiver operating characteristic (ROC) of the authentication system, which consists of the following two curves progressing over the threshold scale: false rejection rate (FRR) and false acceptance rate (FAR). The intersection of the two curves yields the equal error rate (EER), which reflects the accuracy of the whole system.
A wide variety of metrics is commonly used to measure the similarity between signatures [5] , [12] . The dynamic time warping (DTW) algorithm has been shown in recent work to be the more useful of the measures and is quite suitable for the context of our work [1] [3] . Using the DTW algorithm, we can obtain the ROC curves (see Fig. 8 ) for all persons. It is clearly noticeable that the obtained EER in the case of signing with a hand is lower than that obtained when signing with a pen (2.9% versus 12.5%). As we used DTW in both cases, these rates indicate an important difference in the quality of the manipulated data.
PPT Slide
Lager Image
ROC curves of DTW-based verification: (a) signing with a pen and (b) signing with a hand.
- 4. Discussion
These relevant tests show that the proposed approach helps people with physical or emotional difficulties to reproduce a near-perfect replica of their own signature.
In terms of verification accuracy, our technique brought a significant improvement on the quality of acquired signatures. This improvement in quality led to a low EER for signatures done with a hand compared to those done with a pen, where the EER is high due to the fact that a disordered signer can’t maintain a balanced signature. Actually, a closed-hand tracking is less precise than a pen-tip tracking, but it remains advantageous in the presence of tremors since it represents a low-pass version of the second one. In contrast to the tracking of a point, the tracking of a large surface area allows for the retrieval of smooth movements. High-frequency movements will be removed through the differentiation operation discussed previously. Moreover, a person’s hand posture allows them to sign in such a way so that their muscles are less tense when doing so; this is in contrast to pen-grasping, where your muscles are relatively tenser. Consequently, all of these factors have contributed to helping overcome the effect that a hand tremor has upon making a person’s signature verifiable; thus, a person suffering from a hand tremor is able to carry out signatures that are nearly identical to each other.
Finally, the ability to sign with your hand is the major benefit of our system, in particular to those who suffer from a malformed hand(s).
IV. Hardware Implementation
To deal with the constraints of smart control access systems, a real-time signature reconstruction is required. The best way to reach this objective is to perform the reconstruction with a hardware architecture. For a first approach, reconfigurable technology is chosen. This allows us to perform low-cost, quick prototyping of dedicated real-time images and signal processing algorithms [13] [14] . In what follows, we present the implementation costs of the pen-position and moving hand–surface computation modules within a Virtex-II Xilinx field programmable gate array (FPGA) [15] .
- 1. Real-Time Pen-Position Tracking Module
The real-time signature reconstruction requires the development of a hardware module that extracts the position of the pen (see Fig. 9 ). This module is able to compute the pen position for each frame, or at least for a group of three successive frames. Obtained successive ( xi , yi ) values will be used to compute the successive differences Δ xi , Δ yi , and tan φ = Δ xi yi .
PPT Slide
Lager Image
Online signature reconstruction.
As is shown in Fig. 10 , each direction (Dir0 to Dir7) comprises an angle of π/4. Thus, sector 0, which corresponds to direction 0, is comprised between −π/8 and +π/8, sector 1, which corresponds to direction 1, is comprised between π/8 and 3π/8, and so on until sector 7.
PPT Slide
Lager Image
Sector location.
The angle formed by the motion vector representing a given sector and the horizontal direction, will thus always be equal to a multiple of π/8. As tan(π/8) =
2
−1 ≅ 0.4, we approximate 0.4 by 0.5 to compute the direction with simple shifts of the differences values Δ xi and Δ yi . The corresponding sector will be defined as follows:
  • IF |Δx| < 2 × |Δy| → Sector 0
  • ELSE IF |Δy| < 2 × |Δx| → Sector 2
  • ELSE IF (Δx× Δy) < 0 → Sector 1
  • ELSE IF (Δx× Δy) > 0 → Sector 3
The architecture of the pen-coordinate extraction module is presented in Fig. 11(a) . The latter carries out a real-time scan of the whole image with a binary mask of size 5 × 5 pixels, selected according to the size of the pen tip. This task requires the use of four First In, First Out (FIFO) buffers, which are necessary for the construction of a 5 × 5 moving window. The obtained coordinates are finally used by a sector-location module (see Fig. 11(b) ) to extract the corresponding direction vectors (Dir0 to Dir7). The implementation costs of these two modules are presented in Table 4 .
PPT Slide
Lager Image
Pen-position extraction module: (a) pen-coordinate extraction module and (b) sector-location module.
Implementation costs of pen-coordinate extraction.
Logic utilization Used Available Utilization
Number of slices 322 5,120 6%
Number of slice flip-flops 520 10,240 5%
Number of 4-input LUTs 212 10,240 2%
Number of bonded IOBs 23 328 7%
Number of FIFO16s 4 40 10%
Number of GCLKs 1 16 6%
- 2. Real-Time Moving Hand–Surface Computation Module
The general structure of the moving hand–surface computation module is presented in Fig. 12 . Two dual-port memories are used to store the VPP x and HPP y vectors of the successive binary frames In and I n+1 . These vectors will be used to compute MHS x and MHS y according to (3) and (4). A primary implementation cost evaluation of these two principal modules is presented in Table 5 .
PPT Slide
Lager Image
Moving hand–surface computation modules.
Implementation costs of moving hand–surface computation modules.
Logic utilization Used Available Utilization
Number of slices 336 5,120 6%
Number of slice flip-flops 476 10,240 4%
Number of 4-input LUTs 742 10,240 7%
Number of bonded IOBs 24 328 7%
Number of BRAMs 5 40 12%
Number of GCLKs 2 16 12%
- 3. Discussion
The obtained implementation results show that the two reconstruction approaches are well suited for a hardware implementation within a dedicated smart system. Since FPGA is a runtime reconfigurable circuit, the accommodation of the related module inside it can be scheduled according to the ongoing mode of signature reconstruction. Likewise, the act of selecting the appropriate mode is done automatically depending on the state of the signatory (with or without hand problems).
To allow real-time viewing of a signature, a dedicated Zynq-7000 SoC, to embed the two signature reconstruction modules, and a video on-screen display (LogiCORE™ IP) [16] will be used to display the reconstructed signatures in real time on a small screen (see Fig. 13 ).
PPT Slide
Lager Image
Proposed smart signature acquisition system.
V. Conclusion
In this work, we have presented a new non-constraining optical hand signature reconstruction system based mainly on a transparent glass pad placed at the front of a color camera. The main advantage of the proposed system, compared to other acquisition systems, is its capability to perform transparent signature reconstructions based on both pen- and surface-tracking approaches.
Besides being robust to signature imitation attacks, the developed system opens the field to various non-constraining handwriting applications dedicated principally to people suffering from a physical or emotional condition. Moreover, the developed reconstruction techniques have low computational complexity and are therefore well suited to hardware implementation within a dedicated smart system.
Currently, our efforts are focused on the development of a dedicated signature authentication module based on both hand dynamics and morphology. This potentially efficient approach will allow a system to make a link between the signer and their signature, which consequently will improve authentication results.
This work was supported by the Ministry of Higher Education and Scientific Research, Algeria (PNR project reference: 42/TIC/2011, Design and Implementation of Multimodal Biometric Identification System).
BIO
Corresponding Author  belkacem.2006@hotmail.com
Belkacem Abbadi received his MS degree in electrical engineering from the University of Bordj Bou Arreridj, Algeria, in 2009. He is currently pursuing his PhD degree in reconfigurable architectures for image processing at the Materials and Electronic Systems Laboratory, University of Bordj Bou Arreridj. His research interests include real-time image processing and biometry.
mostefaimess@gmail.com
Messaoud Mostefai is a professor at the Computer Science Department, University of Bordj Bou Arreridj, Algeria. He supervises several PhD theses in computer science fields. He is currently responsible for the information theory group of the Materials and Electronic Systems Laboratory, University of Bordj Bou Arreridj. His research interests include flexible and real-time embedded systems and signal and image processing algorithms.
adel.oulefki@gmail.com
Adel Oulefki received his MS and PhD degrees in electrical engineering in 2009 and 2014, respectively, from the University of Bordj Bou Arreridj, Algeria. He is currently pursuing his post-doctoral research in image processing and classification at the Materials and Electronic Systems Laboratory, University of Bordj Bou Arreridj. His current research involves real-time pattern recognition and reconfigurable architectures.
References
Munich M.E. , Perona P. 2003 “Visual Identification by Signature Tracking,” IEEE Trans. Pattern Anal. Mach. Intell. 25 (2) 200 - 217    DOI : 10.1109/TPAMI.2003.1177152
Yu C.-C. 2012 “Video-Based Signature Verification and Pen-Grasping Posture Analysis for User-Dependent Identification Authentication,” IET Comput. Vis. 6 (5) 388 - 396    DOI : 10.1049/iet-cvi.2010.0136
Kholmatov A. , Yanikoglu B. 2005 “Identity Authentication Using Improved Online Signature Verification Method,” Pattern Recogn. Lett. 26 (15) 2400 - 2408    DOI : 10.1016/j.patrec.2005.04.017
Jain A.K. , Griess F.D. , Connell S.D. 2002 “Online Signature Verification,” Pattern Recogn. 35 (12) 2963 - 2972    DOI : 10.1016/S0031-3203(01)00240-0
Impedovo D. , Pirlo G. 2008 “Automatic Signature Verification: The State of the Art,” IEEE Trans. Syst., Man, Cybern. 38 (5) 609 - 635    DOI : 10.1109/TSMCC.2008.923866
Swanepoel J. , Coetzer J. 2013 “A Robust Dissimilarity Representation for Writer-Independent Signature Modeling,” IET Biometrics 2 (4) 159 - 168    DOI : 10.1049/iet-bmt.2013.0011
Gupta G.K. 2006 “The State of the Art in the On-Line Handwritten Signature Verification,” Tech. Rep. Monash University Melbourne, Australia
Oulefki A. 2013 “New Online Signature Acquisition System,” J. Electron. Imag. Lett. 22 (1) 010501 -    DOI : 10.1117/1.JEI.22.1.010501
Gupta G.K. , Joyce R.C. 2009 “A Study of Some Global Features in On-Line Handwritten Signature Varification,” Int. J. Autom. Identification Technol. 1 (2)
Findley L.-J. , Koller W.-C. 1995 Handbook of Tremor Disorders Marcel Dekker New York, USA “Definitions and Behavioral Classifications,” 1 - 5
Rocon E. 2007 “Mechanical Suppression of Essential Tremor,” Cerebellum 6 (1) 73 - 78    DOI : 10.1080/14734220601103037
Lee J. 2004 “Using Geometric Extrema for Segment-to-Segment Characteristics Comparison in Online Signature Verification,” Pattern Recogn. 37 (1) 93 - 103    DOI : 10.1016/S0031-3203(03)00229-2
Hartmut F. , Sadrozinski W. , Jinyuan W. 2011 “Applications of Field-Programmable Gate Arrays in Scientific Research,” Taylor & Francis New York, USA 1 - 4
Johston C.T. , Gribbon K.T. , Bailly D.G. “Implementing Image Processing Algorithms on FPGAs,” Proc. Electron. New Zealand Conf. Palmertson North, New Zealand Nov. 15–16, 2004 118 - 123
2007 Virtex-II Platform FPGAs: Complete Data Sheet v3.5 Xilinx Inc http://www.xilinx.com/support/documentation/data_sheets/ds031.pdf
2011 LogiCORE™ IP On-Screen Display User Guide v2.0 Xilinx Inc http://www.xilinx.com/support/documentation/ip_documentation/ug801_v_osd.pdf