International Conference on Document Analysis and Recognition

ICDAR 2021: Document Analysis and Recognition – ICDAR 2021 Workshops pp 245-259| Cite as

Handwritten Chess Scoresheet Recognition Using a Convolutional BiLSTM Network

  • Owen Eicher
  • Denzel Farmer
  • Yiyan Li
  • Nishatul Majid

Conference paper

First Online: 04 September 2021

Part of the Lecture Notes in Computer Science book series (LNCS, volume 12916)


Chess players while playing in Over-the-Board (OTB) events use scoresheets to record their moves by hand, and later the event organizers digitize these sheets for an official record. This paper presents a framework for decoding these handwritten scoresheets automatically using a convolutional BiLSTM neural network designed and trained specifically to handle chess moves. Our proposed network is pretrained with the IAM handwriting dataset [1] and later fine-tuned with our own Handwritten Chess Scoresheet (HCS) dataset [23]. We also developed two basic post-processing schemes to improve accuracy by cross-checking moves between the two scoresheets collected from the players with white and black pieces. The autonomous post-processing involves no human input and achieves a Move Recognition Accuracy (MRA) of 90.1% on our test set. A second semi-autonomous algorithm involves requesting user input on certain unsettling cases. On our testing, this approach requests user input on 7% of the cases and increases MRA to 97.2%. Along with this recognition framework, we are also releasing the HCS dataset which contains scoresheets collected from actual chess events, digitized using standard cellphone cameras and tagged with associated ground truths. This is the first reported work for handwritten chess move recognition and we believe this has the potential to revolutionize the scoresheet digitization process for the thousands of chess events that happen each day.


Chess scoresheet recognition Offline handwriting recognition Convolutional BiLSTM network Latin handwriting recognition Handwritten chess dataset 

This is a preview of subscription content, log in to check access.



We want to express our sincerest gratitude to Mr. George Lundy, FIDE National Arbiter (US) at Chandra Alexis Chess Club, for his enthusiasm and support in this research.


  1. 1.

Marti, U.-V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. Int. J. Doc. Anal. Recogn. 5(1), 39–46 (2002). Scholar

  1. 2.

Eicher, O., Farmer, D., Li, Y., Majid, N.: Handwritten Chess Scoresheet dataset (2021).

  1. 3.

Eicher, O., Farmer, D., Li, Y., Majid, N.: Handwritten Chess Scoresheet Dataset (HCS) (2021).

  1. 4. How popular is chess? (2020).

  1. 5.

Sudharsan, R., Fung, A.: Reine chess (2020).

  1. 6.

Majid, N., Smith, E.H.B.: Segmentation-free Bangla offline handwriting recognition using sequential detection of characters and diacritics with a faster R-CNN. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 228–233. IEEE (2019)Google Scholar

  1. 7.

Breuel, T.M., Ul-Hasan, A., Al-Azawi, M.A., Shafait, F.: High-performance OCR for printed english and fraktur using LSTM networks. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 683–687 (2013)Google Scholar

  1. 8.

Shkarupa, Y., Mencis, R., Sabatelli, M.: Offline handwriting recognition using LSTM recurrent neural networks. In: The 28th Benelux Conference on Artificial Intelligence (2016)Google Scholar

  1. 9.

Dutta, K., Krishnan, P., Mathew, M., Jawahar, C.: Improving CNN-RNN hybrid networks for handwriting recognition. In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 80–85 (2018)Google Scholar

  1. 10.

Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation, arXiv preprint arXiv:1406.1078 (2014)

  1. 11.

Su, B., Zhang, X., Lu, S., Tan, C.L.: Segmented handwritten text recognition with recurrent neural network classifiers. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). IEEE, pp. 386–390 (2015)Google Scholar

  1. 12.

Suzuki, S.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)CrossRefGoogle Scholar

  1. 13.

Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2016)CrossRefGoogle Scholar

  1. 14.

Graves, A., Schmidhuber, J.: Offline handwriting recognition with multidimensional recurrent neural networks. In: Proceedings of the Advances in Neural Information Processing Systems, Vancouver, Canada, vol. 545, p. 552 (2009)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

About this paper


Cite this paper as:

Eicher O., Farmer D., Li Y., Majid N. (2021) Handwritten Chess Scoresheet Recognition Using a Convolutional BiLSTM Network. In: Barney Smith E.H., Pal U. (eds) Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021. Lecture Notes in Computer Science, vol 12916. Springer, Cham.

  • First Online04 September 2021
  • DOI
  • Publisher NameSpringer, Cham
  • Print ISBN978-3-030-86197-1
  • Online ISBN978-3-030-86198-8

Published in cooperation with

The International Association for Pattern Recognition

Total Page Visits: 807 - Today Page Visits: 2