Posts by Collection



OCR Graph Features for Manipulation Detection in Documents

Preprint, 2020

Detecting manipulations in digital documents is becoming increasingly important for information verification purposes. Due to the proliferation of image editing software, altering key information in documents has become widely accessible. Nearly all approaches in this domain rely on a procedural approach, using carefully generated features and a hand-tuned scoring system, rather than a data-driven and generalizable approach. We frame this issue as a graph comparison problem using the character bounding boxes, and propose a model that leverages graph features using OCR (Optical Character Recognition). Our model relies on a data-driven approach to detect alterations by training a random forest classifier on the graph-based OCR features. We evaluate our algorithm’s forgery detection performance on dataset constructed from real business documents with slight forgery imperfections. Our proposed model dramatically outperforms the most closely-related document manipulation detection model on this task. Download paper here

Recommended citation: Hailey James, Otkrist Gupta, and Dan Raviv. "OCR Graph Features for Manipulation Detection in Documents." arXiv preprint arXiv:2009.05158 (2020).

MRZ code extraction from visa and passport documents using convolutional neural networks

Preprint, 2020

Detecting and extracting information from Machine-Readable Zone (MRZ) on passports and visas is becoming increasingly important for verifying document authenticity. However, computer vision methods for performing similar tasks, such as optical character recognition (OCR), fail to extract the MRZ given digital images of passports with reasonable accuracy. We present a specially designed model based on convolutional neural networks that is able to successfully extract MRZ information from digital images of passports of arbitrary orientation and size. Our model achieved 100% MRZ detection rate and 98.36% character recognition macro-f1 score on a passport and visa dataset. Download paper here

Recommended citation: Yichuan Liu, Hailey James, Otkrist Gupta, Dan Raviv. "MRZ code extraction from visa and passport documents using convolutional neural networks." arXiv preprint arXiv:2009.05489 (2020).


Probabilistic Bias Mitigation in Word Embeddings

Published in NeurIPS Workshop on Human-Centered Machine Learning, 2019

It has been shown that word embeddings derived from large corpora tend to incorporate biases present in their training data. Various methods for mitigating these biases have been proposed, but recent work has demonstrated that these methods hide but fail to truly remove the biases, which can still be observed in word nearest-neighbor statistics. In this work we propose a probabilistic view of word embedding bias. We leverage this framework to present a novel method for mitigating bias which relies on probabilistic observations to yield a more robust bias mitigation algorithm. We demonstrate that this method effectively reduces bias according to three separate measures of bias while maintaining embedding quality across various popular benchmark semantic tasks. Download paper here

Recommended citation: Hailey James, David Alvarez-Melis. "Probabilistic Bias Mitigation in Word Embeddings." Neurips Workshop on Human-Centered Machine Learning (2019).

Printing and Scanning Attack for Image Counter Forensics

Published in EURASIP Journal on Image and Video Processing, 2022

Examining the authenticity of images has become increasingly important as manipulation tools become more accessible and advanced. Recent work has shown that while CNN-based image manipulation detectors can successfully identify manipulations, they are also vulnerable to adversarial attacks, ranging from simple double JPEG compression to advanced pixel-based perturbation. In this paper we explore another method of highly plausible attack: printing and scanning. We demonstrate the vulnerability of two state-of-the-art models to this type of attack. We also propose a new machine learning model that performs comparably to these state-of-the-art models when trained and validated on printed and scanned images. Of the three models, our proposed model outperforms the others when trained and validated on images from a single printer. To facilitate this exploration, we create a dataset of over 6,000 printed and scanned image blocks. Further analysis suggests that variation between images produced from different printers is significant, large enough that good validation accuracy on images from one printer does not imply similar validation accuracy on identical images from a different printer.Download paper here

Recommended citation: Hailey James, Otkrist Gupta, and Dan Raviv. "Printing and Scanning Attack for Image Counter Forensics." J Image Video Proc. 2022, 2 (2022).

Learning Document Graphs with Attention for Image Manipulation Detection

Published in ICPRAI International Conference on Pattern Recognition and Artificial Intelligence, 2022

Detecting manipulations in images is becoming increasingly important for combating misinformation and forgery. While recent advances in computer vision have lead to improved methods for detecting spliced images, most state-of-the-art methods fail when applied to images containing mostly text, such as images of documents. We propose a deep-learning method for detecting manipulations in images of documents which leverages the unique structured nature of these images in comparison with those of natural scenes. Specifically, we re-frame the classic image splice detection problem as a node classification problem, in which Optical Character Recognition (OCR) bounding boxes form nodes and edges are added according to an text-specific distance heuristic. We propose a system composed of a Variational Autoencoder (VAE)-based embedding algorithm and a graph neural network with attention, trained end-to-end for robust manipulation detection. Our proposed model outperforms both a state-of-the-art image splice detection method and a document-specific method.

Recommended citation: Hailey James, Otkrist Gupta, and Dan Raviv. "Learning Document Graphs with Attention for Image Manipulation Detection." ICPRAI 2022



CS50 Curriculum and Software Intern

Harvard University CS50, Cambridge, Massachusetts, 2017

CS50 is Harvard’s introductory computer science course. I spent the summer of 2017 working with the teaching staff to develop new curriculum and teaching materials for CS50. I also worked on an interview project aimed at increasing diversity in CS at Harvard.

AddisCoder Lecturer and Teaching Assistant

AddisCoder, Addis Ababa, Ethiopia, 2018

AddisCoder is a program created by professor Jelani Nelson to increase access to computer science and offer mentorship to high school students in Ethiopia. With the other teaching staff, I created curriculum, gave lectures, and worked individually with students in summer computer science course for 180 Ethiopian high school students. Following their participation in the program, several students have now been accepted to top universities in computer science.

Private Tutor and Teacher

The Academy at Harvard Square, Cambridge, Massachusetts, 2018

The Academy at Harvard Square is a tutoring center aimed at helping recent immigrants, including children, teenagers, and adults, achieve educational and career goals. I taught English-language, math, science and test preparation courses in addition to one-on-one tutoring sessions.

Patel Teaching Fellow

Harvard University, Cambridge, Massachusetts, 2018

The Patel Fellowship is a university-wide financial award to support equity and inclusion efforts in CS education. I used the fellowship to serve as a specialized teaching fellow for Professor Stuart Shieber, working with struggling students and developing pedagogy to support students of diverse backgrounds. My efforts included teaching functional programming to students in individual and small-group sessions 5-7 days per week, coordinating with teaching staff to ensure equity and inclusion, and using new data management practices to track student progress and validate pedagogical choices.