DeExcelarator

About

(Refer to the download section,  for materials used in the experimental evaluations of our publications)

Figure 1: Extracting Information from Diverse Spreadsheets

There are many sources of spreadsheet files out there, as shown in the figure above.  We aim at processing such diverse collections of spreadsheets, in a mostly automatic fashion. The inferred meta-information (e.g., layout and structure of the contents) can be attached to the original file, so to be reused for future information retrieval tasks. However, this meta-information could also serve other purposes, such as data transformation, compliance, and fault detection.

Figure 2: Cell Layout Roles as defined in the DeExcelerator Project

The first step of our proposed solution is layout inference. Contrary to related work, we focus on the cell level. The goal is to predict the layout role of each cell. For this, we trained a classifier in a supervised fashion, considering a wide range of features, many of which not covered before in the literature. We study formatting, contents, formula references, as well as the surroundings of the cell (i.e., features of neighboring cells), to predict its layout.  Furthermore, our work includes novel techniques for detecting and repairing incorrectly classified cells in a post-processing step. The experimental evaluation shows that our approach delivers high accuracy, in a dataset with a considerable diversity of cells. More specifically, our dataset consists of 465 Excel sheets (216 files), randomly selected from three distinct corpora with different characteristics: FUSE, ENRON, and EUSES.

Secondly, we focus on the table recognition task. We adhere to identify and extract data from tables, since they are a great source of structured and factual information. In comparison to related work, we avoid assumptions with respect to the number of tables and their arrangement in the sheet. Additionally, we enforce a certain level of flexibility, which allows us to work even with problematic tables (e.g., containing misclassifications and/or empty cells). Thus, we do not limit our approach to a set of predefined expected formats. This goes hand to hand with our decision to work on the cell level, as our overall goal is to handle spreadsheets differing in layouts and contents.

Figure 3: Table Identification via a Graph Representation

This project among other has produced two publications on spreadsheet table recognition. The figure above is taken from our latest paper, where we use a graph representation to identify candidate tables.  After classification, we group together cells of the same layout role to form coherent regions. These regions correspond to the nodes of the graph. While the edges capture the spatial arrangement of the regions in relation to each other. For more details, please check the related publications.

In addition, this work has produced an annotation tool, which we have used to generate training and testing datasets. The video below demonstrates the tool

Downloads

Use the following links to access the components used for this research project.

• ECIR Dataset of Annotated Spreadsheets
• ECIR Source Code
• Source Code for Spreadsheet Annotation Tool (Git Hub) *
• Executable Jar for the Spreadsheet Annotation Tool *
• KDIR_Annotated Excel Spreadsheets
• KDIR_Full Cell Dataset as Weka ARFF File (Original)
• KDIR_Full Cell Dataset as Weka ARFF File (43 Selected + 6 String Features)
• KDIR_Downsampled Cell Dataset as Weka ARFF File (Original)
• KDIR_Downsampled Cell Dataset as Weka ARFF File (43 Selected + 6 String Features)

* The Spreadsheet Annotator tool it is still in development phase. It has been tested so far in Windows 7 SP1 with Microsoft Office Excel 2013.

Related Publications