Govdocs1

March 12th, 2012 Leave a comment Go to comments

Govdocs1 — (nearly) 1 million freely-redistributable files

In recent years a significant amount of forensic research has involved the analysis of files or file fragments. In the absence of such corpora, researchers and students who wish to work with files first need to collect files—a surprisingly difficult task if one wishes a large number of files of many types from a variety of sources. Although many files can be freely downloaded from the web, building and running a high performance document discovery and downloading tool is not a trivial task. Once files are downloaded they need to be analyzed, characterized and curated. Finally, many corpora that might be assembled cannot be easily redistributed due to privacy or copyright concerns.

For these reasons, we have created and released a corpus of 1 million documents that are freely available for research and may be (to the best of our knowledge) freely redistributed. These documents were obtained by performing searches for words randomly chosen from the Unix dictionary, numbers randomly chosen between 1 and 1 million, and randomized combinations of the two, for documents of specified file types that resided on web servers in the .gov domain using the Yahoo an Google search engines.

Each file in the corpus is presented as a numbered file with a file extension (e.g. 0000001.jpg). The file extension is typically the file extension that was provided to us when the file was downloaded. The file extension is a suggestion—it is not part of the corpus.

We are making the corpus available in several ways:

  • A set of 1000 directories, with 1000 files in each directory, downloadable from our server at http://digitalcorpora.org/corp/nps/files/govdocs1/.
  • A set of 1000 ZIP files, each with 1000 files, downloadable from our server at http://digitalcorpora.org/corp/nps/files/govdocs1/zipfiles/.
  • A a tar file with 109,282 JPEG pictures from the govdocs1m corpus at http://digitalcorpora.org/corp/nps/files/govdocs1/files.jpeg.tar.
  • As a set of 10 subset “threads” (subset0.zip through subset9.zip), each one containing containing 1000 randomly chosen documents. These subsets were specifically created for to facilitate pilot studies and student research projects with the rationale that it’s easier to work with 1000 files than with 1 million. Students are encouraged to use one subset for development and another subset for testing.

    1. Rob Zirnstein
      June 21st, 2010 at 08:05 | #1
      Reply | Quote

      As a software vendor, I think this resource is extremely valuable. It has given us the opportunity to test our tools against a wide range of files collected from the wild. This is much more useful that test data that we have created ourselves. Many software applications create documents with variations, in their file structures, that can cause major problems when identifying and processing files. Another benefit is the opportunity for multiple software vendors to test their tools against a public collection and provide comparable product comparisons.

      Thank you for this valuable resource!

      Rob Zirnstein
      Forensic Innovations

    1. No trackbacks yet.

 
"This material is based upon work supported by the National Science Foundation under Grant No. 0919593. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation."