S3 Information

February 14th, 2021

This page has information about the move of DigitalCorpora to Amazon S3.

Accessing the Corpus!

All of our site data is stored in the Amazon S3 bucket s3://digitalcorpora/. You can download from that bucket. We recommend using the bucket directly in Amazon’s cloud. We get free data storage and transfer from Amazon as part of the Amazon Open Data Program, for which we are thankful!

You can browse the S3 bucket directly using our JavaScript-based browser here: [S3 Browser]. It is fast, but it will not work with wget -r to download many files at once.

You can also browse using our server-based S3 gateway: [S3 Gateway]. It’s written in python. You can find the source code here.

You can also access this resource from the AWS command line interface with the command:

$ aws s3 ls s3://digitalcorpora/corpora/
                           PRE bin/
                           PRE drives/
                           PRE drives_bulk_extractor/
                           PRE drives_dfxml/
                           PRE files/
                           PRE hashes/
                           PRE mobile/
                           PRE packets/
                           PRE ram/
                           PRE scenarios/
                           PRE sql/
2020-11-21 10:56:19         43 README.txt
2020-11-21 10:56:20    1783404 digitalcorpora.org-hashdeep-2020-04-01.csv
2020-11-21 10:56:19    1787101 digitalcorpora.org-hashdeep-2020-05-01.csv
2020-11-21 10:56:19    1794086 digitalcorpora.org-hashdeep-2020-06-01.csv
2020-11-21 10:56:19    1794914 digitalcorpora.org-hashdeep-2020-07-01.csv
2020-11-21 10:56:20    1796103 digitalcorpora.org-hashdeep-2020-08-01.csv
2020-11-21 10:56:20    1796275 digitalcorpora.org-hashdeep-2020-09-01.csv
2020-11-21 10:56:20    1796447 digitalcorpora.org-hashdeep-2020-10-01.csv
2020-11-21 10:56:20    1796619 digitalcorpora.org-hashdeep-2020-11-01.csv
$

Tech Details

https://downloads.digitalcorpora.org/ is an app run on a Dreamhost virtual domain that has a Python program which makes the contents of the Amazon S3 bucket s3://digitalcorpora/ look like a Unix directory that is delivered using the Apache web server’s directory listing facility.

The s3://digitalcorpora/ website is hosted in the AWS region US West (Oregon) us-west-2.

The fastest way to access the corpus is by creating an EC2 VM in us-west-2 and accessing it directly.

Comments are closed.