This item has as tarball of 10,000 files, each with a large number of json records in it. For each non-dark item in the Internet Archive's collection (according to metamgr), we have a record of the metadata in json, then files.xml in json.

There if also a file with just the list of identifiers.

This is a form of record-of-our-times so we might, in the future, be able to say: we had this file in the archive at this time because we have the hash recorded here.

-brewster

time cat ../all-items-12-2015.txt | pv -acbrl | parallel --no-notice -j50 'curl -w"\n" -s "https://archive.org/metadata/{}/me\
tadata?dontcache" "https://archive.org/metadata/{}/files?dontcache" | gzip >> $(( RANDOM % 100 ))-{%}.json.gz'

plus-circle Add Review

comment

Reviews (1)