Please take the joint.
Thank you very much, and this is such an honor to be here.
I hope that I will speak short enough so that we can have conversation.
So my job here is hopefully to be useful to some of the projects that you're
doing, but also to provoke conversation, to go and figure out what do we do
now in this digital era.
One of my favorite topics is digital preservation, and so the opportunity to come
here was not to be passed off.
So some of the lessons that we have learned by doing large scale preservation is
what I would like to talk about today.
The Internet Archive, Nonprofit Research Library, this is what the
building looks like in San Francisco.
This is the Internet Archive Canada in Vancouver.
So please visit in these physical locations.
Being physical is still so much more fun than being on Zoom.
A book that I think is important for this conversation is recently published
called The Library, A Fragile History.
What happens to libraries?
The answer is it's not good.
In general, they are destroyed, and they're destroyed usually on purpose, and
they're destroyed by those that are in power.
And this library has survived, which is fantastic.
There are two walls around this particular building before
you get out to the public.
I don't think that's a mistake.
So I think that if we're thinking about long term, we have to think about what's
the pressures on this.
A lesson number one out of all of this is access drives preservation.
If you want funding, provide access.
If you want something to be preserved, provide access.
If you go and just make dark archives, they will be forgotten.
They may not have been done right, and nobody will fund you.
So I would suggest that is one of the lessons I have learned.
Okay, this is a long outline, but it is in themes, and I will go fast.
But the idea of what are some of the lessons and approaches that I've seen work
in preserving analog, digital, and end with sort of where should we go now.
And that's what I suggest for the end of this talk.
Preserving analog.
Digitizing materials.
I would say actually that long list, it's harder as we go down.
It may sound like digitizing a million pages a day, which we do,
3,000 books a day.
A million books a year is hard.
Yeah, kind of.
It's hard to get going, but it's not the hardest thing.
So this is microphone digitization down the center.
There's books on the sides.
There are hundreds of people doing it.
It's a shipping container every eight days of materials that we're able
to digitize.
We're also digitizing bolognese.
We wanted everything ever written in a language.
I think this is a good idea to go and preserve these languages and support
people that are in diaspora.
And the first people to say yes were the bolognese.
And it's an island with three million people.
They're very proud of their culture, and they want to make it last.
So they said yes, let's digitize all of it.
They write on palm leaf manuscripts.
So talk about a task for AI to help in some of the optical character recognition
of the handwritten, what we've got in it, I call it.
And I'm just trying to show you some diversity of what it is that
we've had to deal with at scale.
This is microfilm.
This is the reels.
On the side there are master copies of this
particular vendor called University Microfilms.
And it is a, they went out of business.
They were going to throw it away.
So we bought it.
And so those are the master copies.
And to show, so each one of those is a year in those canisters.
So each one of those canisters holds maybe 20 years.
And that is a picture of one row of one floor of the vault that we bought.
If you look in the center of that, it's me and really small.
So it's, so you can even scale to these sorts of sizes.
Doing all along playing records, effort, it's not that hard.
All CDs, effort, it's just not that hard.
By the scales of what we libraries do, we're fairly small.
And we can even do this by working together.
Lesson number two, work with other libraries.
Mission align organizations.
If they're not aligned, don't work with them as closely.
Because then you will go in different paths.
And working, we found partnerships with libraries to be key in the success.
So where do we get the hundreds of thousands of CDs?
Where LPs is from other libraries and large collections.
So it's how we work together with mission aligned organizations to do what each
one of us is very good at.
As preserving analog, preserving digital, more digital materials.
This is with IFLA website, looked like in 1998.
Which is kind of fun.
But we're probably best known for the Wayback Machine.
We do all sorts of other things.
We've been collecting away and making it available.
How we do this, we work with other libraries.
We have 1,200 other libraries that we work with.
Including many national libraries there now.
Just in the archive of it, which is the small collections portal project.
It's 6.6 petabytes.
And we always emphasize for people to take their bits home.
But often they don't.
So they leave them with us for the while.
Please, think about taking your bits home.
So anyway, so this is the archive function.
And so we've been digitizing more digital again
in this sort of coordinated approach.
Preserving bits.
Well, you have to keep copying them forward.
And we don't really like the idea of cloud computing.
So the service that I sold to Amazon was an architecture that became AWS.
The Amazon Web Services.
But we don't use Amazon Web Services.
We go and use things that are on campus.
So that we have them on control inside our walls as best we can.
This is just a parade of pictures of what it is our
servers look like over the years.
When we're first starting, it was tape robots.
Because hard drives were too expensive.
This is the partial copy that's in the library of Alexandria in Egypt.
For real.
Again, preservation strategy.
It's worked with other libraries.
So they re-architected their first floor.
So blinking lights could be shown to visitors of the library.
And I highly recommend you go.
It's beautiful.
This is what the first part of when we started putting things into the Stricting
Internet Archive in the Netherlands.
Thanks to Access Paul.
This is 2008.
A row of the machines that we designed and built.
In 2009, we had this idea of making data centers out of shipping containers.
Because I hate building data centers.
They're kind of awful.
So, and Sun and Google and Microsoft, Yahoo all ran with it.
And Sun actually gave us one.
And so we put the lead-back machine in a shipping container.
So I get my favorite joke.
Which is, you see, how big is the web?
The web is 20 feet by 8 feet by 8 feet.
And it weighs 26,000 pounds.
At least at this time.
There was about 85 micrograms a hit.
So that was some big re-back machine was.
This is what the web architecture looks like now.
We bought an old church in San Francisco.
A Christian Science church.
And we actually had the machines in the great room.
We made them quiet enough that they're actually in the great room.
And they blink.
And every time a light flinks is somebody
uploading something or downloading something.
It's about 150 petabytes.
And I know you're thinking, wow, that's a whole lot.
Well, not just in these.
There's lots more.
But it's 150 petabytes.
But we were hearing about a major book vendor here in Italy that has a half a
million books.
A half a million e-books.
E-books is typically one megabyte.
One megabyte times 500,000 is 500 gigabytes.
That's as much as it is in my phone.
[...]
And it's much less that's in my laptop.
This kind of numbers are just not that big.
Our libraries can handle it.
We should buy the bits on the bits.
Because we should go and manage our own collections.
That's what we do as our tradition.
But if you scale up to lots and lots of things, you end up with 150 petabytes.
Okay.
And we're now offering a service called Vault that is mission-aligned for people
that don't have the facilities on your campus or
don't have it on your campus yet.
So you can go and store with us.
We have 50 organizations that have something different than beta.
We have a national archive that just put everything they have in it.
Anyway, that kind of thing.
So that's how to store bits.
I'm going to keep charging for preserving links and context.
This was referred to in the talk about the Italian collection.
The nations are quite the right way to look at it because things link in and out.
And the World Wide Web, or if you take papers that have footnotes, citations at
the bottom, you want to be able to go to the footnote and go.
So how do you do that?
Well, we need new interlibrary loan services that operate digitally at speed.
But we also need to go and have our collections larger than just one
nation, maybe as one field and the like.
And we start to work together in these ways because just the idea of just the
only being the Internet archive is not the right idea.
We have to work together.
So we have done a lot of work to go and fix the links in Wikipedia.
There are a lot of footnotes that have links to web pages.
We collect all of those web pages once it's authored.
And when they break, we go and put in a link to the Wayback Machine.
We have now fixed 20 million broken links.
Woo-hoo!
So that's great.
So people can now count on the Wikipedia more.
And you have to go and do that to go and fix these things.
It's kind of broken.
That's a problem with the World Wide Web, working with Tinder and
Earthly to try to fix slowly.
But until then, we're going to try to put these patches on top of it.
We've also gone and we've woven in links to books.
So we prioritize digitizing all the books that are referenced to Wikipedia so you
can click on them and open right to the right page.
Check the reference.
Go deeper.
If you get a page or two, and if you want more than
that, then you have to check out the book.
But you need page numbers.
E-books don't have page numbers.
It's a problem.
So that's one of the reasons why we need digitized
physical books, is for the citations.
This is a Wikipedia page that referred to a Canadian study that the
government did about it in Somalia.
And when the new administration came in, they took it all out.
So they, now it's the Wayback Machine.
Okay, so that's trying to preserve our digital.
How do you preserve formats?
Well, this is a very popular game from early
1990s.
Actually, I just met this week the author of Prince of Persia.
And he's got a book, a memoir.
And so I got to meet him and he was so happy that we made his game available.
There are about two million people who have played that game in emulation
on the Internet Archive website.
So I'm very glad to hear that the Chinese delegation here is working on
preserving software and doing that kind of thing as well.
Also, types of software that have gone under is Adobe Splash, which a lot of
children have been brought up on Flash games.
And they can't play anymore because one company stopped supporting it.
But to the good side is they didn't go and sue a
company that was making an emulator.
And so we worked with them and then we brought a lot of these Flash games back to
life and much to the happiness of many grown up kids.
And so all of us go.
Preserving the Internet Archive.
So, okay, so these are some of the things that you may have been dealing with.
How do you preserve bits? How do you digitize and scale? How do you do
that? How do you link all these things together?
But what about your institution itself?
We're working to try to preserve the Internet Archive.
How? Well, there's some problems.
We're being actively sued by two groups.
The largest book publishers in the world are suing the Internet Archive to try to
make sure that people can't borrow books.
That they can't own digital books.
So their business model is all about being a Netflix of books, making sure nobody
can own an e-book.
So my question earlier, me it sounded a little bit rude, but I
apologize if it did.
But it's really to try to draw out our libraries owning their
digital materials anymore.
And we think by going and digitizing things we own, we do.
The publishers are saying no.
I'll end with the digital ownership.
Then that is under appeal in the United States.
The first district court was not favorable to libraries.
So that's really not good. Even though there are now standards, it's built into
the NYISO system.
It's built into library platforms, this digital lending.
The courts are saying they don't really want libraries to have digital
files going forward.
It's crippling if that is allowed to stand.
And then the same lawyers, within one hour of that judgment, sued us again.
But on the behalf of the music industry, about 3,078 RPM records.
So we've digitized 450,000. They've been up for 15 years.
And they decided to sue about 3,000. Not tell us to take them down, just sue us.
And they're demanding us under the United States law, 400 million dollars.
So this is what it is the publishers are.
And what they're doing to libraries. In our case, they're also doing it to many
other libraries in the United States.
The libraries in the United States are under attack.
You've probably heard about the book beginnings.
But you may not have heard about all the defundings that are going on.
As well as these license deals that are cranking up the cost of e-books in such a
way that libraries can't have anything but the best sellers.
It's in a problematic. And they're suing not just us, but others.
So it's a time that libraries need to stand together and stand up to be able to
be able to do our basic functions in an environment that's becoming more and more
hostile, at least in the United States.
Hasn't seem to have hit Europe in quite the same way yet.
Preserving the library system.
So if we're going to have a library system that does the basic functions of a
library system, then we're going to need to do some work.
If we don't have digital ownership, then they're going to draw a line and
say, yes, you can have physical books.
You can digitize the out of copyright ones.
But everything else. Nope.
This is a problem.
That means that we if there's a line there that says that we're going to end the
library system with digital, then we don't have a mechanism of keeping a
history together.
If you don't have digital ownership, then the publishers can change books at
any time and they do.
And they can withdraw them.
So it could be the end of history because history is not ours to own.
So on the US library system, there's just some articles.
You can't buy this books by Maria Bustillos.
And another article by me about whether there's going to be libraries in any of
the current sense of what a library is in 25 years.
Again, access drives preservation. That's how we're going to survive.
So some of the library services that are still not will
be legal in the United States.
There's interlibrary loan. There's access for the print disabled. Lending is
problematic.
We have archiving partnerships.
So what now? What should we do now?
I would suggest we need to go and reinforce what libraries have always done.
And it's four points.
What libraries are is we collect things, we preserve them, we provide
access to those materials, and we cooperate with other libraries.
And right now, all four of those things, core features libraries do
in the digital world are under threat.
So let's make sure that these survive. Let's speak a little bit more. What is
collecting digital materials mean?
We need to be able to legally straightforwardly collect materials and
make that a known thing that we can, should, or obligated
to do.
We need to be able to preserve those materials by doing format shifting and
moving from one medium to the next.
We need to be able to lend and provide access to materials,
say to one person at a time.
The same way that we've always done it.
Can we go and buy things and perform our function in those ways?
Cooperate within the libraries by sharing with interlibrary loan, or actually
moving our collections from one place to another so that as disasters will fall,
which always do, that we can survive more readily than, say, a
lot of the libraries in Europe during this 20th century.
So those four, I would say, are absolutely key.
And we're working now with other jurisdictions to try to find ones
that will sign up for these.
And if a jurisdiction will sign up for these, there are foundations lined up
to go and build digital libraries in the nonprofit on soil and put
their materials back in their country.
That's good. It can be our Carnegie moment, or Andrew Carnegie in the United
States said, if you will commit to go and staffing this
library, I'll build it for you.
But you have to sign up. Maybe it's time again to go and have countries sign up
for going on and doing it. We're in Nevada.
It's solid. Let's do it here. Let's do it every
as part of this trip going to Aruba.
And I think that they may adapt it, adopt it, at least at the library level. And
I'm hoping within their legislature.
Who will support digital libraries going forward to make it straightforward to do
our jobs? It shouldn't be scary to be a librarian.
And this is why I'm here for you. So if you want to discuss any of this, please,
we're very much out of light to you.
So what's next? I'd say it's an AI opportunity. And as was mentioned, I helped
start a company called Thinking Machines.
I was schooled in artificial intelligence. The idea of where data is stored. I
needed to go and get a lot of data. So for the coming machines.
And there's a lot of hope. Wait. A lot of these technologies from my perspective
and what I helped push forward is to make it so that we're going to end up being
able to consult the magnitudes of the works of human kind that people aren't
going to read every part of again.
Right. It's just that they're not we're not reading those. So how do we go and
bring them back to life? And some of this artificial intelligence
technology is pretty great.
The last wave of artificial intelligence technology, which was search engines,
also worked really well. When you digitize even old stuff and put them up on the
archive, people make them.
We're about the 200 most popular website in the world. OK, there's 200 more
popular, but there are 100 million less popular. So people want old stuff.
So if we make it available to new in different ways. So I would suggest a next
challenge is to try to figure out how to fair it. And with compensation or
whatever it's going on work, build digital libraries for thinking machines.
Thank you very much.