So, like you Constant Readers have been reading, I’ve been fiddling around with a lot of new tools. Reticulum, FigureShift, IPFS (InterPlanetary File System), Python, SQLite3, Bears, etc. These things all spur new ideas and projects. Been toying a lot with the concepts of using cryptographic hashes as Universal Identifiers, localized replicated databases, Peer-to-Peer data synchronization – all that kind of jibbety-jabber that sums up to a future where TWDB could be a peer-to-peer app you could run on a laptop and have the entire site – photos, files, discussions and all right there locally – updating itself with new galleries, images and discussions anytime it can connect to TWDB Web, or in the case TWDB Web is gone, would sync with any other copy of TWDB Nomad it could find on any network it could reach via RNS. All cryptographically secure and as anonymous as you wanted it to be, and totally portable and replicatable via any other node that exists. This, I think, could be the solution to the “What happens to TWDB after you kick it?” and “How the hell do you accurately synchronize many copies of TWDB that are each independently updatable?” questions – actually real head-scratchers, those. You want a ROBUST DATASET that CANNOT BE KILLED no matter how many maintainers die or lose interest.
And that’s the thing that really needed to be answered: How to do it in a way that is not dependent on empowered admin-level maintainers, transferable but easily-lost skillets and a pool of steady income to rent a server to keep it all up and running? Haha! You thought the question was “Who takes over?”, and that was always the wrong question. The answer to *that* question was always “You guys”, and before now it was difficult to envision that and articulate it. But that is the plan, and it’s roughed out in a way that I’m pretty sure the tools are extant to do it. The details are being sketched in as I learn more and code, code, code, sketch outlines, dream, code, code, etc… It’s a process that’s not unlike a combination of painting and building a freeway system. Anyway, here’s a sneak peek of the preparations:
You see – over the past couple decades, I’ve seen a lot of great Typewriter information vanish into dumpsters and the scrapped hardware of defunct social media sites. I really don’t like that, and I’ve had this mission set in my brain since I built this version of TWDB and the Operation O.O.P.R.A.P. Project to save as much of it as I can, process it and turn it into a format that other people will be motivated to obtain, keep and treasure enough NOT TO THROW IT OUT. MOAR COPIES OF MOAR DATUMS == FEWER POINTS OF FAILURE. Creating the motivation to OBTAIN AND KEEP is just as or even more important than creating the mechanisms to enable that cornucopia of desire. I’ve put a lot of thought and testing into this theory and it’s pretty clear to me that if the desire exists, the mechanisms will be created and improved upon by the motivated. Create worth and what you create will be valued. (and saved)
If only…..
If only we had copies of all the great typosphere blogs that have died.
There were many sites and pages that vanished with Geocities and other places.
I like the idea of preservation.
Synchronizing all the data currently in the TWDB is going to be a nightmare.
I’ve been working on some Python just to synchronize all the drives I have had over the years and just sorting and verifying some small drives is frustrating.
Too much to go into for a comment
Yeah, synchronization is the big puzzler, but I think the hashed encrypted identifier and peer-to-peer synchronization of IPFS may be a solution. Basically, it would always be running in the background looking for other nodes and seeing if they have new or changed content to synchronize with. It allows you to synchronize an entire filesystem, photos, messages, data, whole shebang. At least I *think so*, it’s all very new to me. (:
There’s a huge rabbit hole here, but I’m still gathering data on the protocols. Studying I2P now..
Good stuff, preparing for resilience, EOL etc.
Distributed is good, but also hard (Lotus had it quite nicely with Notes.) Do you or The Archive perhaps make 6-monthly snapshots? Just in case?
Challenge will be that, though always under attack by scrapers and malicious actors (i.e. A.I.), the massive benefit of TCP/IP and HTML (3?) is its ubiquity. Metcalfe’s law and all that.
That having been said, having multiple Plan-B’s will be good! – I so recognize resources vanishing. (WayBack machine scraping is/always has to be spotty).
Yes. also just made a bot-grab of all the photos and rearrange them locally into directories of no more than 1000, so I can work with them. 270,000 photos in a single directory is fine for a webserver that has an indexed database of filenames to pick the file it wants, but if I try to do a directory listing — whew, mama – prepare for a client crash. :D
grabbing them all with a 1-second delay between each to be nice to the server took about 4 days. as of the snapshot time not quite a week ago: 270,042 photos @ 55gb.
That’s step 1 really, grabbing the assets and SQL to a local machine to set up an off-grid beginning to a Peer-to Peer immutable network. (or maybe even more than one)