Episode Preservation Movement

[Wohlstand]

For such cases, I have my own server, where I had to store some of collected episodes.

[Koopulution]

How'd you make a server?

[Wohlstand]

How'd you make a server?

From a hardware I had, and with the Linux distribution (Debian initially, later CentOS). At first I had to use old computers for the server. Later, I bough the special server hardware (Supermicro motherboard, Intel Xeon E3-1220V6 CPU, and ECC-Unbuffered RAM module) that works stable and much better than my previous crap I had before. This server I use for many many purposes: mail server, XMPP, web server to host own and relative's and closed friends' sites; using it for archivation, backups, and storing various data, using as in-family shared storage, etc. My FoxyBot you can see at Discord, physically works at my home server.

Here are some photos you can see: https://wohlnet.ru/photos/Server/Server%206.0/

[Squishy Rex]

Taking into consideration the possible issues that may arise in the future with Google Drive a back-up of what I am working on, in terms of the graphics side of things will need to be added to Wohlstand's server as a precaution. With that said, based on my rough estimates, all versions of SMBX currently have around 3GB of data combined in the way of custom GFX, LUA scripts and other minor assets. This thankfully falls well below the 15GB limit of free GD accounts and should serve well enough as an archive of soley graphical and script based customizations for the general public to access and use as they please.

While I don't see any of the changes mentioned in the blog to pose a major risk, it will be something I will be keeping an eye on in any case just to be sure. Being that it is my account, I will at least receive notifications prior to any actions being taken if any happen to crop up.

However, as I said, this is primarily to serve the general public, however, it is in my best interests to also help get this stored safely as well. We've lost a lot of our history over the years with forum shutdowns and obviously users leaving or losing assets they may have accumulated over the years and we really need to do something about it. With SMBX2 now out there, we need this preservation now more than ever. There's no point in having a brilliant game while our assets and tools are here, there and everywhere.

[OptimusPrimeformers]

I downloaded most of the episodes from the forums (and a few from rpgmaker.net) over the past few months (about 450 episodes), and am currently in the process of downloading all the levels. I can upload them to Google Drive or GitHub, but everything would not be able to fit into the storage space (the episodes take up 60+ GB of space on my computer). Also, I have never heard about GitHub before, can someone explain how it works?

[Wohlstand]

I downloaded most of the episodes from the forums (and a few from rpgmaker.net) over the past few months (about 450 episodes), and am currently in the process of downloading all the levels. I can upload them to Google Drive or GitHub, but everything would not be able to fit into the storage space (the episodes take up 60+ GB of space on my computer). Also, I have never heard about GitHub before, can someone explain how it works?

In addition I can give you access to my FTP upload where I dedicated 500 GB of my extra HDD for episodes uploading (uploaded stuff will appear here). Please PM me for details, or join the Discord server of the Preservation Team, and DM me. I highly recommend you to use the same place for uploads to unite our common goal and make the work together. At my archive is about 35 GB of episodes already uploaded.

P.S. GitHub is a thing that is NOT suggested for preservation, because if they find you do abuse repositories by storing archives on it, you'll get the notify about the removal of it. Developers of GApps (or another thing, don't remind) got that case because they stored a huge big set of archives at the GitHub repository. GoogleDrive usually gives 15 GB of the place, and extra space can be bought for money.

[TDK]

Spoiler: show

As of tilde five minutes ago, I was made aware of this occurrence on Mediafire, and that raises a huge problem for older SMBX episodes. This problem would effect most all old SMBX episodes, and would make them un-installable for newer members of the community.

This, however, is why I've made this post; I'm going to begin downloading SMBX Episodes on my own, and hosting them in places where I believe they won't get smote (Google Drive, MEGA, etc.) and I'm inviting everyone to help. That's right, everyone. And you know who that includes? You. I can't do this alone (I'm only one guy!) so I'm most likely going to need the help of at least three or four people, but extra hands are always welcome.

I don't think I can do this alone, but we can do this together.

Discord Link is in the image; click it

Are you planning to preserve SMBX levels as well?

[Wohlstand]

> Are you planning to preserve SMBX levels as well?

Mainly episodes, but a good idea to also preserve single levels if you have a desire.

[jumpykrusty]

In addition I can give you access to my FTP upload where I dedicated 500 GB of my extra HDD for episodes uploading (uploaded stuff will appear here). Please PM me for details, or join the Discord server of the Preservation Team, and DM me. I highly recommend you to use the same place for uploads to unite our common goal and make the work together. At my archive is about 35 GB of episodes already uploaded.

P.S. GitHub is a thing that is NOT suggested for preservation, because if they find you do abuse repositories by storing archives on it, you'll get the notify about the removal of it. Developers of GApps (or another thing, don't remind) got that case because they stored a huge big set of archives at the GitHub repository. GoogleDrive usually gives 15 GB of the place, and extra space can be bought for money.

Lots of games are archived at the Internet Archive, such as this collection. It has been a non-profit long-term archiving platform for 20+ years and is very well established. I think it would be great to archive SMBX episodes/levels there. What's the total size of the SMBX archive so far? The Internet Archive is capable of hosting archives hundreds of GBs if not more!

[Wohlstand]

jumpykrusty, we with folks had to collect the huge amount of episodes, single levels, packed games, and other misc assets at here: https://smbxarchive.wohlsoft.ru/

Currently, there is 228 Gigabytes already collected.

[jumpykrusty]

jumpykrusty, we with folks had to collect the huge amount of episodes, single levels, packed games, and other misc assets at here: https://smbxarchive.wohlsoft.ru/

Currently, there is 228 Gigabytes already collected.

Great! 228 Gigabytes should be fine for archiving in the Internet Archive. If we store it there, then there wouldn't be any hosting costs, and there wouldn't be any risk of smbxarchive.wohlsoft.ru disappearing if the person(s) who maintain it leave(s). I'd be willing to help upload the current archive to the Internet Archive, but the thing is someone needs to keep it up to date.

[Emral]

It's a great archive for finding all kinds of episodes and wonderfully maintained. What do you think of the ability to download episodes from this website through the SMBX2 launcher? It would be great to grab episode information via a Rest API and then download them directly into the worlds folder I think. Do you think that's something that can be achieved?

[Luna]

If it was possible, I think it would be a great way to have an episode download function built-in into the episode.
Would probably have be linked to both the 1.3 and the X2 folders, though.

[Emral]

If it was possible, I think it would be a great way to have an episode download function built-in into the episode.
Would probably have be linked to both the 1.3 and the X2 folders, though.

If a REST API is possible, the desired compatibility of episodes to be fetched can be requested in the call to the server. So SMBX2 launcher can request different episodes than some other program interfacing with the site would.

[Wohlstand]

Speaking about REST API, there is a sort of database to be build is needed. I also had the idea to make the sort of metadata to simplify others to make various software to browse content or archive. But there was more general idea of the thing that can be used by any SMBX branch to easily retrieve compatible episodes from the archive. But keep in a note that there is not full set of data provided as part of archive collected out of smbx forum digging (taken from other sites, or from personal archives of people). So, just better to build the simple database that will contain only minimal set:
- episode title
- archive filename and path
- compatible engines list
- author
- release date (if available)
- original forum/page link (if available)
- brief description (copied from forum post/etc. Also, if available, or manually written if a well-known episode)
- special note (from maintainers of the archive, for example, if it's a fixed/ported version because original is unbeatable and original creator is unavailable, or something also)

Right now, almost every episode entry contains a text file with description, that can be used to build the metadata.

Added in 16 minutes :
P.S. Don't forget that archive may grow in any time once any episodes has been added, and we don't want to add more buerocracy on registering each episode at the database. But could be made some automatizer that will read text file at new-added episode and fill basic content that can be extended later by manual tweaks.

[Emral]

But keep in a note that there is not full set of data provided as part of archive collected out of smbx forum digging (taken from other sites, or from personal archives of people). So, just better to build the simple database that will contain only minimal set:
- episode title
- archive filename and path
- compatible engines list
- author
- release date (if available)
- original forum/page link (if available)
- brief description (copied from forum post/etc. Also, if available, or manually written if a well-known episode)
- special note (from maintainers of the archive, for example, if it's a fixed/ported version because original is unbeatable and original creator is unavailable, or something also)

Right now, almost every episode entry contains a text file with description, that can be used to build the metadata.

Yeah, I was thinking that the txt could be used to build the metadata. Though I'm aware that it might be a lot of work if the format isn't uniform. Some things to note:
- Compatible engines list should distinguish between engine (legacy, smbx2, 38a, xtech, etc) and there should be a second one (or a list entry) for "intended version" or a list of confirmed working versions. This isn't too important for 1.0-1.3, but for 1.4.4 vs 1.4.5 and 2.0 beta 3 vs 2.0 beta 4 it can be very important.
- Author should be a list, so that teams can be credited and searched for
- If possible, a versioning system could be very nice with regards to release date and such. If you have a separate database table with the episode id as key and versions as the rows. Then all versions could be fetched. That might be useful if an episode has a Beta 3 version and a Beta 4 version, or a 1.3 version and an XTech version.
- On top of a brief description, a screenshot or set of screenshots would also be nice to have, though by no means necessary for a first draft of this idea.
- Notes can be good.

If this is successful and we want to expand it even further, here are some more ideas that cooould make sense but also require the main site to have a lot more functionality.
1) Tracking recent downloads to determine "popularity".
2) Fetching in order of upload date.
3) 5-star Rating system?

Again, those are probably more effort so I'm just mentioning them for the record.

Added in 16 minutes :
P.S. Don't forget that archive may grow in any time once any episodes has been added, and we don't want to add more buerocracy on registering each episode at the database. But could be made some automatizer that will read text file at new-added episode and fill basic content that can be extended later by manual tweaks.

Yep! Adding an episode should be simple. And I think that it wouldn't be too much effort to allow uploads via a FORM that uploads the info into the database down the road. Though it might be good to have entries manually verified, to prevent trolling. The TXT file could be generated from the database information, once that is set up. That should keep the complexity low from the submitter's side.

[Luna]

Added in 16 minutes :
P.S. Don't forget that archive may grow in any time once any episodes has been added, and we don't want to add more buerocracy on registering each episode at the database. But could be made some automatizer that will read text file at new-added episode and fill basic content that can be extended later by manual tweaks.

Yep! Adding an episode should be simple. And I think that it wouldn't be too much effort to allow uploads via a FORM that uploads the info into the database down the road. Though it might be good to have entries manually verified, to prevent trolling. The TXT file could be generated from the database information, once that is set up. That should keep the complexity low from the submitter's side.

Note that everything that is available on the archive has been added manually as of now. Currently, I occasionally scan the forums for new releases and add them to the archives (same for Discord or the WohlSoft forums; and sometimes also episodes that are released to other sites such as YouTube).

[Wohlstand]

should be a second one (or a list entry) for "intended version" or a list of confirmed working versions.

Absolutely right, because SMBX-38A and X2 have incompatibilities between different versions. And it's needed to mention which version is required to run the episode for the case if it won't work on others. Also, for the case of TheXTech, the "minimum version" is needed, mainly because every new version adds new functionality which is not available in the older version.

- If possible, a versioning system could be very nice with regards to release date and such. If you have a separate database table with the episode id as key and versions as the rows. Then all versions could be fetched. That might be useful if an episode has a Beta 3 version and a Beta 4 version, or a 1.3 version and an XTech version.

Yeah, as a way to squish different ports of the same episode into the same entry, but give different archives for each selected engine and version, or just play different variations of the same episode.

- On top of a brief description, a screenshot or set of screenshots would also be nice to have, though by no means necessary for the first draft of this idea.

The screenshot (s) set is for future work after the main base will be built. Mainly, because lots of original screenshots have been lost, and there are few parts of them got been rescued, so, will need to make them by selves.

If this is successful and we want to expand it even further, here are some more ideas that cooould make sense but also require the main site to have a lot more functionality.

All such extra functionality could be done on the wohlsoft side (including database itself), keeping episodes and all stuff physically stored on wohlnet (and possibly on other mirrors that has enough space, at least 500 GB, currently 237 GB is used)

[Emral]

Dunno how to upload episodes to this but someone should go through this page and download everything these users linked to that's still available.
https://smbxjapan.wiki.fc2.com/wiki/%E3 ... 9%E4%BB%8B

[ShadowStarX]

Is that link only accessible via a Japanese region? (so for most people, via a VPN)