I’m sharing a message below from the 4 superstars behind the migration of the DiS boards from Discourse hosting to self-hosting. I am not personally actively involved in the process but I’m looped in on messages etc. You’re all welcome to contact me with any concerns or questions in addition to the migration team, but for the more technical/practical side of things I won’t be able to answer much without checking in with them just so you know.
Here goes:
(Identical versions of this post are posted in both Music and Social to make sure everyone/as many people as possible sees it)
Having been inspired by all the individual user list threads I was going to start a thread for my 100 best albums of all time that were released last month but I’ll wait for the migration I guess.
As you all are aware, I deliberately lowball all my posts so it’ll be good to know when I can let go with some pure gold, safe in the knowledge there’ll be no permanent evidence of when I put the rest of you to shame and you can all keep your reps in tact.
Apologies, as @1101010 pointed out, I didn’t post in this thread to say the back up began moments after this thread was started - Discourse have to do this for us and they started it almost instantly, which took us somewhat by surprise (tbf, they’ve given us 6 months of free hosting and wanted us off by end of the month).
This means posts and replies in last 24 hours and until the new site is online, will likely not transition over to the new forum.
The backup was still completing over night so will have an update from @megalithicrock and @zeal soon.
Morning all - no update from Discourse yet unfortunately, hopefully hear something soon. They are taking a complete backup of the site, and have said they will notify us once it is ready for us to download. Then we will transfer it to the new environment and start the restoration process. There are some fiddly steps, especially around the transfer of uploads (images), but won’t bore you all with the detail!
Just hadn’t realised the DB backup system worked by (I guess?) taking an immediate shadow copy of the DB at the start and then building it off that, or possibly it’s using a date time on the Transaction Log? I think I’ve always assumed you want everyone out of a system when running the backup but maybe that’s just because of resource hogging or if it crashes and such. Not really a consideration here.
It takes a back up every day but this is the full back up of the images (including all the high res images snd gifs we ended up hosting due to a set up issue in early days) and I guess more about the structure of everything.
I think it may also be taking back ups of the back ups which is why it’s taking forever but I’m not sure.
Almost all websites will be taking backups without any downtime. Yeah like @whiterussian said, the backup should be a point-in-time snapshot of the site at some point fairly soon after the backup was initiated (without going into too much technical detail). So this was yesterday late morning.
I do think it is a bit strange it is taking quite so long though, was expecting the backup to be ready for us to download this morning.
Oh we did have the option of making the site read-only during this process, but didn’t opt for that.
Yeah it’s more the length of time I’m confused about. I figured it would be taking incremental snapshots or similar, if it’s being constantly backed up, so the process would be quick and it’s just a case of copying the data.
If you’ve got a daily backup system that takes more than a day each time then that just seems like an issue. Even if it’s weekly, you’re leaving a lot of time for it to fall over.
The daily / nightly backup is DB only which is relatively small - but this backup includes all the uploads (images) which are stored in S3. So guess that is the slow bit, but would still have thought it would take 2-4 hours rather than 12+
I guess in my mind these should be two separate things, really:
You backup all the images etc. then copy them over to the new repository, then you grab a DB backup. It feels odd that they’re mixing the two but maybe they consider it worse to have site content with possible lost image data vs entirely lost data.
Yep totally agree with you and I was pushing for that, but Discourse said not possible. Most efficient way would have been to transfer the images from their S3 bucket to our new one directly - instead they are being added to a ZIP file which I have to download, then re-upload, then run a restore process which will add them all to local storage, then kick off a process to push them all back out to our S3 bucket finally. Painful.
