As most of you know, I'm currently running a Python script (https://github.com/Frenzie/myopera-backup) with the intent of backing up all of the My Opera forums, or at least about as close to it as possible.
I already shared a few hundred thousand posts using My Opera itself, but obviously that would be self-defeating. I considered torrents. They have many advantages, not least of which is that the more popular they are the faster they (ought to) go. But it would be more likely to be a constant drain on my Internet volume, which is currently limited to a minuscule 100GB. (I can make do with it, and I don't want to pay more.) I don't think the ISPs here understand that even buying a game on Steam these days might use up 100GB within a half hour.
Anyway, the best solution seems to be sharing it all on Wuala (https://www.wuala.com/Frenzie/myopera-backup-data/). Downloading all the files should be easy through the Opera Links panel, or some equivalent extension on another browser. Sharing it from this very server is also an option, but in the unlikely case that there's a lot of interest I figure it's better to have it elsewhere.
Download backup files from Wuala (https://www.wuala.com/Frenzie/myopera-backup-data/)
I won't share too much until I've done a data integrity check (https://github.com/Frenzie/myopera-backup/issues/10) and some pruning. These initial files are mostly to (hopefully) get someone interested in processing the available data in some way. To that end, I penned down a few suggestions (https://github.com/Frenzie/myopera-backup/wiki).
Some technical information: 3,331,270 items, totalling 4.7 GB.
ׂ
The compressed .tar.lz files each contain up to 100,000 individual files (as you would have seen had you checked the link :P). This will make for a total of probably about 200 files.
This may not be the final compression format. Like I said, this is just an initial seed sufficient for development of further processing scripts.
ׂ
I might take you up on that, but not until March. I'll probably share the vast majority of what I've collected using Wuala sooner than that.
Btw, I forgot to mention this earlier, but the 3.3 million actual posts represent the first 10 million comment IDs. After comment ID 3.2 million or so, only one in ten are actual comments.
I prepared the first 15 million comments* for sharing. The total size is 639.5MB. I won't upload them yet because I'm really close to the stupid bandwidth limit, and instead of something semi-reasonable like 1Mbit they drop you to an archaic 64kbit if you go over. Is saving the €7.50/month unlimited bandwidth worth it? Well, it does add up to a day trip or two so I'd like to think it does…
* Actually 3.2 + 1.2 million minus private and deleted comments.
ׂ
Sure, but they'd at least let me use up to 2TB or so… I doubt I'd come anywhere near that. ;)
I get capped at 10 megabits for being online too much. And still only during the evening and night.
Heck, that'd be a
lot better than 64kbps. At 50Mbit you can burn through a 100GB limit in minutes if you're not careful. Watch a few YouTube vids in 1080p (you don't even need anywhere near 50Mbit for
that) and you've reached it already. I'm hoping they'll at least up the limit to 150GB or 200GB or so before I'll crack and shell out the money for unlimited, because this is severely cutting into my downloading new Linux distros and the like.
I've started the upload. It'll probably be finished in an hour or so.
I've started the upload. It'll probably be finished in an hour or so.
Downloadable here? https://www.wuala.com/Frenzie/myopera-backup-data/
Is it intended to look like a bunch of tar files, not a single file?
It is for now.
FrenzieSTNG, I opened a different topic where I share my own backups of My Opera: https://dndsanctuary.eu/index.php?topic=69.0
Interesting!
"As most of you know, I'm currently running a Python script with the intent of backing up all of the My Opera forums, or at least about as close to it as possible."Do you backing up data (that's linked with a forum posts) from files.myopera.com/* and external images?
Would you mind sharing yours here or there?
Yes of course. But i didn't finished yet.
Interesting, how much stuff have Yahoo!, Google et al. cashed? I'm using a Yahoo! cashe for a blog (which has been partially deleted by the user), and it - for now - holds quite an outdated (however comprehensive) version of what that blog used to look like.
Do you backing up data (that's linked with a forum posts) from files.myopera.com/* and external images?
Unfortunately no. I haven't had the time to write such a thing. However, I think most of that should be available on the archive.org mirror of the forums. But mine will be (slightly) more complete.
Interesting, how much stuff have Yahoo!, Google et al. cashed? I'm using a Yahoo! cashe for a blog (which has been partially deleted by the user), and it - for now - holds quite an outdated (however comprehensive) version of what that blog used to look like.
In time those caches will be deleted, or at least become inaccessible from the outside.
Judging by Dredd the outdatedness, I feel like not to hurry.
The My Opera forums are being replaced by our new forums. Starting February 26th, the My Opera forums will be in read-only mode. On March 3rd, they will be removed along with the rest of My Opera.
This means I should be able to grab a complete My Opera archive after all.
Make it a news here, will you?
Party's tomorrow. Everybody's present.
Grabbing a complete archive will probably take at least two days.
Sorta backup:
I think it possible that within 25 years, we will trade our freedoms one by one to a "benevolent" government that promises to take care of us, feeding housing and medicating us. We will trade the ability to move about at will for the supposed security of having everything we bring examined and having our persons prodded and probed in every way imaginable. We will trade the ability to communicate freely for a bogus security that scans every email, text, phone call and letter, with PC (that's "Political Correctness" editors cleansing our missives of any "impure" words.
We will wake up one day wondering what happened to "The Land of the Free, and the Home of the Brave" as we bend our backs to the loads placed on us by our masters.
Unless something serious changes between now and then. But, looking at how willingly people trade freedom for security, I fear the wake-up may be too little too late.
Don't worry though. If you have nothing to hide you shouldn't be worried, right? Right????
http://my.opera.com/community/forums/findpost.pl?id=15160072
The last upload goes to 14,999,999 while the last post (up to some minutes ago) is 15,163,152. In other words probably less than 17,000 messages by the deadline of which many will be spam. (Posting was already slow, and after announcement has slowed further.)
Will the script be updated to the final call?
I've been running the script occasionally to grab some more comments since I shared the first "15" million comments. I'll share the remainder sometime after My Opera closes.
I'll probably put the remainder online by this weekend. Possibly earlier if I find the time.
I'll probably put the remainder online by this weekend. Possibly earlier if I find the time.
Why? what for? you never explained me why you do such an effort with such an useless thing.
It didn't take any more effort on my end to collect all posts than it would have to collect just mine. I'll be converting some of them to blogposts (http://fransdejonge.com/2014/02/data-uri-bookmarklet/) because they contain information I consider useful. Others are, like jax said, essentially link dumps. Besides which, I wanted to learn Python.
The last file is now online.
If you want to do some quoting from My Opera in these next two days, here's a little script that'll help:
You could save this as a .user.js in the appropriate folder:
var links = document.querySelectorAll('a[href^="findpost.pl"]');
for (var i = 0; i < links.length; i++) {
var comment_id = links[i].href.match(/\d+/);
var topic_id = location.href.match(/id=([0-9]+)/)[1];
links[i].outerHTML +=' <a href="http://my.opera.com/community/forums/reply.dml?action=quote&commentid='+comment_id+'&id='+topic_id+'">quote</a>';
}Or use it as a bookmarklet:
javascript:(function(){var%20links=document.querySelectorAll('a[href^=%22findpost.pl%22]');for(var%20i=0;i%20%3C%20links.length;i++){var%20comment_id=links[i].href.match(/\d+/);var%20topic_id=location.href.match(/id=([0-9]+)/)[1];links[i].outerHTML+='%20%3Ca%20href=%22http://my.opera.com/community/forums/reply.dml%3Faction=quote%26commentid='+comment_id+'%26id='+topic_id+'%22%3Equote%3C/a%3E';}})();
The last file is now online.
Very cool!!! 8) Thank you SO MUCH for doing this. Frenzie!!! :D
No problem. If you come up with something cool to do with the data, please share it here (or on Github.) I wrote down a few quick suggestions here (https://github.com/Frenzie/myopera-backup/wiki).
Did anyone back up the widgets? I forgot :(
There are three I used to like and use:
- Frenzie's weather widget
- Some other weather widget, a more recent development
- a calendar widget with holidays, reminders, and translations
- Frenzie's weather widget
That'll be tough to find 'cause I never made any of those! :)
Are you perhaps thinking of the weather Unite app made by lucideer as a proof of concept? BS-Harou also made a weather speed dial extension.
No, not unite app. Widget. Was there none uploaded by you? There was a blue weather widget that I for some reason associated with you.
I never made any widgets, or at least not publicly. Privately I'm sure I made one or two "hello world" types of widgets. :P 'cause yeah, they're basically just a type of web pages anyway.
Whoever made it, it was a cute widget. Any sanctuary or haven of Opera widgets anywhere?
johnnysaucepn (http://web.archive.org/web/20130308122158/http://widgets.opera.com/widget/3903/)?
If so, it looks like you can still get it here (http://macdownload.informer.com/touchthesky/). Also see here (http://web.archive.org/web/20100224035703/http://my.opera.com/johnnysaucepn/blog/2007/11/21/an-unbelievable-milestone).
Yup, that's the one. Thanks :)
And sorry for confusing the names. I confuse names in real life too. To me names seem totally random and undescriptive, always did. I tend to not use people's names in communication, because it's embarrassing to mix them up...
And I tried it immediately. I remembered I uninstalled Johnny's widget because it broke at some point. It can be installed, but doesn't work (should be fixable by redirecting to some other source for meteorological data). The other weather widget I liked, it still works http://macdownload.informer.com/multiweather/download/
Opera IRC is still up, btw. By mistake, I suppose.
And so is help.opera.com. Is it a legitimate idea to copy Opera's help pages to operawiki.info?
Is it a legitimate idea to copy Opera's help pages to operawiki.info?
Probably not. I'd just run wget on it with an Opera 12 user agent or some such.
The user agent is unnecessary, I think. The help pages were always accessible with any browser.
In an oddly familiar turn of events, Wuala will stop offering free space from December 31, 2014. Are there any other suggestions for sharing a few GB of data? I suppose I could always turn it into a torrent, but then availability as well as upload speed would be more limited.
If you don't know, then I doubt anyone else knows about such sharing place. What's the max limit of Dropbox? Does it do such sharing anymore?
But torrent always seemed a reasonable alternative sharing option. Publish it in Pirate Bay or such and some fans (such as myself) might remain seeding.
While I could easily put it on a shared folder in Dropbox (I seem to have 20 GB available these days), I don't want that propagating across all of my computers. Also I believe it's hard to download all files from a Dropbox folder the way you easily can in Wuala.
Anyway, I'm looking into setting up some kind of webseed at the moment to get the best of both worlds. You can give it a try using this torrent (http://myoperabackupseed.netii.net/myopera-backup.torrent).
You can give it a try using this torrent (http://myoperabackupseed.netii.net/myopera-backup.torrent).
So, I gave it a try this weekend. It has been stalling at 0.4%. It's precisely weekends when I can pull and share torrents...
Weird, what torrent client are you using? The webseed is supposed to kick in when there's a lack of peers.
qBittorrent
Hm, me too. Let's see, it downloaded 8 MB really quick from the webseed,* but then it picked up on some mysterious peer from Estonia and switched focus. That's as intended, but it slowed my download speed from a few hundred kB/s to significantly less. :P That being said, my qBittorrent is still pulling in 500 kB parts from the webseed every… I don't know, half minute or so. It downloaded about 8 MB in the first few seconds while over the past 10 minutes it only downloaded an additional 5 MB. My guess is either qBittorrent is trying not to overload the webseed or the webseed isn't fond of the fact that torrent clients pull in parts here and there at random instead of properly downloading a file from beginning to end. Either way, that doesn't explain why it's not working for you.
Right, so I've got qBittorrent 3.1.10 built against Qt 4.8.6, Libtorrent 0.16.17.0 and Boost 1.55.0.
* Free hosting, yet it might even be able to host this site. :insane:
I'm at 1.7 % now. Enough for this weekend. Will try more next weekend.
Yes, you picked some up from me after I got it from the webseed. :P
But I think there might be a bug in Libtorrent. Although I've downloaded 60 MB this session, 50 MB of it was apparently wasted(!), i.e., failed the consistency check. This obviously makes no sense because I'm sure that if I'd download the complete file using wget it'd be picked up just fine.
Here (http://code.google.com/p/libtorrent/issues/detail?id=463)'s a bugreport for what sounds like the same issue, and perhaps it offers the answer:
There is a known issue that if the web server doesn't support keep-alive, 16 kiB blocks that would require more than a single request won't work. libtorrent currently must request an entire block within the same connection. If the server closes the connection, the partial block that may have been downloaded is discarded (just like normal bittorrent peers).
The symptom you describe sound like what you would see with multi-file torrents and web server that don't support keep-alive.
I'm not sure how to check if the server supports keep-alive (probably something with wget or curl), but in any case I did put it on the server as many small files of a few MB.
After an abrupt restart my qBittorrent totally fails to launch. Even reinstalling and wiping out the profile didn't help. Other Qt programs such as Qpdfview and Otter are unaffected. I am bad at debugging, so I simply uninstalled the thing. I'm pulling the torrent now with Aria2 which never failed me.
So the webseed is working well with this Aria2? Interesting.
If you mean Aria2 continued where qBittorrent left off, the answer is no. Aria2 keeps an .aria2 file around as long as it's pulling, in order to know where to continue after an obstruction. When it doesn't find the file (and of course it didn't find it now because qBittorrent doesn't do .aria2 files), there's no other way but to start afresh.
No, I mean that if you're actually getting a download then that must mean the webseed is working. Unless anyone else downloaded it in the meantime and is seeding. :)
All downloaded now. Aria2 always was the best. I should not have begun trying those graphical torrent thingies.
https://forums.opera.com/discussion/comment/15198634#Comment_15198634
Is the torrent thingy still relevant? I hope you keep doors open for some odd geeky fans :)
I don't know about the longevity of the seed server, but in principle, sure. :P
Do you know of this storage place? https://files.secureserver.net/
Is it sufficient?
I couldn't say. Can you even sign up for the place?
Can you even sign up for the place?
Not planning to :left:
It's just that someone was hosting a half giga file there...