Downloadable archive of ba posts?

I was wondering if it would be possible to provide a periodic downloadable archive of ba posts? Nothing fancy, just the post text (i.e. no attachments/images if that would be easier)… something along the lines of the content available at https://blenderartists.org/forum/archive/index.php This would be helpful for two situations: 1) offline access and 2) avoiding the search frequency timeout (going through Google to avoid it hasn’t been a great solution for me… sometimes their index/ranking is just messed up)

Sure, it’s possible… but it might not be feasible. BA has a lot of posts… a lot. A separate downloadable archive would be a pretty large download… and it would only grow over time.

Technically you can do it yourself from that archive page you linked, using a tool like wget… but be prepared for a lengthy wait. It might be better to see if there’s another more effective way to address the two situations you’re trying to address.

If ba is open to it, it’s quite feasible. Currently some much larger sites offer database dumps of their sites (i.e. wikipedia-en - 14 GB, stack exchange - 40 GB… those are the compressed sizes.) Archive.org appears open to hosting such content so the bandwidth burden isn’t an issue and usually it’s just a matter of making some subset of an already existing backup public. Sure, they take a while to download… but it’s a manageable amount of data.

I can definitely crawl the archive pages but sites generally frown on that and then there’s the issue of having to re-crawl the site periodically which sites usually really don’t like. But if you’re saying that’s the way to do it, I am willing and able to do so…

I love the Blender Artists forums and rely very heavily on them as I learn more about Blender. When I’m stuck on a Blender problem, I search through the forums first and almost always find an answer.

Unfortunately, I don’t always have an internet connection when I’m using Blender. There are many times where I really wanted to have an offline copy of the forums for searching. I know that there are web scraping utilities which would allow me to get a snapshot of the entire site, but I don’t want to abuse the forum’s bandwidth or get my IP address banned by doing this. (Of course, if you’re OK with users using a web scraping utility on the forum, just say so and I’ll use it!)

So I was wondering…would the owner(s) of Blender Artists consider selling a downloadable, searchable archive of all of the forum posts? What I’m looking for – and would be willing to pay for – is basically a zipped directory with thousands of html files + images + attachments of the Blender Artists forums for offline browsing.

For example, a new archive would be made available once a month. Whoever wants to download the current snapshot could pay, say, $10 USD for a one time archive download. Or maybe access would be given as a perk to Blender Cookie users.

Would this be something that Blender Artists members would find useful? I can’t be the only person who is regularly stuck without an internet connection and needs to look up some sort of obscure Blender functionality.

:eek::rolleyes:

Either prepare the material that boggles your mind…
ie.
Using IrfanView:
Run, press C > 5 (Custom region capture) & take a snapshot. Save.
Also, archive the forum’s post link and embed source info into the properties of an image.

… or Download the whole site yourself for offline browsing (many tools/apps are available).

Instead of paying/forcing others to do service for you, that yourself can do in an instant, donate :slight_smile:

MODERATION
Threads merged

Since it appeared through this thread that it wasn’t quite frowned upon to use a web scraper, I grabbed the forum sections that I’ll find most useful for offline browsing. Already I’ve found this to be massively useful.

Does the Blender Artists forum have a “Donate” link? I feel like I ought to give back something.

Haha… not that I’m aware of. Though I will say that, outside of the moderation team, the forum is managed by CG Cookie. I’m sure if you contacted them, something could be figured out.