?

Log in

Wed, Feb. 14th, 2007, 11:29 pm
qaexl: Amazon S3 and Rails

Amazon S3 (Simple Storage Service) is a pretty simple idea: use Amazon's infrastructure to store data at a very low cost. At a large scale, low cost means doing away with building your own file servers, datacenter, provisioning bandwidth, messing with Raid5-whatever-capable harddrives. And service techs. Meaning that it is unlikely to ever go down, and bandwidth spikes are simply ... handled.

Amazon has also provided a Ruby API for S3 at http://amazon.rubyforge.org

It won't be long before someone writes a mod to file_column or other similar Rails plugins that can store to S3 instead of the local harddrive.

A couple months ago, I had the chance to talk to someone who was creating a Rails application involving serving up images. He was writing up a proposal for some investors he is talking to. He needed something that would scale with volume. The solution at the time was to hash the filenames of the photos and distribute them across multiple servers. Of course, had I known about S3, the solution would be simpler: on uploads, dump it into an S3 bucket, then call the "url_for" method to get back an url and store that into the local database.

For more information: http://www.amazon.com/gp/browse.html?node=16427261

Enjoy.
-Qaexl

Thu, Feb. 15th, 2007 06:11 am (UTC)
leftisto

You're absolutely right, file system storage is for fossils.

Luckily, the plugin you're wishing for is already almost ready for prime time. We've used it in 3-4 production projects:

http://svn.techno-weenie.net/projects/plugins/attachment_fu/

Attachment-fu is Rick Olsen's follow-up to acts_as_attachment. It's a full re-write that sports pluggable image processors and back-end storage.

So, instead of acts_as_attachment :storage => :file_system you can now look forward to:

has_attachment :storage => :s3,
:processor => :rmagick,
:content_type => :image,
:size => 1..5.megabytes,
:thumbnails => { :thumbnail => '100',
:small => '250',
:medium => '350>' }

Right now a_f also support image_science, a non-image/graphicsmagick based library that doesn't bleed ram with every operation.

Thu, Feb. 15th, 2007 06:37 am (UTC)
qaexl

Cool.

I heard people talk about technoweenie's attachment_fu on #rubyonrails, though I haven't used it yet. Is the s3 storage engine available in the HEAD revision right now?

-Q

Thu, Feb. 15th, 2007 06:59 am (UTC)
leftisto

Yes - the trunk is reasonably stable.

Thu, Feb. 15th, 2007 07:32 am (UTC)
qaexl

Cool, thanks for the tip. Now for some coding.

-Q

Thu, Feb. 15th, 2007 09:56 am (UTC)
vvlad

I've read that Amazon S3 had some problems and author of following post thinkgs that S3 is not relible enough for his tasks
http://bob.pythonmac.org/archives/2006/12/06/cachefly-vs-amazon-s3/
Any way I'm sure that S3 is brilliant solution - technically and in many other ways...
But there are so much marketing buzz about S3 and usually no cons were mentioned

Thu, Feb. 15th, 2007 10:49 am (UTC)
qaexl

I read up on some of them after I posted this up. There was also a big outage at the beginning of Jan.

It doesn't change the original use I have for it (remote backup), but using it as a primary store for static files is still up in the air. I don't think latency is as big of an issue because I hand intended to have direct links to the static files.

-Q