It’s been 2 months since I’ve posted about Zencoder. You may be glad to know that the project is still alive and is nearing completion. It has moved a little slower than we initially expected (we had hoped to finish in Nov/Dec), but the end is in sight. So if you are interested in a complete video transcoding solution for your web application, request more info at Zencoder.tv. And if you’ve inquired in the past, and you are interested in a demo, let us know: we’re starting our online demos this week.

But on to the sneak peak.

Scaling

Zencoder scales both horizontally and vertically to a nearly arbitrary limit. You can scale vertically (obviously) by increasing the speed of your machines: the faster the CPUs, the faster the transcoding, and any Linux or OS X machine should work. And you can scale horizontally by increasing your number of servers, as high as your licensing allows.

You can scale the same way on EC2, by moving from small instances to large instances, or by increasing your number of instances.

What’s important is that Zencoder works the same whether you run it with 2 servers or with 200 (or more). The system will happily distribute your jobs across however many servers you have. (Note that it will do this by distributing each job to a different server, not one job across many servers – the latter approach can finish each individual job more quickly, but the Zencoder approach should have a somewhat higher overall throughput, since it is a bit more efficient. And it is significantly simpler, of course.)

This means that Zencoder can grow with your application. If you underestimate your traffic, or your site grows exponentially, just add another server or fire up a new EC2 instance.

Finally, you don’t even need to restart Zencoder to add additional servers, so expansion is almost completely seamless. If you’re running on dedicated hardware, just configure Zencoder on a new server and start it up: it will begin working immediately. On EC2, things are even easier: click the big green “Launch new worker” button, and you’ll have a new server up and running in about 3 minutes.

Reliability

On the reliability front, Zencoder is built with three goals in mind.

First, errors should be few and far between. This goal is obvious, and is shared with most software (except for some software coming out of Redmond, WA). And Zencoder is a reliable system in this respect. The code is reasonably compact, and it is well tested (both by unit tests and by humans).

Second, if any (or all) pieces become unavailable, your website should continue to work. In other words, if a user uploads a video at your site, and Zencoder is unavailable for some reason, your website should continue functioning as-is. Similarly, if your site is down, and Zencoder finishes a job, your website should receive the job as soon as it comes back up. As a result, we have designed the system such that you can pull the plug on any piece – or multiple pieces – and as soon as everything is back online, the system picks up right where it left off.

Third, no job should ever get lost in the system. This is accomplished through integrity checks – between the queue and each transcoding node, and between the Zencoder system and your application. You can even wipe the Zencoder queue, and if your application is still waiting on any jobs, they will be automatically recreated.

The result is that Zencoder is almost a self-healing system. It isn’t perfect (yet), but it is robust, reliable, and scalable.

MinneDemo is a quarterly event in the Twin Cities where local startups and technologists demo their products.

Zencoder is a distributed video processing system, built by Slantwise, that is scalable, reliable, and flexible.

The next MinneDemo event will be held December 6th at O’Gara’s Garage in St. Paul, and I’ll be presenting Zencoder there. This will be the first public showing of Zencoder, and I’m excited to be demoing the system after four months of development.

Whether or not you’re interested in video processing, if you’re reading this blog and you live within a 100 mile radius (or even 270), you should come. MinneDemo and MinneBar are the most worthwhile tech events going on in Minnesota.

MinneDemo works like this.

• Show up at 6:30 for a free beer/wine/soda or two.

• Meet interesting people: developers, designers, venture capitalists, tech lawyers, startup gurus, information architects, enterprise software providers, ISPs, and more. About 300 people typically show up.

• Have another beer/wine/soda.

• Watch five or six software demos, including Zencoder. The demos each last 15 minutes or less, and PowerPoint is not allowed, so you actually see working software, not marketing-speak.

• Continue to meet people, get ideas, land projects, and find partners. We met our attorneys, one business partner, two clients, and several friends at MinneDemo and MinneBar.

The event starts at 6:30pm on Thursday, December 6th (drinks and food), and the demos will start at 7:30pm. The demos will finish around 9pm, but some people will stick around for hours after that.

Many thanks to Dan Grigsby and our own Luke Francl for organizing MinneDemo, and Ben Edwards for starting MinneBar!

So how does Zencoder do its transcoding? What codecs and formats does it support? And what about the critical issue of codec licensing?

Licensing

Let’s start with the question of licensing. Codecs are licensed in three main ways.

1. Commercial software (On2, Nellymoser)
2. Patents and standards requiring licenses (MPEG-2, AAC, MP3)
3. Free or unenforced formats and codecs (Ogg, Matroska, AVI)

If you’re going to transcode and distribute video, category 3 is easy. Category 1 is straightforward, and these commercial codecs work just fine with Zencoder.

Category 2 poses the real problem. In my estimation, three of the most important video codecs – MPEG-4, H.264, and MPEG-2 – and three of the most important audio codecs – AAC, MP3, and AMR – all require licensing direct from their patent holders and/or licensing authorities. Those six invaluable codecs are represented by four separate licensing bodies (representing hundreds of patent holders), each of which has separate terms, prices, minimum fees, etc. And when you move to the second tier of codecs, things get even more complicated.

Zencoder takes care of this for you. With Zencoder, you receive licenses to these and other critical codecs. We’re still working out the specifics, but when Zencoder launches, we plan on including licenses to 10-20 patented codecs and formats, covering all of the major ones and several in the second or third tier.

Compatibility

Zencoder can handle every major video codec, audio codec, and container format. And we do this legally, taking care of most licensing issues for you.

Here is a partial list of formats and codecs that we support.

• H.264
• FLV
• MP4
• 3GP
• VP6 (with commercial license)
• MP3
• AAC
• MPEG-4 Video, XviD
• MPEG-2
• AVI
• Ogg, Theora, Vorbis
• Nellymoser (with commercial license)
• And 100+ more

The goal is to decode everything that you can throw at Zencoder, and to encode everything that you have a good reason to encode.

Slantwise has been working on a video processing product called Zencoder for the last several months. Zencoder is a distributed multimedia processing system, written in Ruby, that is powerful, flexible, and scalable. It also integrates easily with any application that needs video transcoding. The product itself is in its final stages of testing, and will launch soon, so head over to http://zencoder.tv and fill out the contact form if you want more info.

In this first “sneak peek” post, we’ll look at Zencoder from a high level. Where does it run? How does it integrate with your application?

Two Hosting Options

Zencoder is a server, or collection of servers, that you host. The first server is the “mind” of Zencoder, which queues jobs and handles all communication with your application. The other servers are “workers” that do the actual video transcoding.

You can host Zencoder on your dedicated hardware or on Amazon EC2. Both work equally well, but we think that the EC2 approach is especially interesting, and Zencoder does some cool things with EC2. (More on that in a few weeks.) But you can mix-and-match EC2 instances and dedicated servers, so you don’t have to choose. You could use dedicated hardware for most of the transcoding work, and use EC2 for overflow work. Or use EC2 for most of your work, but one ultra-fast dedicated machine for high-priority jobs.

Easy Integration

Zencoder integrates via REST web services, so it is completely technology-agnostic. Your application can be written in Java, Ruby, PHP, .NET, Python, Erlang, Ada, or Assembly, as long as it can communicate via HTTP. It can be a web application, server software, or anything else that can send and receive HTTP requests.

To submit a job to Zencoder, you just need to send a POST request to your Zencoder server with the basic details of the job: input file, job id, and task/recipe. Zencoder will then do its work and send a request back to your application with the results of the job: status, output file attributes, output file location.

So your application only needs to do two things to integrate with Zencoder:

• Send a HTTP request to your Zencoder server when you want to trigger a new transcoding job
• Handle a HTTP request from Zencoder with the results of a job

We also provide a Ruby on Rails plugin that handles these things for you; so if you’re running Rails, this work is done for you. You just have to install the plugin and configure a few things, and your application is ready to go. Eventually, we plan on providing similar integration classes for Java and PHP. But even if you aren’t using one of these technologies, the integration work is pretty simple on your end, and we’ll help you through it.