RASCAL Data Streaming Backend -- Which Technology to use?

[Colin Clark]

Hi David,

Thanks for outlining the issue so clearly. It's been a few years since  
I've had to handle large file uploads on the server, but here is some  
general advice. I'm sure others on the list will have more detailed  
suggestions.

Sending passwords in the clear is a big drag, and most sysadmins will  
chafe at the prospect of using it. An FTP-style approach also  
encourages the likelihood that your application code will get  
fractured into lots of hard-to-manage pieces including cron jobs,  
shell scripts, etc. Building a web app to receive your videos, encode  
them, and organize them is a more straightforward approach.

I think that you'll find streaming large binaries from a Java applet  
to the server via JDBC is also not particularly scalable for a number  
of reasons. From my experience building imaging systems awhile back, I  
found that storing large BLOBs in the database is slow and immensely  
awkward. Also, a two-tier architecture like this tends to be more  
brittle because it requires a closer coupling between your applet and  
database, without the benefit of a mediating data service layer.

I think you'll find that streaming your videos via HTTP will be more  
effective and straightforward. You can build your file upload process  
so that it is essentially stateless, avoiding the worry about session  
replication. It's easy, its straightforward, and it will allow you to  
consolidate the upload, processing, and filing workflow into a single  
application. I think you'll find that your options for scaling with  
standard HTTP are far greater as well.

I hope this helps,

Colin

On 3-Oct-08, at 5:52 PM, David Makalsky wrote:

> Hi,
>
> We have a functional requirement in RASCAL for data to be streamed
> from the clients' browser (applet) to the server.  This is the data
> that will be converted into the video of the users' desktop (and audio
> from the microphone).  This functionality is essential, since without
> it, the data wouldn't get to the server.
>
> Currently, this solution is implemented via ftp.  An ftp connection is
> established from the client to the server and the data is uploaded in
> a separate thread in 1 MB chunks.  We chose this solution for various
> reasons (discussion in the table below) but would like to open this up
> to the fluid community.  Which of the proposed solutions below would
> you suggest, or are there other completely different solutions that we
> should consider?
>
> Regards,
>
> David Makalsky
>
> FTP
> ===
>
> Pros:
> - Already implemented
> - Many java libraries available (license compatible)
> - Only virtual user needs to be created with only write access granted
> - Simple to implement
>
> Cons:
> - Password is sent as plaintext
> - There is a maximum simultaneous connection limit
> - Not transactional
>
> SFTP
> ====
>
> Pros:
> - Password is encrypted
> - Simple to implement (if you have a good library)
>
> Cons:
> - No License Compatible java libraries
> - Same cons as FTP except password is encrypted
>
> Message Queue
> =============
>
> Pros:
> - Transactional
> - Secure
> - Easy to implement on the client side
> - Scalable
>
> Cons:
> - Significant administration and installation effort
> - Resource-intensive
>
> JDBC
> ====
>
> Pros:
> - Scalable
> - Secure
> - Easy to implement on both sides
> - Transactional
>
> Cons:
> - Database will grow quite big unless blobs are written out to files
> regularly and cleaned from database
>
>
> HTTP
> ====
>
> Pros:
> - transactional
> - fairly easy to implement
>
> Cons:
> - Bad for streaming large chunks of data
> - Puts a large strain on http server
> - not scalable easily (will have to cluster web servers with session
> affinity)
> _______________________________________________
> fluid-work mailing list
> fluid-work at fluidproject.org
> http://fluidproject.org/mailman/listinfo/fluid-work

---
Colin Clark
Technical Lead, Fluid Project
Adaptive Technology Resource Centre, University of Toronto
http://fluidproject.org