RASCAL Data Streaming Backend -- Which Technology to use?

[Peter Rowley]

Hi David and everyone,

Following David's suggestion, I've converted an e-mail I sent to him into 
one for fluid-work, so everyone can get in on the discussion.

Based on some discussions within CNS at York, I'd suggest looking more 
closely at using HTTPS, i.e. HTTP over SSL.  HTTPS...

- due to the possibility that the screen recordings contain secure data 
and thus the need to transmit them securely to the server, 
- due to the fact that it encodes credentials used for authorization to 
deposit recordings on the server, and 
- due to its acceptability to sysadmins from a security perspective 
(stemming from the nature of the protocol and greater level of comfort 
with the security of Apache than a typical FTP server), 
- due to its ability to pass through most firewalls,
- due to it being built into Java SE, so licensing is not an issue

As for the scalability, HTTP upload is certainly used for massive sites 
such as Flickr.  Your approach to uploading the file in smaller (e.g. 1MB) 
chunks is probably a good one to continue; you'll need to build (or find) 
a simple protocol above HTTPS to keep track of which chunks still have to 
be transmitted.

As for alternatives, JDBC and JMS are overly complex for this, and SFTP 
has firewall issues in some situations (Google "sftp firewall"). 

Peter




David Makalsky <dmakalsky at gmail.com> 
Sent by: fluid-work-bounces at fluidproject.org
10/03/08 05:57 PM

To
Fluid Mailing List <fluid-work at fluidproject.org>
cc

Subject
RASCAL Data Streaming Backend -- Which Technology to use?






Hi,

We have a functional requirement in RASCAL for data to be streamed 
from the clients' browser (applet) to the server.  This is the data 
that will be converted into the video of the users' desktop (and audio 
from the microphone).  This functionality is essential, since without 
it, the data wouldn't get to the server.

Currently, this solution is implemented via ftp.  An ftp connection is 
established from the client to the server and the data is uploaded in 
a separate thread in 1 MB chunks.  We chose this solution for various 
reasons (discussion in the table below) but would like to open this up 
to the fluid community.  Which of the proposed solutions below would 
you suggest, or are there other completely different solutions that we 
should consider?

Regards,

David Makalsky

FTP
===

Pros:
- Already implemented
- Many java libraries available (license compatible)
- Only virtual user needs to be created with only write access granted
- Simple to implement

Cons:
- Password is sent as plaintext
- There is a maximum simultaneous connection limit
- Not transactional

SFTP
====

Pros:
- Password is encrypted
- Simple to implement (if you have a good library)

Cons:
- No License Compatible java libraries
- Same cons as FTP except password is encrypted

Message Queue
=============

Pros:
- Transactional
- Secure
- Easy to implement on the client side
- Scalable

Cons:
- Significant administration and installation effort
- Resource-intensive

JDBC
====

Pros:
- Scalable
- Secure
- Easy to implement on both sides
- Transactional

Cons:
- Database will grow quite big unless blobs are written out to files 
regularly and cleaned from database


HTTP
====

Pros:
- transactional
- fairly easy to implement

Cons:
- Bad for streaming large chunks of data
- Puts a large strain on http server
- not scalable easily (will have to cluster web servers with session 
affinity)
_______________________________________________
fluid-work mailing list
fluid-work at fluidproject.org
http://fluidproject.org/mailman/listinfo/fluid-work

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20081006/27697a96/attachment.htm>