Streaming data to a webserver

[iamlucky13]

This is a rather complex one, I think.

The company I'm working for (which builds sensors) recently set up a sensor system as a proof-of-concept/test for a new type of sensor, and they told me they're interested in being able to use their website to show it in operation to potential customers. What they want is basically a single page that shows a display of data taken over short period of time and possibly a daily summary. Keep in mind the sensors are continuously taking real-time data.

Important considerations:
- Our site is hosted by a webhosting company located who knows where.
- The PC that receives data from the sensors has an internet connection.
- The data can be "pre-processed" before being sent to the webserver

If anyone has suggestions about how best to go about this or links to resources, I would love to hear them.

My first thought was to have the test computer periodically (~1 min) upload data summaries to a MySQL database. This could be used to approximate realtime data. One of my coworkers can write a program to automate transmitting the data. Unfortunately, they would prefer realtime, and I'm not sure this would convey to the customer what they really need. Are there considerations I should be aware of here beyond database size.

My second thought was perhaps there is some way of using the server as a proxy between the test PC and a user to actually stream data to the user and have it display in an applet. Is there a way to create a connection capable of doing this between the PC and the webserver?

Other ideas?

Whatever happens, it looks like I have some more learning cut out for myself

[ray326]

Something like your first idea will be your best bet. There's no particular reason the sensor data couldn't be transfered a lot more often than once per minute, though. Depending on your overall system, that process could probably run once a second. Then your problem becomes purging the database. 8-)

[iamlucky13]

In some ways, a database could definitely be simpler, but it seems like a clumsy way to handle this.

It occurred to me after I posted, chat programs must work in a way not too dissimilar from what I'm looking at. Two ordinary PC's use a webserver to mediate real-time communication. Does anyone have any links with information about the workings of online chat?

Thanks

[AdamGundry]

I think you're on the right lines with a chat-like system. If you want this to be pseudo-realtime, the best option is probably to run a basic server on the test computer which accepts client connections and gives them the data directly, rather than sending it to the web server as an intermediate step. Your web server would then provide for download a Java applet (preferably) or an ActiveX control (not a good idea) which connects directly to the test machine and displays the data. This is similar to how chat servers work - there is a seperate process to the web server handling chat data - though obviously in this case the machine it is running on is different.

Adam

[ray326]

The applet will have to be signed and the user would have to allow it some non-standard permissions for it to make a connection directly to the test PC.

[iamlucky13]

Thanks both of you for your suggestions.

Originally posted by ray326
The applet will have to be signed and the user would have to allow it some non-standard permissions for it to make a connection directly to the test PC.

Ray, I'm not as much concerned about convincing the potential customer to click through a warning or two at the moment as I am at the technical side, but your point is noted.

As for the test computer accepting client connections, don't some ISP's restrict that in addition to normal OS settings? The sensor system is actually somewhat mobile, so the ISP account might not always be the same, and this sort of web-monitoring could potentially become a feature of the system, so portability is desireable.

By the way, at the moment there is no urgency. The idea was suggested, but mainly I want to be prepared to explain what this would entail and what the restrictions would be if they decide to go ahead with it.

[CardboardHammer]

How would the endusers be able to tell the difference between realtime and simulated realtime? Go with the once per minute batch sends and run things a few minutes behind on the webserver so that it behaves as though it's realtime (the data retrieved by potential customers flows over time instead of appearing in bursts, even though it's really delivered in bursts). Who'll know the difference?

[ray326]

I still believe your most workable solution is to have the test PC "push" its data to the web server for consumption by the browsers running demos. One reason I like that is it fits a classic "message queue" sort of "publish/subscribe" pattern. The fact that the test PC is using DHCP just strengthens my conviction on that.

[AdamGundry]

It's possible the ISP would block incoming servers, but somewhat unlikely, particularly if you run it on a nonstandard port. The bigger problem is likely to be DHCP, as Ray mentioned - if the data server's IP address is changing you will need to provide the current one to clients either via the web server or using a dynamic DNS facilitiy. Neither would be that complex, but storing the data on the web server is a possible option, though you would have to send it in fairly large blocks to justify the overhead of using HTTP.

Adam