I wonder if someone might have some prior experience to the following issue.
Any advice would be greatly appreciated.
I am constructing a very large web app that will house a lot data.
The data is pulled in from partner services via RPC.
To streamline the data, it is automaticly put into its relevant database table enabling more accurate searches.
My concern is for the future as the data increases.
My current thought process is to have a server acting a controller routing requests through to a different server holding the requested data. Something like a 1 to many relationship.
As data is added to the database its put (in a relevant way) in to a server on a rotational basis.
This is what i am leaning towards.
Data is brought in and processed over time in to server 1.
Then it is replicated on server 2 for usage.
This doesnt make much sense to me, but could be the caffeine.
I would really appreciate any ideas, debates or approaches that others have used.