Database options for (very) large databases

[blue-eye-labs]

I was just wondering if anyone could recommend a server platform for use with very large databases. I'm currently rather well versed with MySQL but if anyone knows of any problems with it's performance when it comes to large database sizes then could they let me know? I seem to remember that both Google and Youtube use MySQL don't they?

[tattooedscorpdc]

I would look at the SQL Server 2008 clustered on a Windows HPC server.

Hope this helps,

Larry Darrah
MS Architect Evangelist

[blue-eye-labs]

Thanks. I'd rather not use anything Windows based to be honest, I have a strong aversion to anything to do with M$.
What about MySQL's handling of a table of say, 10'000 rows? Anyone?

[Charles]

Oracle seems to work well and work well with big chunks of data. And SQL Server does have its limitations size-wise.

[svidgen]

MySQL should perform quite nicely. It's the DBMS of choice for most web apps, big and small. And most of the caveats that apply to MySQL will apply to any other DBMS as well. I recommend taking a look at High Performance MySQL 2nd ed. (http://oreilly.com/catalog/9780596101718/?CMP=AFC-ak_book&ATT=High+Performance+MySQL,+Second+Edition,).

[chazzy]

define "large database" how many concurrent users? how large is the data? how many schemas?

[blue-eye-labs]

Thanks for the replies, I will definitely take a look at that book, I'll either order a copy or cycle down to my library. As far as the database size goes, it'll be about 10'000 rows, as far as data size, it'll mostly be quite small, probably about five or six columns, with mostly short text (categories, names), there might be one big column of text, but even so that cell won't contain more than about 300 words.
I'm not sure about concurrent users. It shouldn't be more than say, 500 but that could be a severe underestimate. At this stage I'm not sure.

[russell]

That is nowhere near what I'd call large. MySQL will handle it fine

[blue-eye-labs]

Initially I thought it'd be bigger, so I decided to cover my bases, as it were.
Since I am going to use MySQL, what can I use to search it? I ask this because I'd like to use Sphinx or Lucene (preferably Sphinx) but both use persistent processes (which I gather a lot of hosts dislike). I could use MySQL's built in full-text search but it isn't as quick and efficient, or so I've been led to believe...
Any advice?

[svidgen]

If you get a VPS or dedicated server you don't have to worry about what your hosting provider thinks about running additional daemons. Sphinx is supposed to do a really great job with fulltext indexing. And it's flexible too. I'd grab myself a VPS (at least) and use that in conjunction with MySQL.

Though for 10,000 rows (or even 100,000 or 1,000,000), I think MySQL's fulltext search would do fine. I think it's mostly a question of flexibility in search features at the scale you're dealing with.

[blue-eye-labs]

This is true, I could write my own search functions and add functionality to the MySQL full-text searching but I think it'd get slow if I did that and someone's already done it, in the form of Sphinx, so I'm tempted to just go with that anyway. Thanks for the advice. I think I'll get a VPS, I dedicated server is unnecessary at this point.