[uday]

It is no longer unusual to see 5000+ users online at the same time. We sometimes see 6000, and the trends indicate that growth is continuing.

This raises some unpleasant issues for the bbo code. We're approaching limits of various sorts, and I need to find a way to allow for this growth.

One alternative that I have is to split BBO into multiple independent sites.

From your perspective, the login screen would be followed by a screen that asked you to choose from a few (or more) BBOs. Once you selected a BBO, things would be much as they are today -- but your BBO would presumably be smaller, with fewer players, tables, tourneys, etc. Each BBO would be almost completely independent of the others.

Each of these mini-BBOs would be completely independent of the others, and could have its own yellows, rules, official language, whatever.


Okay. This would be easy to do from a programming perspective.

Do you think this is a good idea? Do you think that many smaller BBOs will be easy to deal with? Do you think you'll settle into one particular bbo with most of your friends?

How do you think we should break up the BBOs ? By language? Country? Affiliation (ie, acbl)? By skill levels ? By niceness-levels? By nothing at all (BBO1,BBO2,etc)?

Should one or more of these mini-BBOs be selective about who can enter? Should we allow interested parties to run their own BBOs ?

Will splitting up the BBOs make the overall experience worse because of the "Network effect" that Hrothgar (Richard) mentions in another post? Will (say) a mini-bbo that is limited to 4000 users be large enough to allow you to find games and make friends?


Lots of questions. You can assume that we'll find a way for Vugraphs to be available to all mini-bbos. You can't assume the same about tournaments or friends, although I think we'll have to make sure you can get a list of your friends on other BBOs from somewhere in the program. Perhaps also a list of tourneys.

[inquiry]

I would hope any move to split BBO into multiple BBO's would be a temporary fix to the growth problem. With the idea to fix it so there could be just on large BBO when the magnitude problem solves itself.

Richard is often wrong on many issue (that is when he disagrees with me), but he is right on with the network effect issues. Large number of players is good, small numbers is bad. IF BBO1 had 3000 players online and BBO2 had 3500, there wouldn't be much difference than what BBO has been for years, a fine place. The problem is the tipping that will occur when the good players decide they like BBO2 better. So the stars go over there. Then others follow. Soon you are back to the problem if BBO1 doesn't ahve enough people to substain the player pool.

Maybe you could divide BBO into TEAMS/Tourneys and Main room. So all players playing in a tourney or team games are logged into BBO2, while all main room an private club people are in BBO1. Kibitizers can migrate wot which ever one they want.

Anyway, if I had a choice, I would vote for one huge BBO but practical issues have to take priority.

Ben

[Brandal]

uday, on Mar 18 2005, 11:16 AM, said:

It is no longer unusual to see 5000+ users online at the same time. We sometimes see 6000, and the trends indicate that growth is continuing.

This raises some unpleasant issues for the bbo code. We're approaching limits of various sorts, and I need to find a way to allow for this growth.

One alternative that I have is to split BBO into multiple independent sites.

From your perspective, the login screen would be followed by a screen that asked you to choose from a few (or more) BBOs. Once you selected a BBO, things would be much as they are today -- but your BBO would presumably be smaller, with fewer players, tables, tourneys, etc. Each BBO would be almost completely independent of the others.

Each of these mini-BBOs would be completely independent of the others, and could have its own yellows, rules, official language, whatever.


Okay. This would be easy to do from a programming perspective.

Do you think this is a good idea? Do you think that many smaller BBOs will be easy to deal with? Do you think you'll settle into one particular bbo with most of your friends?

How do you think we should break up the BBOs ? By language? Country? Affiliation (ie, acbl)? By skill levels ? By niceness-levels? By nothing at all (BBO1,BBO2,etc)?

Should one or more of these mini-BBOs be selective about who can enter? Should we allow interested parties to run their own BBOs ?

Will splitting up the BBOs make the overall experience worse because of the "Network effect" that Hrothgar (Richard) mentions in another post? Will (say) a mini-bbo that is limited to 4000 users be large enough to allow you to find games and make friends?


Lots of questions. You can assume that we'll find a way for Vugraphs to be available to all mini-bbos. You can't assume the same about tournaments or friends, although I think we'll have to make sure you can get a list of your friends on other BBOs from somewhere in the program. Perhaps also a list of tourneys.

Separating MBC from the rest would probably be most
effective,if you had "everything" on each mini-BBO it
would be likely to have lots of people in one mini-BBO
and alot less in the other(s)?

[hrothgar]

uday, on Mar 18 2005, 07:16 PM, said:

This raises some unpleasant issues for the bbo code. We're approaching limits of various sorts, and I need to find a way to allow for this growth.

One alternative that I have is to split BBO into multiple independent sites.

Hi Uday

Hard to make any concrete recommendations without a better understanding regaridng the code base. With this said and done, you seem to be floating a "trial ballon" regarding splintering BBO into a series of separate parallel sites.

Its unclear to me whether such a drastic solution is necessary. As I noted earlier, network effects are very powerful. I think that it would be a mistake parallelize the site based on scaling issues. (Please note, I don't rule out parallelization, however, I believe that this should be based on "social" concerns such language groupings or regulatory structures)

From my own perspective, most of the issues that you are dealing with are relatively easy. For example, consider all of the rigamorole associated with serving tables and the like. All of this could be completely parallelized while remaining completely transparent to end users.

With this said and done, you do face one enormous challenge which is properly architecting the communications systems. The communications structure is the linchpin upon which the entire site hinges.

If it were me, I'd start with a needs based analysis regardng the communication system. How and why are customers currently using the communications system? Equally significant, are there any ways in which this can be re-designed to handle the scaling issues. Case in point: In a couple earlier threads I suggested that BBO needed better ways to match players with potential partners and opponents. Adding these types of features could decrease the load on the main communications grid. Equally significant I also sugested that the current broadcast components could and should be complemented with a series of channels. Individuals would have the option to subscribe to sets of channels that they found particularly interesting. Channels serve two key purposes:

1. Channels limit the number of end user who will recieve information, decreasing the amount of system resources required.

2. Here, once again, channels can be parallelized without impact the user experience.

I suspect that if you can get the communications sytem "right" you find that a relatively simple architecture will fall in palce relatively quickly...

[cherdano]

uday, on Mar 18 2005, 04:16 PM, said:

Do you think this is a good idea?  Do you think that many smaller BBOs will be easy to deal with? Do you think you'll settle into one particular bbo with most of your friends?

How do you think we should break up the BBOs ?  By language? Country? Affiliation (ie, acbl)?  By skill levels ?  By niceness-levels?  By nothing at all (BBO1,BBO2,etc)?

Don't. I would take a huge bet, that if you tried to split up BBO according to any of these criteria, you would get loads of complaints, this forum would see flamewars (respectively busy moderators) etc. etc.

I think you cannot force a grown community like BBO to split without making some people unhappy. What other game servers try is to offer different rooms, so that chat, browsing player lists etc. can only happen within one room. And try to make users build smaller communities within which they are happy. Splitting users into different rooms is done by
a) letting the default room depend on parameters (language, level, etc)
make it unnecessary to be in the same room as other people to start or watch a game
c) limit the room size as a last resort.

Of course, the problem is quadratic scaling of network traffic etc. But in my opinion, most of the quadratic bottlenecks that I can see from here are unnecessary.
1. The list of all people logged in is too big to be useful. I would be happy just with the list of my friends, list of yellows, plus a way to contact the partner with whom i just signed up for a tourney via the partnership desk. Similar with the list of other kibitzers at my table/vuegraph room, as soon as it gets bigger than 30.
2. Lobby chat is a disaster anyway, so you could abandon lobby chat, and only allow tournament chat, team match room chat, MBC chat etc, and try to further subdivide this when it becomes necessary.
3. Similar room list MBC -- I only need tables with vacancies, tables with starts, and tables with my friends.

Of course, this is all much more programming work than offering different servers. But maybe not so much more than splitting in different servers, plus adding a mechanism to tell me which of my friends have joined Mini-BBO2, etc.

[DrTodd13]

I absolutely hate the idea of multiple mini-BBOs.

I know Fred has an affinity for it but I think you are going to have to abandon the idea of clients always being in synch. I agree with a previous poster that the huge player list is not really useful. All I want to see are my friends and stars. Perhaps at login, no players are displayed by default and the client sends a list of people they want to be displayed. This list could include specific names, stars, certain countries or wildcards (e.g., everyone). I suspect most would just do friends. Also, same thing with tables or tournaments. I don't care about full tables unless my friends are there or stars are there. Any solution where everybody on BBO has to be notified that 50 new tables were just created during a round change in some tournament that they aren't interested in seems like a fundamentally unscalable solution.

I also agree that I would need to know where the bottlenecks are in the system before I could help. Is it network bandwidth, CPU usage, or bus bandwidth that is the problem? A lot of techniques to reduce network usage could increase CPU usage or bus usage so we really need this information.

At this point, I trust Uday that he has done all the quick fixes he can think of while still maintaining the overall structure of BBO. So, I think we are in for either 1) restructuring of BBO (e.g., multiple mini-BBOs or abandoning clients being in synch) or 2) complex code changes. I'll reiterate what I've said in the past. Processing the bidding and play of the hand at each table is a function that could be performed by the computer of one of the people at the table (probably the table host). Some people have firewall issues so you would have to try to connect to their machine and if you could then you could offload table processing to that person. If the table host can't do it then there is no reason some other person at the table couldn't do it. So, in most cases, you could distribute the processing of each hand to some other computer and that computer would send the entire result of the hand to BBO at once.
If you get 4 people all with firewalls then you still need this table hosting ability in BBO itself so know you have the complexity of moving table hosting responsibility from person to person as each one leaves the table or potentially reverting it back to BBO...again complexity.

If vugraph times are in particular a problem, you could form an overlay multicast tree from all the people watching the vugraph and in this way offload BBO server processing to clients...again, more complexity.

[luis]

One split that won't hurt is to have one server for the playing area and another one for vugraph. After all either you are playing or watching vugraph. Even then the list of players can be "shared" between the servers so messaging will work even if the players are on different servers.
This I think will reduce the load a lot, and eventually can lead to an independant application -sort of- for vugraph with it's own server and everything that you need for a better vugraph experience. For example how far are we from installing an el-cheapo webcam in the place where the players are and have a table image while watching vugraph ?

[uday]

Yes. I'm almost out of short-term enhancements. Now, I need to find something that will last us longer.

The miniBBOs are attractive from a programming perspective (almost no work to get them going). They will allow for a convenient mechanism to allow an organization to run its "own" BBO. Is a split in itself desirable? I dont know. Would our italians, french, and poles like a site which is "for" italians, french and poles?

The alternative is to find a way to allow the servers to handle increasing growth while maintaining a single large BBO. At some point the client will have to change as well; if the login blast is unattractive now, it will be scary in a year. But the client wont *need* to change for a while yet. I don't know that I can say the same for the server.

I think I can scale up the server side as one large bbo without impacting the client. It will be far more work than simply splitting the service, but far less work than a rewrite or anything like that.


What I hope to get from this discussion is what I am already getting. A sense of direction. Split BBO or grow a single BBO?

What the client has to do to survive a huge BBO, or the server to support it is another issue, and I'd be delighted to have my plans vetted by email.

I'd not previously considered actually moving real work to the client side.

[TimG]

I think splitting into independent mini-BBOs is a bad idea. I don't want to have to search multiple channels for friends or for experts to kibitz, for instance.

I do think it would be a good idea to create more playing areas: have an open area and then have smaller areas meant to cater to English, Polish, women, gay, senior, whatever. But, I would be disappointed to see these areas run independent of each other.

I do not pretend to understand the programming issues involved, but it has always been my impression that users are given far more information than is needed. The prime example is the complete lobby listing upon login. I seldom have any need (or desire) to see a list of everyone who is online. And, when I do want this information, I ought to be able to request it. I'm not sure, but I get the impression that the lobby information is constantly kept up to date -- when I switch from chat to lobby there is no call to the server for a fresh list of lobby members, but rather the display is simply modified to show the information that I already have. This constant background updating seems to me to be a bad idea. I would think you'd save a ton of bandwidth simply by changing this. (I don't suppose bandwidth is your concern.)

Tim

[hotShot]

If you split up BBO, it does not matter how you intend to split it, it will end up with some sort of localisation.
If I tell my friends I'll be on BBO2, they will come to BBO2 too. They will tell their friends they can be found on BBO2, and so on. This will end up with somethin like a real BBOITALIA or at least BBO EUROPE, BBO Amerika, and BBO Asia, because of the local daytime, there will be ups and down in the player number, and i might be not attractive to join a BBO server that has his "night time" player minimum.

In the end BBO might loose it's "world wide bridge community" feeling.

So I would wote against it. But since this does not solve your problem, perhaps you can be a little more specific on how it's done today.


<client side approach>
You have control about the client, you could make the client run 3 connection to different servers, one is updating lobby information, one is transfering the chat data and one is handling the game. This way you could distribute the server load to more servers, without reprogramming much at the server side. And even the changes to the client should not be to hard, because all you need to chance is the connection used to transfer the data.

<Server side approach>
I guess you have at least 2 computers running, one with the database server, and one serving the clients. In that case it should be possible to get some kind of loadbalanceing done with a second server. One simple idea would be handling odd and even table numbers on different machines, or separating tourneys and team matches from the MBC. If the client knows wich server to contact, there is not much that has to be changed with the serverscripts. If all transactions are logged to the database this should be a "small" problem.
"Local communication between the servers can be done using Gigabit-Ethernet so there should be no bottleneck there.

If you use PHP/Perl for the serverside processing, you will have to work with precompiled scripts by now (otherwise you are using up to 80% of your servers computation power to recompile the scripts every time they are called.).
But are your scripts able to keep the database connection open between calls?
Lots of PHP-scripts I've seen, need more time to open and close the database connection than they need for the work they do. In this case you might benefit from running some of your CGI's in C or JAVA.
[Warning: The following might cause a flame war.....]
Don't let some incompetent people tell you that JAVA is slow, if you let it handle real workloads it can compete with C any time (don't use tomcat!). A JVM is loaded when the server is started, a servlet is compiled when it's first called, and it stays compiled until server shutdown. If can keep it's database connection between calls, to get maximum performance.
If you use e.g. a Apache webserver you can connect many servlet engines on different computers, and it will do the load balancing for you.
[End]

But without "insight" to some details it's hard to give qualified advice.

Of cause reprogramming the whole seversoftware is a lot of work, and should be considered last resort. But doing a propper boad balancing is not so much different from what you need to do to syncronize several BBO servers.

[nikos59]

I can't really add much to the erudite comments here,
but if you want to collect opinions then please note
that I hate the idea of splitting BBO, especially along
geographical lines. The only split I can stomach is
the one suggested by Luis, i.e. Vugraph vs Playing.
I also don't need the huge list of 6000 players,
only friends-stars and vacant tables.

[bestguru]

I agree that splitting BBO is a bad idea except for Luis's idea of splitting of VuGraph.

I love BBO and truly appreciate the free service, so if i can help with vetting designs or even code let me know. hollywood77s <at> aol <dot> com.

[Walddk]

I will join the vast majority. Keep it as one BBO if at all possible. I am absolutely clueless as far as clients, servers and bandwith are concerned, but I do know that we will lose too much of the BBO we know today if we divide this lovely site into multiple mini-BBO's.

I also support Luis' suggestion regarding vugraph. That could be one independent part of BBO if that solves any of the major technical problems. The same goes for the huge lobby list. Honestly, who needs this?

Quite strange really. I was told how delighted Fred was when user #100 joined BBO. Now we seem to have a positive problem. We are too many, and we will become even more. Hope we are not about to suffocate from the enormous success BBO is.

I fully trust Fred and Uday to solve the problems and at the same time make as many members as possible happy.

Roland

[bearmum]

nikos59, on Mar 19 2005, 08:52 AM, said:

I can't really add much to the erudite comments here,
but if you want to collect opinions then please note
that I hate the idea of splitting BBO, especially along
geographical lines. The only split I can stomach is
the one suggested by Luis, i.e. Vugraph vs Playing.
I also don't need the huge list of 6000 players,
only friends-stars and vacant tables.

I too have no expertise as to the difficulty of coding Bandwidth etc, but agree I would HATE to see many "mini" BBOs
Splitting VuGraph sounds good, and perhaps when in a T room ONLY seeing table chat and director's comments?

[Dwingo]

I will be really saddened the day BBO splits. I sincerely pray that Fred & Uday can solve this problem.

Once it splits, it will lose its competitive edge. Size matters, especially if you have plans to raise resources through Advertisements on the site.

Change is tough. But this one, if it happens, will be a huge shock to many.

[luke warm]

'splitting' bbo might not be a bad idea, in theory, and might even be necessary, but i would certainly hate to see it happen... i'm sorry i have no solutions to offer, this sounds like a serious problem

[Aberlour10]

In my opinion is this what I call "one-world-bbo-spirit" the most important value
of the site, please save this feeling if it's technical possible...

Robert

[keylime]

I actually would like BBO to have some scaling that is regional based - might help improve both the rate of infusion for new players to the online form of the game and also give the group of players a sense of identity in their part of the world.

What I'm thinking the problem would be, is for those who have international friends and keeping track of them all.

[epeeist]

I also would hate to have it split.

If, for technical reasons it has to be, then I think separating by function -- e.g. tournaments, teams, clubs, vugraph as has been suggested -- would be better than separating by language or region. One of the nice things about BBO (well, sometimes...depends upon whether or not there's a language in common ) is interacting with people from different parts of the world. If it had to be separated, at least a separation by function would preserve some of that interaction.

[Brandal]

epeeist, on Mar 19 2005, 09:07 PM, said:

I also would hate to have it split.

If, for technical reasons it has to be, then I think separating by function -- e.g. tournaments, teams, clubs, vugraph as has been suggested -- would be better than separating by language or region. One of the nice things about BBO (well, sometimes...depends upon whether or not there's a language in common ) is interacting with people from different parts of the world. If it had to be separated, at least a separation by function would preserve some of that interaction.

I agree with everyone who think splitting is a bad idea.

BUT.....I understood Uday as something's gotta give,
and my first thought was move one or two "parts" of
BBO to a second play-area.

I would assume since vugraph doesn't happen all day every
day,that wouldn't be enough?

Or maybe it's when vugraph takes place the masses
cause problems,I don't know.

I do not like the regional division.