http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

The math is a bit advanced, but the idea is to calculate a confidence interval for the win percentage, and then use the lower bound to rank.

95% confidence intervals for your examples:

20-3: 0.68-0.95
 1-0: 0.21-1.00

Since the 20-3’s lower bound is higher (0.68 vs 0.21), it would rank higher.

Here is a very simple solution, not so sophisticated as the chess rating systems suggested in the comments, but easier to implement: divide your current score value (win-loss)/(# of games played) by a “factor of uncertainty f”, where f is a high value if the total number of games played is small, and f converges to 1 the more games a player has played. For example, you can choose

f(k)=10-k 

after k games as long as k<10, and

f(k)=1 if k>=10

Or, if you want to double the confidence with each game played, choose

f(k) = 2^(5/k)

(the ^ means here “to the power of”, translate this to your favorite programming language).

The values 5 or 10 are just arbitrary choosen, reflecting the confidence you want to assign to the a player after one game, and the point where you want to reach (almost) full confidence. Choose your own values accordingly.

Just want a ranking system with win/loss. so lest say 20-3 is top in league right now. he is, say 23 weeks in so far. if one guy comes in and wins first match against anyone, I don’t want him to take #1 spot over people thats been there longer and have a great winning record.

If you want to do it uniformly, temper the win fraction with the amount of participation. The more games a player plays, the more his win fraction counts in the rankings.

An established player with a 20-3 (0.869) record who’s played all 23 (1.000) of the possible games has a ranking of 0.869 x 1.000 = 0.869. He loses nothing because of his high participation.

A new player plays one game and wins it (1.000) but is cut back by the fact that he’s played only one of the 23 (0.043). His ranking is 1.000 x 0.043 = 0.043. This puts him at the same level as someone who’s played all 23 and won only one.

A very simple solution is to give points for any game played, lost or won.

Score = 3 * win + loss

A new player won’t appear in the top ranks until he has a number of matches comparable with other players. This can be good, or bad – if your number of games isn’t bound new players might have no practical chance to ever get to the top.

Since you’ve dismissed the tried and tested option of an ELO style ranking system then you have to re-examine your needs.

[if a player] wins first match against anyone, I don’t want him to take #1 spot

… seems to sum things up best. It suggests that a crude win/loss ratio seems to be adequate and that the only requirement is to stop new players from appearing on the leaderboard.

To do that really you just need a minimum game count.

Something like:-

winRatio = win / math.max(win + loss, minMatches)

…would mean that someone would have to win minMatches in order to get that 100% win rate and after that many games they should be starting to find their level anyway.

Ultimately the fairest system for unmatched competitors is more along the ELO lines and you’ll find you will want that in the end unless all your players are very evenly matched – I would suggest you code for it now, start using it in the backend and then when they complain that the scoring system is still unfair you can look responsive to user demands when in fact you knew it was coming all along… sometimes our game is all about perception.