Analysis: Framewerk (Cthulhutech) RPG System
As a direct result of Catalyst Game Lab’s decision to drop all the prices on their PDF’s, I decided to go ahead and pick up the Cthulhutech core rulebook for a fairly cheap $15. I have been curious about this game for some time now, mostly because it manages to combine two of my favorite things: Lovecraftian cosmic horror and giant fighting robots. Now, many of the self-proclaimed Lovecraftian purists out there on the interwebs have decried the game universe as a disgusting perversion of all that is the Cthulhu mythos. To be fair, the game is a far cry from the antiquarian investigations of the 1920’s that are more typically of the genre, but after giving the core book a good read, I would say that the authors have actually maintained a strong sense of the cosmic horror that is at the core to the Cthulhu mythos. The universe itself is basically a thought experiment along the lines of “What if the stars WERE right?”
But that is for another post and another time. In this post, I want to share my analysis of the Framewerk system that is the basis for the Cthulhutech RPG. I have always had a strong interest in RPG design, particularly in the statistics that underlie dice mechanics. So I present here a statistical analysis of how the Framewerk RPG actually plays out by the numbers. This may be the first in a series of analyses of some of my favorite (and most hated) game systems.
Tests
The basic game mechanic at the core of Framewerk is a test system that will be fairly familiar to most RPGers. On any test, the player must beat (or tie) a given difficulty number in order to succeed. The basic structure is:
(attribute) + (skill) >= difficulty
The book provides guidance on the various difficulty levels. Easy is an 8, while incredibly hard is a 28. Attributes range from 1-10 (at least for humans). Skills are the interesting part. Skills range from 1 (student) to 5 (master) but rather than a flat number, the level indicates the number of 10-sided dice to roll. So a skill at novice (level 2) would provide 2 dice. These dice are not summed up, but rather various patterns are looked for to determine the ultimate value. There are three possible values you can take from your dice roll:
- The highest number
- sum of multiple values (pairs, three of a kind, etc.)
- sum of straights (of at least three dice)
So, for example, if I had an Intellect of 7 and an expert (4 dice) level in Occult, then I would roll four dice. Lets say I get 4,6,7,8. The 6,7,8 form a straight so I would add these up to get 21 which I would then add my intellect to for a total value of 28. Thats a pretty good roll.
One nice thing about the system is that it creates a sort of poker mini-game around the dice rolling that makes this process exciting and somewhat unpredictable, while at the same time keeping it fairly simple and unobtrusive. My interest is in what the distribution of these various dice rolls looks like. Its fairly difficult to calculate exactly because of the vast number of possible outcomes. The easiest way to figure out what these distributions look like is to run Monte Carlo simulations. With enough dice rolls (or simulated dice rolls on the computer in my case), you will begin to see the underlying distribution. In my case, I performed 10,000 rolls for each dice level from 1d10 to 7d10 (characters can specialize in skills which gives them an additional 2 dice, thus you can really roll up to 7 dice if you have mastery of the skill and a specialization in the particular area being tested). Here is a summary of the distribution for each level:
| Roll | Min | 25th | Median | Mean | 75th | Max | SD |
|---|---|---|---|---|---|---|---|
| 1d10 | 1.00 | 3.00 | 5.00 | 5.48 | 8.00 | 10.00 | 2.89 |
| 2d10 | 2.00 | 6.00 | 8.00 | 7.70 | 9.00 | 20.00 | 3.01 |
| 3d10 | 2.00 | 8.00 | 9.00 | 9.80 | 10.00 | 30.00 | 4.11 |
| 4d10 | 3.00 | 9.00 | 10.00 | 11.92 | 14.00 | 40.00 | 5.20 |
| 5d10 | 4.00 | 10.00 | 12.00 | 14.11 | 18.00 | 40.00 | 6.02 |
| 6d10 | 5.00 | 10.00 | 16.00 | 16.17 | 20.00 | 45.00 | 6.56 |
| 7d10 | 5.00 | 12.00 | 18.00 | 17.72 | 21.00 | 50.00 | 6.84 |
The increase in means for each increment in skill levels hovers around two. The distributions underlying these means however can change dramatically with the skill level. Here are histograms of each distribution:
The 1d10 is just a uniform distribution (duh), but after that funny things begin to happen. First, between 1 and 10, the distributions look like half of an increasingly steep bell curve as the results become more and more likely to be high within this range as you add more dice. Second, the frequency of values begins to increase outside of the 1 to 10 range as you begin to hit multiples and straights. This is gradual at first but by the time you get to the 5 to 7 dice distributions you start to see the emergence of a second modal point at 18. In fact, by 7d10, the modal point of 18 is more likely than 10 itself. There is also some interesting bumpiness in the post-10 part of the distribution. Prime numbers above the 1-10 range, for example, have zero probability of occurring and certain other values seem to be more likely. For example, 34 is much more likely to occur than 33 or 35. So underlying that mean increase of 2 are what I can only describe as some freaky distributions.
Is that a good thing or a bad thing? Well, on the plus side, it creates a lot of excitement and suspense because you have the possibility of hitting some pretty wacky numbers. On the other hand, it creates a good deal of randomness. While everyone (almost) wants some randomness in their game, its never a good thing for random variability in the dice mechanic to completely swamp things like attributes and skill levels. So lets take a look at success rates under various situations.
Tests against static difficulty
First, lets look at the success rates of different skill levels against different difficulty levels. For this example, I am going to look at these success rates at three different attribute levels: 4, 6, and 8. These attribute levels capture the range you are likely to see in most human characters. Here are the resulting success percentages.
Attribute Score 4:
| Dice | Easy | Average | Challenge | Hard | Inc Hard | Legendary |
|---|---|---|---|---|---|---|
| 1 die | 70.3 | 29.7 | 0 | 0 | 0 | 0 |
| 2 dice | 93.6 | 55.1 | 5 | 1.8 | 0 | 0 |
| 3 dice | 99.2 | 75.6 | 17.7 | 8.7 | 1.6 | 0.1 |
| 4 dice | 99.9 | 88.4 | 35.6 | 18.5 | 4.9 | 0.9 |
| 5 dice | 100 | 95.4 | 54.4 | 30.5 | 9.2 | 2.6 |
| 6 dice | 100 | 98.1 | 69.8 | 42.9 | 15.2 | 4.8 |
| 7 dice | 100 | 99.4 | 79.7 | 52.1 | 20.4 | 7.4 |
Attribute Score 6:
| Dice | Easy | Average | Challenge | Hard | Inc Hard | Legendary |
|---|---|---|---|---|---|---|
| 1 die | 90.2 | 50.2 | 9.8 | 0 | 0 | 0 |
| 2 dice | 100 | 78.5 | 23.8 | 2.9 | 0 | 0 |
| 3 dice | 100 | 93.4 | 42.6 | 11.5 | 1.6 | 0.1 |
| 4 dice | 100 | 98.4 | 61.5 | 23.2 | 5.1 | 0.9 |
| 5 dice | 100 | 99.5 | 77.6 | 37 | 9.8 | 2.6 |
| 6 dice | 100 | 100 | 87.7 | 50.7 | 16 | 5 |
| 7 dice | 100 | 100 | 93 | 60.2 | 21.3 | 7.7 |
Attribute Score 8:
| Dice | Easy | Average | Challenge | Hard | Inc Hard | Legendary |
|---|---|---|---|---|---|---|
| 1 die | 100 | 70.3 | 29.7 | 0 | 0 | 0 |
| 2 dice | 100 | 93.6 | 55.1 | 3.9 | 1 | 0 |
| 3 dice | 100 | 99.2 | 75.6 | 14.6 | 5.3 | 0.9 |
| 4 dice | 100 | 99.9 | 88.4 | 29.6 | 11.8 | 3 |
| 5 dice | 100 | 100 | 95.4 | 46.1 | 20 | 6.5 |
| 6 dice | 100 | 100 | 98.1 | 61.1 | 29.6 | 11.2 |
| 7 dice | 100 | 100 | 99.4 | 71.2 | 37.5 | 15.3 |
There are a few things worth noting here. First the differences across difficulty levels are huge here. As an example, someone with an adept skill and an ability score of 8 has a 75.6% chance of succeeding at a challenging task, but only a 14.6% chance of succeeding at a hard task. In general once you move into the Challenging difficulty and beyond the sequential differences in degrees of difficulty are very large, although the exact placement of the big drop varies by skill level. So for someone rolling 7 dice the big drop is from Hard to Incredibly Hard, while for someone rolling 2 dice, the big drop is from Average to Challenging.
Second, atrributes are important here. I was initially worried that the variance of the dice mechanic would swamp attribute differences, but you actually see a nice progression here. Attributes seem to matter a bit more at lower skill levels, but that makes sense to me.
Third, there is always room for suspense because at the higher difficulty levels even the most skilled and gifted characters will face a challenge. Overall, I like the look of these success rates. Next, I move onto contests (i.e. opposed rolls).
Opposed rolls
Opposed rolls are called contests in Framewerk and they work much like they do in other systems. Each character makes a test based on some skill (i.e. Observation vs. Stealth) and the person with the higher roll wins the contest. Opposed rolls are a critical component of all game systems. You want to create variability in your system but you also want to have enough regularity that better skilled and gifted contestants win more often than not. How does the Framewerk mechanic stack up?
First, I will make a comparison of a contest between two different characters who have the same attribute level but different skill levels. This allows me to isolate the influence of skill levels on contests. Once again I run 10,000 contests between each dice level from one to seven and calculate the percentage of times that the more skilled opponent won. I should note that I ignore ties here as the core rulebook is ambiguous about what to do with them.
| 1 die | 2 dice | 3 dice | 4 dice | 5 dice | 6 dice | |
|---|---|---|---|---|---|---|
| 2 dice | 64.5 | |||||
| 3 dice | 77.3 | 59.9 | ||||
| 4 dice | 85.2 | 72 | 57.3 | |||
| 5 dice | 91.5 | 81.4 | 69.2 | 56.8 | ||
| 6 dice | 95.3 | 87.9 | 78.2 | 66.6 | 54.7 | |
| 7 dice | 97 | 92.1 | 83.3 | 72.8 | 62.3 | 52.3 |
In order to read this table, choose a value on the row and then find the corresponding entry on the column. The percentage value is the percent of the time the skill on the row will beat the skill on the column. For example, a character rolling 5 dice will beat a character rolling 2 dice 81.4% of the time.
Overall, I think the skill progression generates probabilities that most people will find reasonable. Myself, I find the probabilities a little on the low side, although still acceptable. For example, I would like a master of a skill to beat a novice virtually every time, rather than only 81.4% of the time.
You also get some interesting information if you read along the diagonal. Each diagonal give the probabilities associated with winning if you are X steps higher. So the first diagonal is the probability of winning when you are one step higher (64.5, 59.9, 57.3, 56.8, 54.7, 52.3) while the next diagonal is the probability of winning if you are two steps higher (77.3, 72, 69.2, 66.6, 62.3). It can be shown that these probabilities decline across all the diagonals. What does that mean? It means that skill differences at lower overall skill levels matter more than skill differences at overall high skill levels. So the difference between a student and a novice is greater than the difference between an expert and a master. Whether this is a good thing or not is subjective. I tend to like it as it encourages characters to spread around skills more rather than trying to max out a few.
For my second comparison, I take two characters with the same skill level but different attributes in order to determine how important attributes are to opposed rolls. Once again I run 10,000 contests for each skill level and for a margin of attribute difference ranging from 1 to 8. The percentages reported are the percent of times that the character with the higher attribute won.
| Dice | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| 1 | 55.2 | 63.6 | 71.7 | 79.4 | 85.4 | 90.2 | 94 | 97.3 |
| 2 | 56.1 | 66.9 | 76.5 | 83 | 88.5 | 92.4 | 95.3 | 96.7 |
| 3 | 57.2 | 66.8 | 75 | 80.6 | 84.8 | 87.3 | 89.9 | 90.8 |
| 4 | 55.9 | 63 | 70.3 | 74.4 | 78.5 | 80.4 | 83.3 | 85.1 |
| 5 | 55.2 | 59 | 65.7 | 68.8 | 72.9 | 75.2 | 79.5 | 81.1 |
| 6 | 54.6 | 56.8 | 63 | 65.6 | 70.4 | 72.2 | 77.4 | 78.7 |
| 7 | 55 | 56.2 | 62 | 64.2 | 69.5 | 71.6 | 76.5 | 78.1 |
There is a lot going on here, so it might be worthwhile to also show this graphically:
There are a couple of trends worth noting. First, the clearest pattern here is that attribute differences matter more at lower skill levels than at higher skill levels. You can see this in the graph both in the higher overall value of the lower dice lines and in their greater slopes. So a difference in attributes will play more of a role in differentiating students than say masters. Second, there is a caveat to that rule for 2 and 3 dice vs. 1 dice where attributes actually matter more for two and three dice in terms of the level but not the slope.
Overall, these results show a pretty consistent pattern and attributes do make substantial differences in the success rate. I would personally like to see attributes matter more, but its not a huge gripe. In most cases, you are unlikely to see attribute differences of much more than 3 or 4 and these values give a range of 83% to 62% success depending on the skill level.
So far my analysis of the Framewerk system is fairly positive. It has an exciting dice mechanic that can produce unpredictable results, but those unpredictable results don’t really swamp skills and attributes, and actually create some interesting patterns that one could argue are beneficial (like skill differences at the lower end mattering more than at the higher end). but now we come to the one really boneheaded mistake in Framewerk.
Critical Failures are a critical failure
Critical failures in Framewerk occur when half (rounding up) or more of the dice come up with one’s. My prior analysis has ignored the issue of critical failures and treated them just as another dice result. Now I will take up the issue. Unlike the dice mechanic overall, the probability of critical failures is easy to determine by using the formula for the binomial distribution. Here are the relevant probabilities:
1 die: 10%
2 dice: 19%
3 dice: 2.8%
4 dice: 5.2%
5 dice: 0.9%
6 dice: 1.6%
7 dice: 0.3%
A quick glance at these numbers will reveal a couple of obvious and deadly flaws. First, the probability of a critical failure does not necessarily go down with skill level. Rather, all even number dice pools have a higher probability of a critical failure than the immediately prior odd dice pool. This results from the issue with rounding up. Both a 3 dice and a 4 dice pool require at least two one’s but the 4 dice pool has one more die to produce those two snake-eyes thus giving it a higher probability overall. The second problem is the ridiculously high probability of critical failure for 2 dice. at 19%, novices are going to critically fail about one in five tests. That is very clearly ridiculous and suggests strongly to me that the critical failure system was not well-thought out.
I can’t claim to be the first person to notice this glaring problem, as many people have complained about this issue on the forums. Solutions seem harder to come by. Some people suggest rounding down the number of ones needed rather than rounding up, but that doesn’t fix the problem it just reverses it so that every odd numbered dice pool is significantly higher than the prior even-numbered dice pool. It also doesn’t make the 2 dice problem go away and creates an even worse problem for the 3 dice case which would then have a 27.1% probability of critical failure!
One solution would be to make critical failures symmetric with critical successes so that failing by 10 or more results in a critical failure. This would have the side effect of making the probability of a critical failure vary by the difficulty level (i.e. the more difficult the task, the more likely a critical failure becomes). I actually kind of like that set up. Another solution would be to require snake-eyes on all dice, but this makes critical failures virtually non-existent beyond the student and novice level.
Overall Impression
I have yet to take Framewerk for a true test-drive, but these diagnostics bode well. So long as you can figure out a house rule for the boneheaded Critical Failure problem, then the system seems to offer an exciting and suspenseful dice mechanic at a minimum of complexity.