Author Topic: Amber's Luck and Dungeon Results: an Empirical Study  (Read 4766 times)

0 Members and 1 Guest are viewing this topic.

Banzailizard

  • Newbie
  • *
  • Posts: 12
    • View Profile
Amber's Luck and Dungeon Results: an Empirical Study
« on: September 12, 2015, 03:34:41 PM »
On starting up Armello for the first time I took a look at the roster of characters and picked Amber.  The parasol Katana seemed somewhat entertaining weapon.  I also rather related to checking the box for curiosity ten times over.  So I played my game and had fun. 

While I am relatively new to this game, (maybe having 20 single player games under my belt), I have played enough games in general to like to pull apart mechanics.  So I got to thinking how lucky is Amber?  (I warn you right now, I do not actually answer this, things are less straight forward then they seem).

The Method

The simplest way would be to just ask here, but that would be boring.  As a result I took the simplest experimental way; I would started a game where I could reach a dungeon on my first turn, get a result, quit, and reload.  Run 30 trials of this, and 30 on another hero.

What is key here is the way her trait reads; she is more likely to get positive results, rather then more likely to get a better value on positive results.  Thus let us compress all positive outcomes (magic, gold, follower, and treasure) into a positive outcome and all negative results (banes, and arguably portals) into a negative outcome.  Voila, blunt an instrument as it is, we have a binomial equation. 

X = P*N

Use our experimental data, we can slap in a p hat value, n = 30 and we have an EV (X) for Amber.  Do the same for the "unlucky" hero,  set up a ratio between there EV's,  and multiply the result by 100.  Now we have a % difference.

Why this matters:


Well besides my being curious, it matters when trying to make a strategic decision.  In fact I will use this screenshot from my game.



Let's say it is night time and one has this scenario.  One can either go for the town, and more or less guarantee a resulting 1 gold income from it at dawn.  Alternatively one can go for the dungeon and take a chance to get something good.  There are of course a lot of factors going into this, do you need gold most of all? How many trickery cards do you have which could benefit from the settlement discount. etc.  However, ceterus paribus, one option here must produce better results.  It is also possible that Amber's luck is enough to push the balance one way or the other.

Conducting the Experiment

If you want the result or just to view the data table skip to the bottom.

I was faced with the three normal options, a follower quest, a spirit stone quest or a treasure quest.  In order to minimize variables, I deiced to pick each 30 times.  It quickly became apparent this was not necessary.  Whenever I selected the follower quests (for a blacksmith) resulted in my getting +1 Magic.  Every time that I selected a spirit stone quest I got +5 gold, and every time I picked the Mirror Cape true sure quest I got +1 magic. I limited myself to 10 trials of each to be sure.

I started to wonder if this lack of variance was only true on the first day, as a way to prevent the snowball effect with one player falling horribly behind early from a bane.  I then decided to pick a quest, not move from my clan grounds and run 30 trials at night.

I picked the Blacksmith quest, and I let the AI take their turns.  When it was my turn once more I moved over to the dungeon and received a mirror cape.  I then reloaded, waited for the banes to spawn, and guard to fight them (interesting side note there was never ant variation to either movements or battle damage)  and went back to the dungeon.  This time I got a portal which carried me to the nearest dungeon 2 hexes away.





Could this be some variation?  No.  28 trials latter (yes I did them all) every remaining reward was a portal, and always to the same dungeon.  Now I had something else to test, it seemed that reloading my game changed the result.  To test this I went to day 2 and did not leave my clan grounds.  My first result was a spirit stone, my subsequent results were +1 gold.  ( I stopped at 5 trials this time, as I had to sit though a bunch of guard moves, and a royal decree each time.)

Ok I thought maybe this was Amber's luck.  Rather then rolling for each dungeon she was guaranteed a specific number of positive results preset each round, more like a face down chip then a roulette wheel.



Again to keep variables down I used Barnaby (also rabbit clan), picked the same ring and amulet. This time I muted the game and turned on some of my favorite spreadsheet music ( sadly I do in fact have a spreadsheet filling music playlist). Again I ran 10 trials of each possible quest.  Once more the spirit stone quest was singled out as having a different reward from the dungeon.  While the follower (miner) and treasure (Wyld Talisman) each resulted in my picking up a spirit stone, the spirit stone quest gave me +2 Magic.

Once more I repeated the night trial, staying in my clan grounds until the sun went down.  Once more the first continuous trial (sans reload) produced a different result (Portal) then the subsequent reloading trials (Poppet)



The following day 2 trial confirmed that reloading the save changes the results. I got +2 gold without a reload and +1 gold with it.

I wanted one finial hero to confirm my findings that the quest selection affected the outcome of the first dungeon reached.  Here I picked Thane and only ran 5 trials of each quest.  It seems that it does impact the result but the pattern was somewhat broken, each quest produced a slightly different result, with the treasure (royal shield) producing the most unusual result (a bane).

In the instances where I killed the bane and stepped back on the dungeon I got +2 gold both times, this is hardly conclusive though as the sample size is small.

Thane produced another unusual result.  continuing the save had no different outcome from reloading it during the first night, all results were Banes.  In the two instances I killed the banes at night and stepped back on the dungeon I got further inconclusive results, one +1 magic, and a second bane. 

Even more baffling the second day test with Thane confirmed my other earlier findings on reloading the save.  First result was +5 gold, subsequent results were a follower (bard).



I have attached my data set for those so interested

So I did not really answer what I set out to which is how lucky Amber is.  I did learn a few things though:

Dungeon results appear not to be entirely random - given a constant set of in game actions, results are consistent.  This was itself the most consistent finding, only slightly undermined by the change as a result of reloading saves.

In game actions and choices do seem to have an effect of some sort on the outcome. -  During my Amber trials I consistently picked the same quested running though them one at a time (i.e. 10 follower quest trials in a row, then ten spirit stones, and finally 10 treasures).  For my Barnaby trials I intentionally varied the order of the quest selection at random.  This did not change the consistency of the correlation between quest choice and dungeon result.

Reloading the game in single player can affect the outcome.

Generally I would have to do more testing to draw any further conclusion.  However, I think I am due to play a few more complete rounds of the game and enjoy myself before I take on anything like that.

CloudCuckooCountry

  • Backer
  • Member
  • **
  • Posts: 467
  • Outstanding!
    • View Profile
    • CloudCuckooCountry Book Reviewing Channel
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #1 on: September 12, 2015, 04:09:55 PM »
It's been known for quite a while that dungeon rolls are seeded, so unfortunately your research doesn't actually tell us anything new.

I would recommend getting a member of the team to explain what "better results" means probability-wise. That's really the only way that we're going to know for certain.
This game needs more birds.

Banzailizard

  • Newbie
  • *
  • Posts: 12
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #2 on: September 12, 2015, 10:54:24 PM »
It's been known for quite a while that dungeon rolls are seeded, so unfortunately your research doesn't actually tell us anything new.

I would recommend getting a member of the team to explain what "better results" means probability-wise. That's really the only way that we're going to know for certain.

Really?  That is a pity guess its what I get for trying to do things alone (as I often do).  I suppose the best choice is to ask the devs then.

Kletian999

  • Backer
  • Member
  • **
  • Posts: 2534
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #3 on: September 13, 2015, 04:40:37 AM »
If you had started a new game each time instead of just reloading, you'd have a real picture of the variable spread of outcomes, but I know the time spent finding games where a dungeon was in reach could be pretty painful.

Banzailizard

  • Newbie
  • *
  • Posts: 12
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #4 on: September 14, 2015, 03:05:10 AM »
If you had started a new game each time instead of just reloading, you'd have a real picture of the variable spread of outcomes, but I know the time spent finding games where a dungeon was in reach could be pretty painful.

I will admit I am not sure how the process of seeding works in game design beyond a very basic understanding.  As a result of my ignorance here, I cannot assume safely that there are not some games where there is a greeter or lesser likelihood of failure.  That is to say in one game heroes might have a 20% chance of failure and in another a 25%.  There might be ways around this but I am not familiar with them.  (My statistical background is in economic and its subfamily econometrics, so I am far more comfortable with linear regression models. ) Alternatively if you could provide some assurance that p is constant between games, then yes I could test that and will.

Jason Bakker

  • League of Geeks | Developer
  • Newbie
  • *****
  • Posts: 44
    • View Profile
    • Jason-Bakker.com
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #5 on: September 14, 2015, 12:05:42 PM »
Hi Banzailizard!

So the tricky thing is that the dungeons themselves aren't seeded - the game as a whole is seeded, and the seed is saved out when a save occurs and restored when the saved game is loaded again. This is why as you got the same results over and over again if you did things in the same order. (The difference between the first time and subsequent times is actually a bug - we'll look at fixing it for the next update  :P)

We're not planning to release exact numbers for the dungeon changes or how they are modified by different variables (eg. Amber's Hero Power), but I'll explain the way it generally works below.

Amber's ability modifies the percentage change of certain dungeon outcomes - so every time you go on to a dungeon, certain positive outcomes have their chance increased, and negative outcomes have their chance decreased. The dice roll occurs, and the results will (on average) be improved compared to another Hero. But similar to D&D, even if your percentage changes are improved, you can still get a "critical fail" (roll a 1) and get a Bane or another result.

Banzailizard

  • Newbie
  • *
  • Posts: 12
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #6 on: September 14, 2015, 12:44:56 PM »
Hi Banzailizard!

So the tricky thing is that the dungeons themselves aren't seeded - the game as a whole is seeded, and the seed is saved out when a save occurs and restored when the saved game is loaded again. This is why as you got the same results over and over again if you did things in the same order. (The difference between the first time and subsequent times is actually a bug - we'll look at fixing it for the next update  :P)

We're not planning to release exact numbers for the dungeon changes or how they are modified by different variables (eg. Amber's Hero Power), but I'll explain the way it generally works below.

Amber's ability modifies the percentage change of certain dungeon outcomes - so every time you go on to a dungeon, certain positive outcomes have their chance increased, and negative outcomes have their chance decreased. The dice roll occurs, and the results will (on average) be improved compared to another Hero. But similar to D&D, even if your percentage changes are improved, you can still get a "critical fail" (roll a 1) and get a Bane or another result.

Thanks for the response. It is interesting to know that the whole map is seeded, though I am still a little unclear does this actually cause near random results? 

Also is there a reason specific numbers will not be released?

CloudCuckooCountry

  • Backer
  • Member
  • **
  • Posts: 467
  • Outstanding!
    • View Profile
    • CloudCuckooCountry Book Reviewing Channel
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #7 on: September 14, 2015, 12:51:57 PM »
Is there any particular reason you don't want to release percentage numbers? As far as I understand it, if Armello was a regular boardgame then the dungeon rolls would be determined by physical dice, which could be examined by the players and thus let them know what the percentage chance of any given result would be through simple math.

I know that you guys set out to "lean on the platform" of computers, which allows for things like Stealth, playing cards out of turn, Scout, etc, etc to be implemented gameplay-wise in ways that can't be implemented in a physical board game, but I thought that one thing that was carried over from board games was the transparency of all of the game elements.  Dungeons are the only random element in Armello which are obscured. For everything else the players know the chances of all outcomes.

Why obscure the percentage chance of dungeon rolls, but have transparent chance for everything else?
This game needs more birds.

Jason Bakker

  • League of Geeks | Developer
  • Newbie
  • *****
  • Posts: 44
    • View Profile
    • Jason-Bakker.com
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #8 on: September 14, 2015, 04:22:35 PM »
Thanks for the response. It is interesting to know that the whole map is seeded, though I am still a little unclear does this actually cause near random results? 

Also is there a reason specific numbers will not be released?

It's not so much that the map is seeded as that the random system is seeded. Across the entire game we use a system that basically rolls dice for us in the background - which quests you receive, which outcomes you get out of a dungeon, your quest results, and a lot more all use this system, and this is the system that is stored and restored upon save and load.

Regarding your and CloudCuckooCountry's posts, several gameplay elements are not transparent to the player - chances in Dungeons, the exact amounts of certain cards in the decks, how Quest rumours are selected, how the two King's Declarations are selected, how Spirit Stones crystallize, etc. and each decision was made from a gameplay perspective - whether it's more interesting for the player to know the exact odds, or to have an idea of the odds but not know the exact values when they make the decision - our goal was to have a mix of fuzzy and clear knowledge that goes into a player's decision-making process.

That said, Armello is a living game that we're constantly improving, and we're taking all feedback on board as we make decisions for the future of the game.
« Last Edit: September 14, 2015, 04:24:30 PM by Jason Bakker »

Kletian999

  • Backer
  • Member
  • **
  • Posts: 2534
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #9 on: September 14, 2015, 08:45:57 PM »
It's been discussed, if not outright stated to Alvorn that "notification of when the deck was reshuffling" was going to be put in to allow card counting if he wanted to put the research behind it.

Likewise the card rarity system lets us guess 1,2,3,4 copies per rarity.  I know these are tangents, but they are the other issues of transparency that we in the community would hope you increase.  Since steam is swimming in "Too much RNG" complaints, a little transparency might help calm some of the Boardgame stalwarts down.

dragoncrescent

  • Backer
  • Member
  • **
  • Posts: 679
  • ...is actually a rabbit.
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #10 on: September 15, 2015, 05:42:59 AM »
Hi Banzailizard!

So the tricky thing is that the dungeons themselves aren't seeded - the game as a whole is seeded, and the seed is saved out when a save occurs and restored when the saved game is loaded again. This is why as you got the same results over and over again if you did things in the same order. (The difference between the first time and subsequent times is actually a bug - we'll look at fixing it for the next update  :P)

We're not planning to release exact numbers for the dungeon changes or how they are modified by different variables (eg. Amber's Hero Power), but I'll explain the way it generally works below.

Amber's ability modifies the percentage change of certain dungeon outcomes - so every time you go on to a dungeon, certain positive outcomes have their chance increased, and negative outcomes have their chance decreased. The dice roll occurs, and the results will (on average) be improved compared to another Hero. But similar to D&D, even if your percentage changes are improved, you can still get a "critical fail" (roll a 1) and get a Bane or another result.

Quick question: what counts as a 'positive' outcome versus a negative one? There are various instances where Dungeon Warp or Banes could be considered positive.

Sorry Stoat

  • Newbie
  • *
  • Posts: 41
  • Most weasels agree; the stoat is sorry.
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #11 on: September 15, 2015, 08:02:14 AM »
Well the wording in Amber's ability is chance of Rewards. To heck with LOG if Banes are considered to be rewards in the game. xD

Andi1066

  • Newbie
  • *
  • Posts: 35
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #12 on: September 15, 2015, 07:13:13 PM »
i think ambers ability only increases the chance to get traesures out of a dungeon. Cause yeh...as one said already -> its situational what is considered to be a positive outcome.

And since the treasure symbol occurs right above her head before entering a dungeon it would make sense. Though this is just what im guessing. Due to experience i would also say this fits!

Biologist

  • Member
  • **
  • Posts: 318
    • View Profile
Re: Amber's Luck and Dungeon Results: an Empirical Study
« Reply #13 on: September 15, 2015, 08:40:16 PM »
I think "positive outcome" is basically "anything that isn't a Bane or a portal". While Banes and portals can both be useful in the right situation, they are generally the most disruptive to anyone who gets them. Usually when I step on a dungeon, I'm passing through on my way to a quest or the palace. I don't want to get warped to the other side of the map, or waste AP (and possibly get sent to start) fighting Banes.

Getting magic when you wanted gold, or a Follower card when you wanted a Treasure, isn't really a "bad" outcome, it's just sub-optimal. It'd be like complaining that you only found a $1 bill in a gutter instead of a $20 bill. It's still free money. I will say that Spirit Stone feels like a "bad" outcome, because unless you're focusing on that victory type or stumble into getting the other three, it's basically the same as finding nothing.
Pet peeves: Apes are not monkeys, jellyfish are not fish, and tomatoes are not vegetables!