Well the colder it is outside, cold air is "heavier" than warm air and as your stove burns and the hot gases are sucked up the stack, colder denser air creeps/pushed in through all the house's cracks and needs to be heated up and will make the house cooler because of it unless your stove is red hot all the time.
In your case, the air is not that cold and the outside air isn't being forced into the house like it would in the extreme cold like we deal with here, the hot air going up the stack isn't being pushed out of the house as fast either when it is warmer outside, so it doesn't produce as much heat.
When it is 20 above, there isn't much of a draft as at -40, then you would think the stove had a supercharger on it.
To solve that, put ducting from outside to the back of the stove so the cold incoming air hits the hot metal and is heated up, and it stops/slows the other house cracks from letting cold air in, your stove will breath easier and heat the house a lot more in both cases.
When it is -20 and colder, you can walk around your house and see where all the air leaks are at because of the frost around where the cold air comes in. You seal the house and the stove can't breathe either.
Also putting a fan behind the stove will help as well since it allows the air going past the stove to heat up faster, verses the cold air coming in cooling things down.
May not make a lot of sense the way I am discribing it, but that is what happens and I may not be saying it right, but I understand me!