head to head
Neither is simply harder - they are hard in different places. Banished demands more optimization; The Elder Scrolls V: Skyrim demands more execution.

Optimization · Pure Optimization
Harsh village-builder; balance population, food, and winter.

Optimization · Pure Optimization
An open-world RPG mostly about discovering and stacking exploitable progression systems.
Banished leans Optimization (Pure Optimization); The Elder Scrolls V: Skyrim leans Optimization (Pure Optimization).
The Elder Scrolls V: Skyrim leans clearly harder on execution.
Banished
no mechanical layer
The Elder Scrolls V: Skyrim
Combat is forgiving; aim and timing rarely punish you.
Same score on reads - which means the number cannot separate them. The difference is in what each one asks of you.
Banished
no opponent
The Elder Scrolls V: Skyrim
Enemy AI is scripted and exploitable; not adaptive.
Banished leans a little harder on variance, though the gap is small.
Banished
weather, disease events
The Elder Scrolls V: Skyrim
Loot tables and leveled lists inject mild randomness.
Banished leans clearly harder on optimization.
Banished
population ratios and food planning
The Elder Scrolls V: Skyrim
Skill stacking, enchant loops, and exploits are deeply solvable.
A matching score does not mean a matching test. Two games can both sit at 10 on execution and demand completely different things of you - which is why the reasoning above matters more than the number beside it.