After a few days without major issues, most of the team started feeling relieved and enjoying a fresh start to the week. However, tech issues also know about Mondays, and a little mistake can create huge problems. This is our third Story from the War Room, and we call it the DNS Debacle - yeah, the title spoils it a bit, but as some people would say, ‘It is always the DNS’.

-------

At 16:45 UTC, some players began to encounter login issues. They were abruptly disconnected and found themselves unable to log back in. As the affected players were scattered across various regions, shards, and zones, our monitoring systems only detected a slight decrease in user activity and did not sound an alarm. Keeping a watchful eye on our active Discord community, one of the Mainframers spotted the troubling trend and quickly decided to raise the alarm and activate the War Room.

------

By 17:00 UTC, we were in diagnostic mode. The issue was with logins, and it was affecting a significant (and growing) portion of our player base. Although we still had a large number of active players, the curve was flatlining or starting to decline—right when it should be on the rise.
Our initial investigation through monitoring, metrics, graphs, and alerts pointed to a couple of potential smoking guns. One of our central services that handles logins was seeing increased CPU load, and our main authentication system logs were spewing errors. Additionally, we noticed issues with the system handling entitlements (determining who owns the game and what type of license they have) and decided to keep an eye on it while focusing on the most likely offenders.

-----

By 17:20 UTC, we decided to restart the authentication system, which seemed to be at the heart of the issues. In hindsight, this was a mistake. With players already facing login problems and constantly retrying, restarting the system exacerbated the issue. We ended up needing to close off all authentication requests temporarily to allow the system to stabilize, and as the game needs to refresh its authentication tokens regularly, this meant that over time all players were unable to continue playing.

Users playing across our three regions.
Troubles started at 16:45 UTC, and were subsequently made much worse by ourselves at 17:20 UTC

----

With our authentication system back on its feet, we decided to re-allow all requests to the system. As we now had all our players attempting to log in simultaneously, the previously noticed problem with the entitlement system became glaringly obvious. The system was overwhelmed and went completely offline, meaning even those users who managed to connect appeared as if they no longer owned the game. At this point, we knew that we needed to fight on two fronts at the same time. The issue with the entitlements system couldn’t wait for the root cause behind the disconnections to be resolved. Thankfully, we were on the verge of solving this one.

---

So, what triggered this bombardment you will ask? Unfortunately, it was a self-inflicted wound. During our diagnosis, we found that some DNS hostnames were not resolving correctly. One of the engineers on the call had identified the problem - a missing DNS record, specifically a critical NS delegation record. Another engineer immediately realized what had happened.
Earlier in the day, a cleanup task involved removing unnecessary infrastructure, and this crucial record had been mistakenly deleted. The last commands in the cleanup had been issued around 16:40, and it so happens that the default TTL (time-to-live) for many DNS records is exactly 5 minutes. A few manual steps later, the record was recreated, and we had to wait patiently for DNS caches around the world to update.

--

The root cause, however, is only the triggering moment, and with the on-call staff in our War Room assessing the situation and a couple of senior backend engineers ready for action, we still needed to tackle the entitlement service issue. We decided to fight it from two fronts at the same time:

• The escalation and support route. We contacted and escalated the issue up the support chain of the provider responsible for our entitlement storage. They confirmed they were overwhelmed and promised a solution to be deployed soon. The clock started ticking at 17:40 UTC.
  
• The hotfix path. We modified our entitlement code to include a fallback method favoring our players. If we had issues resolving entitlements, but the player had previously owned a certain license type, we would assume they still did and let them pass. We already had a code path similar to this one, but it was handling the case of being rate-limited. Adding the case to have a similar result for when the entitlement service was unresponsive was an easy task, and the hotfix was reviewed, compiled, built, and rolled out within 10 minutes.

-

We now had three racehorses competing in the same race: our hotfix letting players in while the service suffered, the entitlement service itself stabilizing with fewer requests coming in and fixes being deployed, and the DNS caches worldwide updating to recognize the recreated record.
By 18:35 UTC, the situation had vastly improved. Our players were logging in successfully, the entitlement system was catching up, DNS propagation was nearing completion, and the War Room team had turned a dire situation around.

The DNS Debacle had been resolved, and we were ready for the next challenge on the horizon.

Here’s a recap of the issues on Day 3 of our Early Access and their status:

Thank you to everyone who continues to report issues and bugs and find workarounds for some of them.

❖ We applied some fixes yesterday and the issues below are now solved:

• Clan issues (can’t invite/disband)
  
• Recipe unlocks and chat messages

❖ We’re still investigating the points below:

• Crafters/Chest Issues
  We are looking into an issue regarding removing some pieces like chests and crafters.
  
  
• Mining nodes becoming non-interactable
  We are looking into this and will provide an update as soon as possible.
  
  
• Teleporting to the home shrine from another province does not work
  We advise against using teleport if you’re not in your home province until we fix this.
  
  
• Founder Role on Discord (Steam players)
  We have an issue with the process of flagging the Founder Role on Discord for Steam players. We should have an update early next week.

Thank you for your patience while we’re solving these issues.

We encourage you to continue reporting bugs and issues in-game via the F8 tool. You can contact our support team for any issue affecting your account (https://support.playpaxdei.com/)

Pax vobiscum,
The Mainframe Team

For us, D-Day started relatively calmly. All servers in all 3 geographical zones were up and running, pre-warmed, and waiting for the expected onslaught. At Noon UTC, the first players started streaming in, and during the ramp-up, all systems were holding well, with no critical automatic alerts being generated in our monitoring systems.

-----

After about an hour, we started noticing that servers handling some of the more populated zones were starting to grow an alarming number of NPCs with a commensurate effect on performance. At the same time, early signals from the ground that we obtained through Discord started mentioning heavy rubber-banding in some zones which would be an expected consequence.

Those familiar with our military history[playpaxdei.com] from Alpha might remember that a runaway population of rabbits once brought us to our knees. In this case, it wasn’t specifically rabbits, but a more general swarming of mobs that apparently was triggered by a dense and very active player population where our population control was not able to keep up with the spawning of mobs.

As a temporary fix, while we were troubleshooting, we started identifying those servers that were on the brink and selectively restarting them. This meant a brief disconnect for players connected to those zones, but at least the server temporarily went back to a sane state.
Fortunately, we had been working on possibly fixing this in the previous days though we had never been really able to reproduce this under lab conditions.

We decided to fast-track the fix, which was a server-only fix that did not require a patch from the client. Building and deploying the fix to our staging servers took roughly 2 hours. We then decided to deploy it selectively to a few of the worst zones to see if it fixed the issue before deploying it in general.

Deploying a hotfix live to our whole armada of servers is always a risky proposal, and in this case, we prefer applying it to a subset of servers out of caution. Furthermore, once deployed on a zone, we would have to wait at least an hour before we could ascertain that the fix was actually working. Until then we continued to manually restart zones in distress, which was an acceptable stop-gap solution.

----

But in an invasion of this scale, we rarely have to face just one issue. From Discord we also started to receive reports of problems with recipe unlocks, with heavy delays involved in related operations.
Looking at our server, we already knew about one weak link in that system: our Redis server, which was used for replicating recipe notifications to all players (along with various other notifications, such as the damage state of party members, etc.).

This specific server can not scale simply by adding more hardware, so if it was reaching breaking limits, we would have to address it at a more fundamental level.

Again, we could temporarily mitigate the problem by restarting some related services, but we immediately started work on a more permanent fix, but that fix requires some more thorough testing before we can deploy it, so we will probably have to live with this instability one more day.

---

This distraction held our attention while a more fundamental problem was brewing under the surface and partially hidden in the fog of war. At around 16:50 UTC we started receiving reports from Discord that people had trouble placing plots and some were not seeing what they built while resources were still consumed. Originally, starting in the US region, this state also spread to the EU region but strangely left the SEA region unscathed. This battle ended up being the one that took most of our time and was the most difficult to troubleshoot, not resulting in a fix until around 02:00 UTC (D-Day + 1).

--

The culprit in this case was a link between our database and a service responsible for replicating building pieces to all players as they get built. The latter service is a managed service that has some pre-configured quota limits, which we hit at some point, leading to a throttling of the service.

Normally, this wouldn’t be a problem as we can dynamically increase the quota, but in this case, when the quota was increased and the database started communicating again with it, it had to restore a very large backlog of transactions that had piled up. Even though it was doing it diligently, it simply could not catch up, leading to very long delays between creating building blocks and replicating them to clients.

At some point, we could see a whole hour's delay. So for players, it looked like they could not build a plot, but the reality is that they didn’t see the operation conclude until after more than an hour delay. This is all very obvious with the clarity of hindsight, but it took a while for us to troubleshoot this and find the root cause. As we were still in discovery mode, we opted on the side of caution and allowed the service to drain itself at its pace to make sure no player state got lost, which took a couple of hours.

-

Once we found the root cause, we had some simple fixes to the configuration that we applied and restarted the services involved. The building state was nearly immediately restored and all regions were back to normal after this.

During all this, we had nearly forgotten about our initial worry: zone server performance due to mob population but we were happy to observe that after having applied the hotfix to selected zones, they seemed to be well under control and actually performing much better than before, which was great news. So, in parallel to all of the above, we started a rolling update of all zone servers across all our regions, which resulted in a brief disconnect for players while zones restarted but resulted in a much healthier CPU profile for all the worlds.

All in all, at 03:00 UTC (D-Day + 1) last night, after a nearly 24-hour shift, everyone in the War Room felt tired but happy. We could still hear some distant rumblings and we knew that the war was not over, but at least we had established our beachhead and were ready to move forward.

Our first day of Early Access has been a bit hectic, but our teams were—and still are—hard at work identifying issues, investigating them, testing them, and fixing them.

Here’s a recap of the issues on Day 1 of our Early Access and their status:


❖ Can't log in - returning to character screen loop: create a character on another World, log it in, log out, log your ‘looping’ character. It is not guaranteed, but it worked for some players.

❖ You can’t invite people to join parties or clans: ask your new recruit to look for your Clan in the Clan Finder and apply to it. You will then be able to approve them.

❖ Recipes are not unlocked properly: Place the corresponding item (the one that was crafted and should have unlocked a recipe) into a chest and pick it back up. This should give the recipes right away.

Thank you for your patience while our teams are working on fixing the issues.

Pax vobiscum,
The Mainframe Team

Here is the second part about our vision for Pax Dei. The first part talks about the high-level vision and the genre of Pax Dei—a social sandbox MMO, if you have any doubts.

Now, let’s look at the road ahead! As indicated, Early Access will last at least until June 2025. The game will change heavily in this timeframe as we keep building towards our vision.

We know where we want to go and how, but we also know that our plans will have to stay dynamic and will change to adjust to reality and the feedback our Founders will provide. All that to say - while this document aims to give you a good idea of our priorities in the next few months, please don’t consider it carved in marble.


As you’ve maybe seen in the playtests we’ve run so far, the village building, crafting, and resource loops are starting to work and feel nice. There’s something magical about wandering around in nature in Pax Dei and collecting the herbs and mushrooms in the forest, watching a sunset from a cliff, or discovering a little house hidden in the woods. Our system already supports many types of buildings, from little huts to gigantic castles through mansions or factories.

So, what’s the future of this part of the game? We have plans for advanced features, but right now, we are focusing on improving balancing and skill progression, polishing the first hours (yes, ‘pin a recipe’ is on the list), and adding more depth to the crafting system. The first content update after the Early Access launch will bring additional building pieces to complete some of the current sets (among them, new walls adorned with grapevines - a great way to add some cachet to your buildings!). We’re also happy to confirm that prop signs are coming to label storage or specific areas in your village - this has consistently been one of the most requested features for that part of the game.


Yes, the combat system needs work. Following player feedback in Alpha 2, we’ve spent the last few weeks focusing on improving the characters’ responsiveness in combat. We don’t think we’re there yet. Currently, our PvE is at a minimum functional baseline, and PVP is kind of fun but definitely not where it needs to be for the game vision to come together. The current PvP system is a test implementation and will be radically different moving forward. Overall, this part of the game will remain an ongoing priority for the foreseeable future.

Our mission here is challenging. We’ve taken an action-oriented approach to combat and want the skill ceiling to be relatively high. As an RPG, we also want gear and skill progression to offer depth and feel meaningful. Finally, we’re also an MMO, so the game needs to work for group combat in dungeons all the way down to a solo player hunting in the woods.

We’re assuming getting each of these aspects right and finding a good balance between them will take time, as the work related to fine-tuning responsiveness, animations, and NPC behaviors… is inherently slow. Our plan here is to keep improving things with every patch and engage with all of you in our Discord so that we can figure this out together.

In addition to tuning the combat feel, we’ll be adding more content—more weapons, armor, enemies, and spells. On the mechanics side, we are also focusing on combat flow from a tactical level, such as improvements for crowd control, role specialization, along with balancing special maneuvers and spells so they work both for PvE and PvP.

Finally, we will also start unveiling the game’s lore. It will be a progressive and iterative process, adding depth to the world through indirect storytelling. Who knows, mythical creatures might even start being sighted in remote corners of the world at some point…


This pillar is the part of the game where an active player community is critical to continue development meaningfully.

Our goal is to eventually offer a choice of roles and systems, allowing a rich social fabric to emerge in all the shards. All of it will be player-controlled and structured so that you can participate in the political intrigue or warfare as much (or as little) as you want. Ultimately, players will be able to create social structures above and or across the clans and go to war… or have eternal peace if they choose so.

Getting to that vision will happen one step at a time. In this area, we currently have the baseline player clans but almost no other features. This part of the game will be one of our main focuses in the coming months.

In Pax Dei, (almost) everything is player-made, whether creating coal for the blacksmith, baking a cake, or crafting a piece of magic armor using the rarest of Relics found in the bottom of the deepest dungeon. We’re convinced that trade will completely shift the game's dynamics, making it easier for players to specialize and collaborate at scale. No matter what part of the game you want to focus on, there will always be things you can produce that are useful to others.

How will that materialize in-game? Our team is working on bringing (back) gold into the game and introducing the first system allowing trading between players via a new type of item: the market stall. You will be able to build a market stall on your plot in the same way as any other prop. The market stall will unlock asynchronous transactions with other players, using gold as the currency. Gold, in turn, can be earned either by selling items and resources to other players or as loot from PvE.

This continues further. More features will be added to this corner of the game during Early Access, including altars and miracles - the regretted teleport feature may make a comeback. And also some ‘roles’ that will sit outside of clans. The first such roles we want to introduce are the knight and the soldier. But this is a story for another time; we’ll tell you more about them as we get closer.


For us, Pax Dei is the game of a lifetime. We’re fully committed to developing it for years to come and invite you, the players, to be part of this journey. If you believe in the game's vision and want to support a team making something fresh in the MMO space, please join our Early Access and help us build the virtual world you can inhabit for years to come.

With Early Access right around the corner, we wanted to restate the vision behind our game. Pax Dei is an unusual game, and we’re getting many questions about what it is exactly.

This article aims to tell you more precisely what Pax Dei is, and we will start with what it is NOT - not because it is funnier like that, but to be sure you won’t feel misled. In the second part, we’ll discuss the current state of the game and our plans for the coming months.


We do not have NPCs with yellow exclamation marks telling you they’ve lost a daughter in the woods. There is no linear, scripted gameplay that forces players into fixed group sizes or private instanced dungeons.

Zones and gear are not gated behind minimum-level requirements. There are great games out there that do this, but this is not what Pax Dei is about. In Pax Dei, the world itself is a mystery. Nobody holds your hand, and you can venture as deep as you want - at your own peril.


While inspired by the European Middle Ages and the mythology and superstitions of the time, Pax Dei does not attempt to be historically accurate and does not take place in our real world. Ours is a world where the myths, magic, miracles, and monsters of that era are actually real, which in turn reflects the ground truths of being alive during those times.


Yes, Pax Dei features a lot of gameplay that is similar to games in the survival genre. Your character starts with nothing, but you won’t starve from not eating or not sleeping. A big part of Pax Dei revolves around crafting and building, but players and NPCs can’t raid your home.

While Pax Dei and the survival genre share some similarities, we didn’t design it as a survival game. Once the social fabric is established, once some clans rule some valleys, the game will drastically depart from the survival genre.


In Pax Dei, you won’t be able to rush through the content to reach the maximum level in a week. There is no character max level. You also won’t be able to find wild animals behind each tree or some clearings with all the primary resources close to your starting point.

Instead, you will learn your surroundings, the landscape, the shortcuts, and where to find good spots for resources - and share them (or not) with your Clan or this friendly neighbor who often passes by to check if you need something.

In Pax Dei, expect things to take time. Building a castle takes a lot of effort. So does making gear that enables you to go to a deep dungeon run. The world is vast, and travel takes time, so exploring the whole Creation is a huge challenge.

We want Pax Dei to become a living shared world that can give you thousands of hours of joy, alone or with friends (old or new), at your chosen pace.


A kingdom of one has no true king. We do want the soloing experience to be highly enjoyable, and you can play without joining a Clan. That being said, we designed Pax Dei to encourage group play, and the harshest corners of the game cannot be soloed.

At its core, Pax Dei is about playing in a world shared with other players, offering ways for different play styles to connect and complement each other. We believe where Pax Dei will truly shine is in these social interactions between players, whether in cooperation or conflict, structured or unstructured, planned or emergent.



Pax Dei is an entirely non-linear game where everything happens in the (open) world itself. There are no per-party instancing mechanics like instanced dungeons, and there are no areas that remove the players from the shared playground.

You build your home inside the world itself for everyone to see and visit. You can travel the world freely and visit other players and their villages, trade with them, or join up for adventure with people you meet randomly in the wild. The goal here is to maximize the opportunities for meaningful social interaction.


Pax Dei is structured around three distinct game pillars, which map to different play styles but are all eventually entwined indirectly.

In the Heartlands, the (mostly) peaceful Homestead gameplay loop is about resource collecting, crafting, and building your home.

Adventure gameplay will bring you into the Wildlands, to prove yourself in PVP and PVE combat and earn rare resources.

Last but not least, the Civilization-level gameplay will allow large numbers of players to come together for a common purpose, whether cornering the market on morel production, building a Cathedral, or declaring war on a rival kingdom.

No matter how you decide to play, we want Pax Dei to feel enjoyable and meaningful - whether that’s being the best blacksmith in your valley or the most ruthless PvPer on the shard.


What we strive to build with Pax Dei is a living, breathing world. A world that comes alive through the actions of its players working together or against each other. A place where you can build your homestead, set up your smithy, or keep your Inn. A place of emergent civilizations. With towns and cities. With vibrant trade and artisan guilds. With churches and cathedrals, forts and castles. Where players can build kingdoms, gather armies of knights, and go to war… or sell flowers on the market. Pax Dei is what you make of it.

—
We hope this manifesto has given you a good grasp of our vision for Pax Dei. In just a few weeks, the game and the world will feel and look very different once clans and other social structures are firmly established.

Some valleys will be crowded with houses and forts, with plenty of opportunities for trade and social interactions; mercenary clans will offer their services to recover valuable inventory in deep dungeons, and scavengers in underpants will lurk in Lyonesse for unfortunate adventurers to loot!

The moon, though, will still bring comfort at night, and the sun will still light the world.
Stay tuned for part 2. It should come… before too long!

Pax vobiscum
- The Mainframe team

We are thrilled to announce Pax Dei Preview Week - Gallia's Grand Gathering, an exciting online event that will stream live on Twitch[www.twitch.tv] and YouTube from June 7 to June 11.

Gallia’s Grand Gathering will bring together content creators and their communities from all over the world to immerse themselves in Pax Dei.

During this special week, content creators will engage in a series of friendly challenges designed to test their creativity, teamwork, and gaming skills. By completing these tasks, they can win various prizes for their communities, adding an extra layer of excitement for everyone involved.

The highlight of the event is the Villa Extravaganza challenge, where creators and their clanmates will compete to build the best-looking village, castle, or countryside house. Our esteemed dev team will judge the entries and select the winners based on creativity, design, and overall appeal. This contest promises to showcase the incredible talents and imaginations of our community.

Enter the fray. Pax Dei is a social game, you need a team to succeed! Each of our partner creators has been granted a bunch of invitations to distribute to their community. This means you have a chance to support your favorite creators and play Pax Dei with them before the Early Access starts.

Throughout the event, you'll be able to follow the streamers' adventures on Twitch and other streaming platforms. To make it easy, here's a link to follow the Pax Dei category on Twitch:
Pax Dei on Twitch [www.twitch.tv]and YouTube.

To cap off the week, the event will conclude with a closing stream on June 11, hosted on AnnieFuchsia [www.twitch.tv]& DannehTV[www.twitch.tv]’s channels. This special broadcast will feature the announcement of the Villa Extravaganza challenge winners, an exclusive interview with our dev team, and a surprise reveal that you won't want to miss! It's the perfect way to wrap up an exciting week of community-driven fun and creativity.

• The Great Expedition: Explore Pax Dei and hunt some specific creatures.
  
• Gallic Herbarium: Gather a list of resources scattered around Gallia.
  
• Para Bellum: Gear up for combat, defeat some named, and set foot in the PvP zone - at your own risk.
  
• Villa Extravaganza: Build the nicest house, village, or castle with your clan.

You can check the list of participants on our discord server.[discord.com]

Mark your calendars, spread the word, and get ready to join us for Pax Dei Preview Week - Gallia's Grand Gathering. It's going to be a week filled with creativity, competition, and community spirit. We can't wait to see you there!

Pax vobiscum.
- The Mainframe team

We’ve said it before: Pax Dei will only exist with the support of its players. From the game's reveal to the formation of our early group of supporters on Discord or the two public Alpha tests we ran in the past few months, Pax Dei’s story has been one of community. Today, we’re inviting you to embark with us on the next step of the journey.

Pax Dei will enter Early Access on June 18th. The game is still in Alpha, heavily in development, and not yet feature-complete (as we write this, our teams are furiously making fixes and improvements based on the feedback of the recent public test). We also reached the point where we need to see it live to keep developing it.

Help us build the virtual world you can inhabit for years to come. Your feedback during Early Access will be invaluable in shaping the final game. We’ll also reinvest the revenue from Early Access into making the game better and bigger. Your support doesn't just keep the lights on; it fuels our passion to create a vibrant, warm, and living world.

We will be back in the coming days with more details about the game's state, our roadmap for Early Access, and some early thoughts about what the first big update will focus on.

And before that, head’s up! We are planning an event in partnership with some of our favorite content creators. It’s coming before too long very (very!) soon. Stay tuned!

From the bottom of our hearts, thank you for all the support. We can't wait to see you in-game!

Pax vobiscum.
- The Mainframe Team