Setting Up Enemy Systems That Don't Break When Players Hit Them

Most games handle enemies poorly because people treat them as afterthoughts rather than core systems. You will find this everywhere from mobile clones to indie titles with big budgets. The problem is not that writing enemy behavior is hard. It is that it requires understanding state machines, pathfinding edge cases, and performance budgeting at the same time. I spent three years building combat systems for a mid-tier RPG studio. We shipped two titles before moving on. The biggest headache was never the visual effects or the dialogue trees. It was keeping Enemies functional across every environment we threw them at without the game stuttering or agents getting permanently stuck.

The Real Work Behind Enemies

Before you write a single line of enemy code, you need to define what states an Enemy can occupy. This sounds obvious but most teams skip straight to movement logic. The typical state breakdown involves idle, patrol, alert, chase, attack, and fallen. Each state needs clear entry conditions and exit conditions. If your chase state has no exit condition other than the enemy dying, you are already designing a broken system. The state machine itself should be data-driven if at all possible. Hardcoding state transitions means you are going to spend hours debugging why an enemy is stuck in alert mode during a boss fight. Put the transitions in a scriptable object, a JSON file, or whatever your engine supports. This cuts iteration time from twenty minutes down to about forty seconds when you are tweaking behavior curves. Pathfinding is where things fall apart for most teams. NavMesh-based navigation sounds like a complete solution until your enemy gets cornered against a corner geometry that the baker missed. I ran into this on a project where our procedural levels created narrow corridors that were slightly under the minimum agent radius. Enemies would approach the player, hit the wall, and loop in place for eight to twelve seconds. The fix was not to lower the agent radius because that broke navigation elsewhere. Instead, I added a secondary local avoidance layer that gave the enemy a small random offset vector when it detected it was colliding with a static collider for more than two seconds. It felt like a hack but it eliminated the stuck-enemy bug across the entire build.

Performance and Optimization

Enemy AI is expensive if you treat every unit as equally important. The moment you have twenty enemies on screen, naive implementation means twenty full pathfinding updates per frame. That does not scale. You need culling based on distance and relevance. Anything outside the player view frustum by more than thirty meters should drop to a minimal update cycle. Update raycasts and navigation checks every third or fourth frame instead of every frame. Sensor systems are another area where beginners waste resources. Running a full sphere cast for every Enemy in sight every frame is overkill. Use a two-tier system where close-range detection runs on a tight loop and long-range detection runs on a slower timer or triggered by acoustic events. I structured one of our earlier games so that ambient noise from combat only activated high-priority detection in nearby enemies. Distant units stayed in patrol mode unless they received a broadcast signal from an alerted teammate. This reduced CPU load by roughly forty percent during large combat encounters without making the AI feel dumb. Combat behavior itself needs tuning beyond basic attack timers. A common mistake is giving every Enemy the same damage output and attack speed. Uniform enemies are predictable and boring, but varied enemies without balancing lead to frustration rather than challenge. Use layered difficulty modifiers instead of raw stat changes. An enemy that attacks slightly faster but deals less damage feels different than one that hits harder but telegraphs longer. Players respond to pacing, not just numbers.

Get the Full Details

Friends vs Enemies: Visual Guide for Kids’ Social Understanding #3745482 | Clipart Library
Friends vs Enemies: Visual Guide for Kids’ Social Understanding #3745482 | Clipart Library

Common Pitfalls When Building Enemies

The biggest pitfall is assuming your nav mesh will work perfectly in playtesting. Bake your nav geometry after every level layout change. Do not trust the default settings. Minimum folder distance, step height, and slope angle all matter. If you ignore these and ship a level where the agent cannot climb a single stair, half your enemies will walk in place at the entrance instead of entering the room. Another pitfall is over-designing enemy variety early in production. I have seen teams spend six weeks creating five unique Enemy types with full animation sets, only to cut three of them during vertical slice review because they did not fit the combat loop. Start with one or two solid templates. Get the AI framework working. Add variety once the core loop is proven. This approach saved us about two weeks of wasted animation work on our second project. Don't ignore the player's ability to exploit AI weaknesses. If your Enemy pathfinding relies entirely on a single NavMesh, smart players will learn to kite by retreating through narrow chokepoints. This is actually a feature, not a bug, as long as you have fallback behavior. Add wall-following or peek-ahead logic so the Enemy does not simply stand still when the player runs into an un navigable area. A simple behavior tree with utility nodes can weigh options like flanking, waiting, or calling for reinforcement instead of blindly following a path that leads nowhere.

When the System Fails Completely

There are scenarios where even a well-built Enemy system will break down. Multi-floor buildings with complex vertical navigation remain problematic across nearly every engine. NavMesh baking does not handle moving platforms well, and runtime navmesh generation introduces its own latency spikes. If your game requires enemies that traverse elevators, ziplines, or dynamically changing terrain, you should consider combining navmesh navigation with waypoint graphs for critical transitions. This hybrid approach adds development overhead but prevents the most embarrassing failure modes where an enemy simply stops moving because it lost the navmesh connection. Another hard limit is memory. If you are targeting consoles or low-end mobile devices, keeping detailed AI state for more than about fifteen active Enemies simultaneously is risky. The GC pressure from frequent state allocation can cause frame drops that make the combat feel unresponsive. Use object pooling for state objects and avoid spawning temporary collections inside update loops. A small profiling pass during a full combat scenario will show you exactly where allocations pile up. If you are starting from scratch and do not want to build an Enemy framework yourself, the Unity Asset Store and Unreal Marketplace have several mature options. For Unity, Opsive's Behavior Tree and Navigation packages handle most of the heavy lifting around five to seventy dollars depending on your needs. Unreal has its own AI system built in, though the behavior tree editor has a steep learning curve that will cost you a weekend to get comfortable with. Both solutions have documentation but neither replaces understanding the underlying concepts. Buying an asset without knowing how state machines work will just give you a fast way to create a fast way to hit problems later.

The best Enemy systems feel invisible. Players should notice the challenge, not the code behind it. That means spending equal time on balancing and edge case handling as you do on writing the initial behavior. Debugging tools that let you visualize sensor ranges, state transitions, and pathfinding decisions in real time are worth more than any feature you add directly to gameplay. I learned that the hard way during a public demo where an enemy chased a target through a wall because the debug view was disabled and nobody had caught the broken navigation path. Five minutes with the visualization tool would have prevented that entire incident.

What are the most common enemies in RPGs? - Game Voyage
What are the most common enemies in RPGs? - Game Voyage