java - Best 2d AI movement system

I'm creating a top-down rpg (no tile-based) using libgdx. Now I need to make enemies move on map with obstacles. What is the best approach to realize that? A*(I think, if enemy see player, he should turn to Pursue+Raycasting, because A* is bad for following moving targets) or just combining steering behaviors Seek+Raycasting?

Answer

There is no 'best' solution to this problem. Ultimately you're going to have to find by trial and error something that gives the best tradeoff between performance and believable intelligence.

However if you want to use any sort of path finding algorithm you're going to need to subdivide your world in some way. Whether you go with tiles, polygonal zoning or way-point nodes, your path finding algorithm needs some sort of graph to traverse. Using simple steering behaviours without any path finding will result in your AI getting stuck behind obstacles.

For seeking targets over large distances I use Jump Point Search which is considerably faster than A*, especially if your map has open areas.

For localized searching, such as aggro detection I use a simple Dijkstra algorithm. Unlike A*, with Dijkstra you don't ask it to search for a specific end point, you instead describe the type of end point and it will find the closest for which there is a valid path.

For example, you might have a soldier class within your game and you want it to attack any enemies that come within a certain radius of the soldier. You can simply feed the Dijkstra algorithm only the tiles that are within the radius. You then tell it that the only valid end points are tiles which contain an enemy. This will cause the soldier to always move towards an enemy if it is within aggro range.

You can couple the above approach with Steering Behaviours / Boids. Do the Dijkstra scan as described, then once an enemy target is found, stop scanning and allow your soldier to follow the path. Once your soldier is quite close switch to following by Steering Behaviours. This will work most of the time but because Steering Behaviours are not a path finding algorithm your soldier will occasionally get stuck behind objects in crowded areas. To solve this you must keep track of the distance to the enemy target. If the distance increases by X amount for Y seconds you know your soldier is likely stationary and the target is moving away from it. In this case, stop following with Steering Behaviours and perform another Dijkstra scan.