No, this cannot be avoided.

The AirServer does not have its own wlan connection, therefore it cannot lose it. 

If you first change networks with your phone, then connect to AirServer, that can never have any effect, as you were not connected with AirServer at the time you changed networks.

You can’t really change networks while casting to AirServer: either you are on the same network as AirServer, or you are on the hotspot of AirServer.