Problem Statement: From the player perspective you are routinely deploying patches to Shaiya which introduce bugs to the game that significantly impact the playability and enjoyment of the game.
Assumptions as to what is causing this: Nexon are apparently NOT able to test what they deliver in an appropriate manner. Perhaps they test at the unit test level or subsystem level but it would most certainly seem that they cannot test at the system level in the configuration used by AGE for functionality, performance, stability or interworking with the AGE components in the deployed system.
Add a Test Server, completely seperated from the OS and PX Servers, on which you can run new updates before deploying to OS and PX.
The Test Server need not be open full time, in fact you would likely only want to open it to players for designated periods.
You will need the Test Server to have a fairly significant population (for load testing) and it will also need the full range of classes, levels and "uberness" represented within its population. To achieve this I would propose that you start the Test Server by cloning one of the existing player databases.
The Test Server should most likely not be connected to the Item Mall unless you feel that this needs to be tested but if so then it should have its own Item Mall and all items should cost 0AP (you can't expect people to pay to test the game for you*).
Once in place I recommend the following test strategy:
+ Phase1: AGE internal (i.e. Test Server not open to the players) use case driven functionality testing. By this I mean that a defined set of basic actions and interactions are run through by AGE staff. These use cases would likely include:
-- Testing player interactions with the NPCs such as the WH, Blacksmith, BankTeller, Goldbar Merch, Pot Merch, ...
-- Testing attack/damage in 1v1 situations both in PvE and PvP situations, to do this I'd recommend you create a test matrix to ensure you cover all class combinations and the levels (15, 30, 45, 60, 70, 80) and you would also need to add some use cases to cover the range of uberness. This can probably be simplified to some extent because I doubt they'd be much value in testing Lv80 vs Lv60 damage (unless you want to check the VR senario!), i.e. it's probably sufficient to test within fairly tight level boundaries.
+ Phase2: Closed Beta (could be combined with Phase3). The objective of this phase is "live user experience" testing and it should only start once results from Phase1 are positive. What you want out of this phase is increased confidence that the "corner case" functionality that wasn't covered in Phase1 doesn't throw up problems - the point being that real players will always try stuff that you didn't even think about. You would likely want to restrict this phase to a nominated group of players who you can trust to provide level headed and constructive feedback - the GS pool would probably be a good starting point but you'd need a bigger population of Beta testers to get value out of this.
+ Phase3: Open Beta. Open to all players with the objective of load/stability testing the new content with a larger population. I'd suggest to incentivize people to take part in the Open Beta you run PvP events or such like.
-- My perception of things is that today you are basically going straight to Phase3 on the live servers with each new patch - i.e. using us, the players as Beta Testers, so you may well ask why run Phase3 on a Test Server. The answer is "expectations", people will have very different expectation when taking part in something as a Beta Tester vs being a Paying Customer (we all pay in one way or another with Shaiya, whether it be real cash or time).
Business Case: Ok, running a Test Server will come at some cost; AGE staff testing will come at some cost so how do you justify this?
+ What is the cost of not doing this?
-- lost revenues as players dessert Shaiya (and possibly AGE).
-- reputational damage?
-- risk of litigation?
+ What are the benefits of doing this?
-- better gaming experience on live servers will help retain existing player base and encourage new/returning players into the game = increase revenues.
-- increased player base = improved market share = strong story to investors
-- less server down time and fewer glitches = more game time for players to spend $$$'s
-- better quality of feedback to Nexon will strengthen the levers that you have to get that relationship working and possibly give AGE a way to pass on costs to Nexon for failures in their deliverables.
Finally, if you need any help in fleshing out any details on an appropriate test strategy or test plan I'm more than happy to put some time in to help you out.
* as you do now