So you’ve made some code changes that should hopefully speed up some part of an application. But there is just one problem – you don’t know how it will perform in live.
Different networks, different machines, different memory, different configuration, different times of day, different users, different server loads etc etc etc
So…
(a) Do you ask them if that bit of the application has got faster?
(b) Do you ask them to monitor that part of the application?
Or…
(c) Do you just ask them to see if they notice any difference in the application in general? After all, you can’t be 100% sure you haven’t affected something else.
Each of these approaches would seem to have problems (I know this as I’ve tried them all):
(a) You’re asking a leading question. Before such a fix goes life, there is an emperor’s new clothes scenario. The developer believes the code should be faster so there is some kind of crusade to get everyone believing it before it goes live. So…it goes live. Some people say it is faster, some slightly slower and others about the same. What then?
(b) Again: Some people say it is faster, some slightly slower and others about the same. What then?
(c) You’re no longer asking a leading question, but you start getting hit with a whole torrent of issues that people have known about for months (maybe years) but you’ve now opened Pandora’s box. How can you be sure which if these (if any) are new issues?
Or maybe I should be asking different questions entirely. Any thoughts?
EDIT:
No doubt there are processes leading up to this point that could potentially be improved. But the nub of the question is this:
The change is tested and has been put live. You have a myriad of stats to prove the changes should improve performance. Your boss has given the stats a once over but now wants to make sure the users are happy before signing off the release. What question(s) should you be asking the users?
15
What question(s) should you be asking the users?
Here is the next version of our program. Please report any unexpected behavior.
Unless your users are using stopwatches, or your program has some very long-running processes, users are unlikely to be able to provide anything but subjective feedback. And for optimization I’d be more worried about behavior not caught by your automated tests than an uneven speed increase.
2
You should concern yourself with data rather than subjectivity. Thus you should be monitoring/measuring/profiling to determine improvements. That monitoring needs to be in both production and test (and dev) environments.
Ideally, you shouldn’t be optimizing anything until after you see the results of performance. Otherwise you risk a “premature optimization” which doesn’t improve anything meaningful and perhaps was a waste of time and money. It is very easy to see a piece of code and think, “This is terrible, it will never perform well.” only to find out later that it is only executed 1% of the time and the real bottleneck is somewhere else.
4
Add a configuration switch to choose between old and new coding and put it into the production system. Run it one week with each setting, measure, and compare.
1
Can you not measure performance in a development environment, before and after fixes? If it’s 20% faster, it’s usually safe to assume it’ll be 20% faster in production too.
You should usually be able to find out quite easily the main bottlenecks in your infrastructure from past experience (or experience of other developers) and take that into consideration when profiling.
1
Your problem here is that you are involving users as testers, don’t do this.
Your question is quite valid – how does an app perform once it gets to the “real world”, with real world CPUs, user load, data volumes… but once you ask this question, you need to get some objective performance figures and users are never going to be much use for those.
So you need to emulate the user load, in a QA or performance test environment. Get a representation of the end-user environment, get some automated test tools that can hammer your system as if 10, or 10,000 users were using it, populate the data sets with masses of dummy data. Then you can run your tests and get some meaningful data. Then you can also re-run your tests with a new version of your system and see what differences there are.
1