Locating performance problems in massively parallel executions

Performance problems in asynchronous massively parallel programs are often the result of unforeseen and complex asynchronous interactions between autonomous processing elements. Then performance problems are not inefficiencies in source code, but gaps in the algorithm designer's understanding of a complex physical system. The analyst forms hypotheses about the probable causes or possible improvements, and verifies these hypotheses by modifying the program and testing it again. These hypotheses can be formed by a variety of methods, from simple and mostly fruitful techniques for suggesting possible source code improvements to the difficult, indirect, and possibly futile activity of visualizing execution. The author describes a visualization system for massively parallel execution data and shows how drawbacks in other analysis methods sometimes make visualization necessary despite its difficulty