I'd like to put another spin on the reactions to Tom's excellent think piece. It has for some time seemed to me that many of the criticisms of research in the IT field miss what for me is a basic point: IT is not a discipline, it is an applied field of practice. It draws on a number of disciplines (e.g., psychology) which do apply the disciplines of a science. But IT is defined more by the messy real-world problems it addresses than by the rules of inquiry of a discipline.
If you agree with the above argument, then I think a couple of conclusions follow:
(1) Aside from the native incompetence of researchers in the IT field, part of the reason the research isn't very scientific is that as a field of practice, IT doesn't really lend itself to formulation of testable scientific hypotheses, nor are the methods of science a good fit for many of the kinds of inquiry which are needed.
(2) As long as our expectations are for disciplined scientific inquiry, IT will come up short.
By the way, this argument applies not just to IT, but to practically any of the "applied" specialties in education, human performance technology, and even clinical practice in the helping professions. My thanks to Don Cunningham and A.W. Foshay for leading me toward this position.
The outcome of this line of reasoning is generally consistent with what Tom proposes for preferred methods of investigation in the field. Thanks again, Tom, for a useful examination of the issue.