5 Oct 95
Martyn Wild

The following account of on-going discussions over two of the AERA listservs (SIG ENET and ERL-L, Educational Research) might be of interest to ITForum members, particularly in light of the current lack of discussion, and the last paper published here--Lynne Schrum's Ethical Research in the Information Age: Beginning the Dialogue.

At 11:10 am, 9/30/95, Barbara Fuchs wrote:

I am beginning a dissertation on profiles of educators who use the Internet. I wonder if you would be so kind as to answer the following questions? How old are you?, male or female, education, teacher or administrator, are you married?, if so, do you have any children, what state do you live in ? What was your impetus in using the Internet? What will make you continue to use the Internet? Thank you for answering these questions. I don't think that many surveys of educators have been conducted on-line. I am looking forward to reading your responses. Thanks again.

Barbara

At 8:41 am, 9/30/95, S.D. Wiley wrote:

Hi Barbara;

How do we know you are really at Columbia and really doing a dissertation? Perhaps you are just doing marketing research? What do other members of this listserve think?

S.D.

I tried two or three methods of responding to the survey so not as to send my answers, some personal and quasi-confidential, to the person requesting. I indicated that the survey was poorly constructed and that I had difficulty believing that a dissertation committee and especially a major dissertation advisor would approve such a survey in its present form, lack of format, lack of control, and especially, lack of being able to identify me as a legal respondent. I forgot that the surveyor may not have been an honest one. Never-the-less, as much as I tried, I lost my ability to perform and send the response . . . . :) JR

[quoting Wiley] How do we know you are really at Columbia and really doing a dissertation? Perhaps you are just doing marketing research? What do other members of this listserve think?

I don't know, but I would guess marketing would be much better designed :-) I tend to just delete surveys as junk mail. Fortunately, there seem to be fewer and fewer of them, as more people tend to consider them impolite. There were so many surveys a couple of years ago, I am very surprised at the following statement accompanying this survey. The author must be very new to the net!

[....] I believe this is one of the first on-line surveys of educators who use the Internet.

When my students suggest online surveys, I suggest four things:

• That they don't do it. But if they insist:
• Do not simply post a survey to a list or newsgroup
• It may be more polite to post a short message to a list, briefly describing the topic and asking for volunteers
• Do not SPAM by sending their requests to multiple lists or groups

Yes, asking for volunteers affects your sample, but so does a blanket invitation on a list or newsgroup.

Don

Yesterday, I forwarded to you a survey concerning a dissertation that I will begin at Teacher's College/Columbia University. Some of you have expressed concern about the format of my survey. One person also questioned my honesty about being a doctoral candidate versus a marketing person.

Let me assure you that I am indeed a doctoral student, just beginning my dissertation, this is why the survey is still so crude. My professor has approved my choice of topics but has not approved a survey as of yet. How can I prove to you that I am not a marketing person, but indeed a doctoral student? Well, I have just passed a comprehensive examination in educational administration in May, 1995. Therefore, I am now enrolled in a research seminar at Teacher's College in order to begin my dissertation. In the coming weeks, I will begin to "tighten up" my surveys and I hope you will respond.

*If you haven't responded yet to my initial request, please try to take the time to do so.

Thank You.

Sincerely,

Barbara Fuchs

Dear Barbara Fuchss--

a few questions: (1) who is your dissertation adviser? Perhaps he/she could post a voucher for you to the list. (2) If you are at Columbia, then why are you accessing the list via the "EddieMurr@aol.com" address? Doesn't Columbia provide e-mail access to you? (3) Why have you not responded to the claim that many people consider the use of surveys on lists to be rude and therefore a critical percentage of your audience is already alienated? (4) If you plan to tighten up your surveys in the next few weeks, then why on earth would you ask us to respond to your survey now? (5) Is your work and attitude toward research typical among the students who currently are admitted to Teacher's College? If you are a sham, then you face some serious consequences, because frankly you are a poor reflection on the program, if indeed you are really a member. We await your replies

gary shank
gshank@niu.edu

Why don't you have a Columbia e-mail address?

I found your letter "skepticism" very interesting. I just answered the query without giving it much thought. Thanks for raising some interesting questions. I'll be a more careful reader in the future.

I also am shocked every time I see a "survey" or other research being done via the Internet. I have yet to see one that looks like it comes from anyone above a sophomore college student. May I add a few thoughts to the present discussion?

It may be tempting to think of doing research over the Internet, a "survey" of some kind, but ... who knows what the research design is, whether the instrument has been pretested, what the construct validates are, how biased the sample is, and, therefore, what both the internal and external validities might be?

1. If she is who she purports to be, kinda gives some insight on higher ed "at some places" don't it?
2. I don't think we'll ever get away from thoughtless solicitations whether they be "legitimately educational," driven by ulterior motives, or a complete sham...unless lists are moderated. Some lists are better served by moderation (screened by the list administrator before submission to the list) than others. But some lists are more conducive to (and benefit from) the spontaneity of immediate interaction. Of course, there's the issue of being taken in.....and possible overreaction.

I say it's a cross we have to bear for the access and "interactivity."

Although I agree generally with Gary Moore's concern about--was that "sophomore" or "sophomoric"?--survey research over the Internet, I would urge some patience with the intrusion as well as the violation of good design principles. First, feedback from professionals such as Dr. Moore has potentially important educational value for the would-be researchers. The Internet allows students to reach beyond their classes, instructors, and institutions. This ought not to be undisciplined, but it ought to be encouraged. Second, these proto-surveys of list subscribers can be valuable pilots of various aspects of survey construction, content, and method. Research is expensive and I'd rather see some piloting that exploits this technology (and our time and patience) than one-shot, all or nothing, ill-conceived research done more to satisfy constraints of resources than revelations of intellect.

As with many such issues, rather than issue stern rebukes to the students who intrude with such surveys, we might strive to persuade their mentors and teachers--our colleagues--to assist and insist that students use the more responsibly and reflectively for planning or conducting research.

As a data collection tool the self-report survey is THE MOST ABUSED method in the annals of research...regardless of how it is disseminated. I'm more annoyed by unsolicited advertisements in listservs than I am about quick-and-dirty surveys here.

After assisting 1,200 graduate students on their thesis and dissertation research, I can attest to the misconception that surveys are "easy to make," "easy to administer," and "easy to interpret." Nothing could be further from the truth.

Before anyone embarks upon surveying Net users...especially members of a newsgroup or listserv, they ought to first seek guidance (and approval) from the list owners. They need to determine if the listserv membership is representative of anything more than a collection of computer users. Most of all, they should ask permission of the list users if they would like to participate in a survey, and, then inform the users as to the objectives of the study, how the data will be used, and whether they will see any summary of the results.

The listserv provides a captive audience... but, that should not be the sole reason to select it. Otherwise, there is a tendency for e-mail to be just like junk mail.

By no means should anyone here be so self-righteous and sanctimonious in their condemnation of someone's attempt to do research. We all started somewhere. Rather than get on someone's case, we should encourage them and guide them to do proper research. This should be done via private e-mail.

We don't need anymore of these "public floggings."

Ron Polland states in his "signature" note, that he knows a lot about many things and a little about everything. I wish I could say that. He also mentioned having assisted over 1,200 students on their thesis or dissertation research; quite a record.

I do take exception that the self report survey is THE most abused research strategy devised. How else can you access hundreds of kinds and pieces of information that are necessary in so many research investigations? When you give students tests in class, are you not asking for self reports? After all, the students are reporting to you about what they SAY they know. When you give attitude instruments, are they not self reporting how they feel?

I dare say that if we did away with self report as a mechanism of collecting research data, our research pot would be reduced to say 1/10 of what it is now. Maybe that would be good, but I doubt it. There are plenty of problems with self report data, that is for sure, but this form of data collection is essential for research. You cannot gather many sources of important information any other way.

Anyway, that is my opinion.

Dennis, I'm not advocating getting rid of surveys any more than I'd be advocating eliminating cars because people who drink have accidents in them. In other words, the fault lies not with the survey, per se, as it does with the surveyor who thinks the survey is a "one size fits all" research tool. There are well-done surveys. There are a whole lot of poorly done surveys. There are lots of bad surveys for reasons other than surveys are ubiquitous. I stand by my assessment.

They are easy to create and easy to use. Validating them is another story. I've seen too many that were neither reliable nor valid.

No, I didn't serve on 1,200 graduate committees (if that's what your comment alluded to). I was a statistical research consultant for six years (full-time) and six additional years (part-time). The 1,200 were graduate students from 30 different departments at Florida State University. Guess which department's graduate students selected a survey as their research tool most often?

As for my signature, I'm not ashamed of my talents. Besides, I've never built myself up at the expense of others. I don't criticize others to make myself appear smarter or better than they are. I've never denied anyone help on any subject that was asked of me. As a consultant, I don't tell people what to do: I make suggestions which they may follow or ignore.

I agree with you: we can't do away with surveys...just the bad ones. :-)

The discussion is ongoing....

Martyn Wild
Dept. of Computer Education
Edith Cowan University
Pearson Street
Churchlands, Western Australia 6018

Phone: (61)09 273 8022
Fax: (61)09 387 7095
E-mail: m.wild@cowan.edu.au