Last spring, the ISTE editorial staff (Davis N. Smith and Kate
Conley) and
I developed a questionnaire in an effort to assess member satisfaction
with
JCSE Online and its print compendium JCSE Annual. Other
ISTE publications
were undergoing similar evaluation. Our goal was to determine how well
we were
doing our job as we intended and to see how we might improve the
journal. There
was also the possibility that our members would not consider the
journal useful
enough to continue.
We developed the questionnaire and, after review by the SIGCS
Executive Committee
(Pedro Leite, Rich Lamb, Chris Stephenson, and Dennis Jones), placed
it online
for members to complete via the Web. Only a few members responded.
Thinking
that we might have missed teachers as they finished up their school
year, we
asked again this fall (via the SIGCS e-mail list) that members
complete our
survey. The material below reports those results and my interpretation
of their
implications for JCSE Online.
We received responses from only 11 people. SIGCS has 1,078 members
(as of October
10), Thus, the response rate was approximately 1%. Not particularly
good. However,
only 188 of our members are "standard" members, i.e., they
specifically joined
SIGCS rather than selecting an all-inclusive membership. Presumably,
some of
the all-inclusive members are actually interested in SIGCS. I think it
reasonable
to suggest that we have about 200 to 250 people who are hard-core
members of
SIGCS. If we had 220 such members, the response rate would be 5%.
Neither the 1% nor 5% response rate is particularly good. They are in
line
with a similar survey of ISTEs Learning and Leading
publication.
I believe they had about 100 respondents from over 10,000
membersa very
similar response rate, but a sizeable number by itself. A response
rate of 1-5%
probably means the results must be assumed to have no validity with
respect
to representing the overall views of the membership. However, we can
use it
as data to help guide our actions.
We developed several categories of questions. They overlapped
somewhat and,
at times, different questions addressed the same issue from slightly
different
perspectives. We allowed substantial opportunity for ideas beyond the
questions
and issues we thought of. The issues we did think of addressed general
satisfaction
with the journal, form and frequency of publication, nature and
content of articles
published, and general importance of JCSE Online as a part of
the activity
of SIGCS. The results of these categories is presented below.
A major goal of this activity was to determine if our readers were
satisfied
with the last few years of the publication of the journal. We had come
off an
unexpected transition in editors and moved to online publishing. The
first set
of survey questions addressed reader satisfaction with the
publication. We asked
four questions about reader satisfaction with specifics aspects of
journal publication
and about their overall satisfaction. We also asked for ideas to
enhance their
satisfaction. The questions and responses are listed in the table
below.
Two respondents had suggestions they thought would increase their
satisfaction
with JCSE Online & Annual. One person suggested more
diverse-teaching
oriented material and the other thought increased frequency of
publication would
help.
When examining questions that have these possibilities, I would
expect that
if the satisfied customers do not constitute a solid majority, there
is a problem.
That majority might need to approach 2/3 to 3/4 of the respondents on
"satisfaction"
items. Using that benchmark, it appears that readers are generally
satisfied
with the journal but we clearly need to work on frequency and
regularity of
publication.
3 Form and Frequency
The second set of questions asked more specifically about the form
(online
or print), size (articles per issue), and frequency (issues per
academic year)
of the journal. The questions had unique responses.
When asked, "Do you prefer online publication, print publication, or
a combination
of the two?" the responses were:
- Both: 55%
- Print only: 18%
- Online only: 9%
Nine percent did not respond.
Responses to, "How important is a printed version of the journal?"
were:
- Very important: 55%
- Somewhat important: 18%
- Neutral: 18%
Nine percent did not respond.
An online publication need not be of a particular size. We wondered
what readers
might prefer on a continuum of one article per issue, some indefinite
quantity
called "many", or something in between these two end points of a
continuum.
The responses were:
- One at a time: 36%
- Many at once: 9%
- Something in between: 55%
We also asked about the regularity of publication, i.e., should we
try to maintain
a regular schedule or publish material as it became available. The
responses
in this case were:
- Regularly: 64%
- As available: 36%
Finally, we asked how often we should attempt to publish an issue. We
try to
span the school year with our journal so the maximum number of issues
is likely
to be nine (September to May). At the other extreme might be one issue
per year
with , two, three, or four issues seeming reasonable alternatives in
between.
As one might expect most choices were selected by someone:
- One: 9%
- Two: 9%
- Four: 27%
- Monthly/nine: 36%
It was suggested that we not try to publish more often than
monthly.
Results here suggest that we should maintain the use of online issues
with
an annual print compendium. Responses about frequency suggest we
should try
to be more frequent but probably try to publish at least a couple
items in each
issue. As noted above, regularity is important to the readers.
4 Content
The next set of items related to the nature of the content of the
journal.
We have gone from a newsletter format to a peer-reviewed journal to a
relatively
formal journal with mostly peer-reviewed articles. At all times, the
nature
of the journal was largely the choice of the editor (with, with some
impact
from the submissions received). We were hoping to gain some insight
into what
content and style our members would prefer in the future.
The first question addressed the "formalness" of the articles we seek
to publish.
We might wish to publish academic, often research-oriented
publications that
would often come from writers who teach at the college level. On the
other hand
we might look for more informal articles that people at all levels
might write.
The responses here were:
- Formal: 9%
- Informal: 9%
- Both: 82%
Academic journals generally require peer review (refereeing) of the
article
they publish. We wanted to see how important our readers felt refereed
articles
were to our journal. First we asked about the quantity of
peer-reviewed articles
that should be included. The responses were:
- All: 9%
- Most: 27%
- Some: 55%
- None: 9%
Approaching the same topic from a slightly different angle, we also
asked about
the importance of including peer-reviewed or refereed articles. The
responses
were:
- Very important: 36%
- Somewhat important: 36%
- Neutral: 9%
- Totally unimportant: 18%
We have been republishing articles from other venues relatively often
during
the last several years. This has been mostly due to not having enough
new articles
to fill the issues. We wondered if that mattered to the readers and
asked about
the importance of publishing only original articles. The
responses for
this question were:
- Very important: 18%
- Somewhat important: 18%
- Neutral: 45%
- Totally unimportant: 18%
Next we asked about the kinds of articles readers would like to see,
suggesting
a variety of possibilities. The choices offered and the percentages of
respondents
who thought each should be included are:
- CS Ed research: 100%
- Informal teaching discussions/examples: 91%
- Reviews of books, applications, programming environments, etc.:
82%
- Computing tutorials (e.g., how encryption works): 73%
- Scholarly pedagogical articles: 64%
- AP CS-related articles: 64%
- Editorials and opinion pieces: 55%
- Letters to the editor: 27%
Clearly we should continue publishing refereed academic articles on
computer
science education. However, we should also be accepting of much other
content
that relates to teaching computing. Indeed, we probably should seek
out such
material.
5 Importance of JCSE
Finally, we asked about the importance of the journal in relation to
the other
work currently done by SIGCS. Respondents rating of how important an
element
of SIGCSs work the journal is were:
- Very important: 55%
- Somewhat important 27%
- Neutral: 18%
This last item was very gratifying. Over 80% of the readers think the
journal
is an important part of SIGCS's work. No one indicated it is
unimportant.
6 Summary
While the results cannot be assumed to represent all readers, I
believe we
can validly reach several conclusions. We should
- continue to publish JCSE Online and the JCSE
Annual
- publish more often
- work much harder to adhere to schedule
- broaden the content of the journal
In a companion piece in this issue, I discuss my vision for the
journal in
the near future. That vision includes doing all the above as well as
expanding
level of coverage from high school to most of K-12.