European Libraries and Electronic Resources 
in Mathematical Sciences   

 

 What are the Expectations and Needs of Users for the EULER System?

Results of the EULER User Questionnaire (1998)

The following public report describes the main results of the EULER User Questionnaire. This public report belongs to the Deliverable D.1.1, "Report on User Needs of the Work Packages WP0 (Requirement Analysis)" of the EULER project.

Contents:


Abstract

The main aim of the EULER user questionnaire was to obtain the prospective users' suggestions and expectations for the EULER system. The questionnaire was completed by 132 persons from 16 countries, 85% of them are from universities, about 10% from libraries. The electronic version of the questionnaire was much better accepted than the print version which was primarily disseminated at ICM 98. Most respondents work with the Internet on a regular basis and are sufficiently equipped to make use of advanced technological solutions.
The answers to the questionnaire confirm that the basic design decisions for the EULER system are well accepted and also imply suggestions for the design of specific details of the services to be offered. The comparedly poor yield of the paper version of the questionnaire, some disparities in geographic distribution of answers, and the low number of answers from non-university related persons indicate that special efforts should be made to reach some target groups and to improve user feedback mechanisms.


Overview

 
The aim of the EU project EULER is to provide user-oriented integrated access to networked information resources in the field of mathematics for mathematicians as well as for other scientists and interested users. The aim of the questionnaire is to propose action to design the functionality of the EULER system on the basis of the research findings. 

The questionnaire was discussed in the EULER Work Package task 1.1 of the EULER project by SUB, EMS, NetLab, UNIFI and CWI. Responsible for the design was the task leader SUB Göttingen. All EULER partners distributed and introduced the questionnaire to the mathematical community and the associated library scene. Furthermore, all visitors to the EULER sites were invited to fill out the questionnaire. See Study Design and Implementation. 

The results show how the respondents to the EULER questionnaire expect the design and the functionality of the EULER system. The questionnaires give indications about the users' approach to the Internet. See Analysis and Interpretations and Summary and Conclusions. 

The EULER system will be designed in accord with the user needs. The conclusions of the EULER questionnaire for the EULER design and functionality will be drawn and arranged. See Summary and Conclusion. 

In order to see how the user needs and expectations will be taken into consideration one may look up the first version  the EULER search interface design and the planned result representation at: 
 STILL UNDER CONSTRUCTION ! 
 


Study Design and Implementation

 
All EULER partners in Work Package task 1.1 of the EULER project participated in the design of the questionnaire.  Responsible for the design was the task leader SUB Göttingen. Common requirements of questionnaire design were taken into consideration (e.g. Burton, 1990, p. 62-76). The following list shows the design of the thirteen blocks of questions with altogether 117 questions in 20 sections. 

Design of PART I: Participants Background:  sample group and participants background 

  • first block of questions (PART I, questions 1 to 3): questions on the respondents' backgrounds in the sense of profession, employment, education and work field. The aim of this block was to get information which persons belong to the sample group of the questionnaire. This was necessary because of the used method of free distribution on the Net.
  • second block of questions (PART I, questions 4): question on the Internet behaviour of the sample group and their attitudes in the search of information. The aim of this block was to evaluate the experience of using Internet services. From the participants' experience, specific suggestions for the EULER system could be determined.
  • third block of questions  (PART I, question 5 to 6): questions on the technical equipment. This could be a basis for the technical requirements for the design of EULER.
Design of PART II : functionality of the EULER system. Suggestions and attitudes for the design and the functionality of EULER 
  • fourth block of questions (PART II, questions 1): question what the users want to do on a one-stop-shopping site - The answers should show the expectations for a one stop-shopping site.
  • fifth block of questions (PART II, questions 2-3): questions which form of search and navigation is preferred - From this question we get the most useful ranking of search fields and the most useful form for search and navigation from the point of view of the participants.
  • sixth block of questions (PART II, questions 4- 6): questions which information resources the users want to search with possible restrictions of the type of information resource or the information provider - The answers locate the most interesting information resources for EULER with the possible search restrictions from the point of view of the participants.
  • seventh block of questions (PART II, question 7): question how the users want to get the search results - This is a suggestion for the most useful representation of results from the point of view of the respondents.
  • eighth block of questions (PART II, question 8): question to the need for getting a document and in which way - The answers establish the need for document delivery for the EULER system.
  • ninth block of questions (PART II, questions 9 to 11): questions which languages should be represented in EULER - Here the expectations of multi-lingual features could be seen.
  • tenth block of questions (PART II, questions 12 to 13): attitudes to search engines - This question shows the expectations and experiences to (common) search engines and databases of the participants.
  • eleventh block of questions (PART II, question 14): question what the participants want to do on the EULER site - This are the suggestions for the functionality in EULER.
  • twelfth block of questions (personal information): question if the participants want to get further information about the EULER project - In addition to the first block of questions here information about the sample (e.g. the countries where they come from) could be seen as well as the interest of the sample group on the development of the EULER system
  • thirteenth block of questions (technical information): in addition to the third question block 

  •  
SUB, EMS, NetLab, UNIFI and CWI distributed and introduced the questionnaire to the mathematical community and the connected library scene. Additionally, all visitors of the EULER sites were invited to fill out the questionnaire. The announcement of the EULER questionnaire took place in several publications, on mathematical conferences such as ICM ´98 and also on the Internet. 
From the EULER project sites and other sites like the European Mathematical Information Service (EMIS) sites (offered by the European Mathematical Society (EMS)) their visitors could follow links to the EULER questionnaire site (see Questionnaire). The questionnaire data were collected by a server in Lund. 200 paper versions of the questionnaire were distributed at the ICM ´98 (International Congress of Mathematics). 
The start of the Internet questionnaire action was July 15 and the end October 20. From this distribution method we got an unknown sample. The evaluation and analysis of the completed forms was done by the task leader SUB Göttingen with perl programs and other standard software. 
Mean values and graphic representation show the distribution of the assessment of the questions and mean values were chosen. In a detailed Annex percentages, distributions and numbers of answers to all different questions can be seen. The mean values are only valid for the special questions they were asked for. Because of the small group of participants specialized statistics computations were left out. In some special cases answers of special groups of the respondents could be seen, e.g. from librarians or from persons from special countries. 
 

The Sample Group of the Questionnaire

 
We received completed questionnaires from 132 persons with different professional and institutional backgrounds and nationalities. 
Figure 1 shows that 93 participants (70%) are from Europe, 15 persons (11%) are from non-European countries and 24 persons (18%) could not be located. 
 
Figure 1: Geographic distribution of the questionnaire participants given by the email addresses (N= 132; the expression "other countries" means Mexico (2 persons), Switzerland (2), Uzbekistan (1), Egypt (1 with an US email address), Columbia (1), Czech Republic (1) and Russia (1). ) 
 
Figure 1 shows that persons from the participating European countries are equally represented. Remarkable is the unusual distribution for persons who come from non-European countries. One reason for this could be their participation on the ICM `98 and another reason could be the contact of one person of EULER project to them (e.g. to respondents from Israel). 

The following Figure 2 shows that the overwhelming part (85%) of the persons, who completed the questionnaire, come from the university community. 

 
  Figure 2: Professions of the sample group (total answers: 176 (on 130 completed forms), multiple entries, A) university: 111 answers, B) industry : 2 , C) educational: 28, D) information resources manager: 6, E) library: 24, F) other: 5) 
 
 
Only 19 persons (15%) are from other institutions; 14 persons of these are librarians. Together with the 10 librarians, who work also in universities, the group of librarians represent nearly a fifth (18%) of the respondents. The university people and the librarians together make up 95% of the participants. Only two persons have some connection with the industry; three other persons with educational institutions (e.g. schools). That means that only the university community is very well represented but other communities such as industry or  private brokers not at all. The received answers to "job title" at the end of the questionnaire confirm this view. The largest group there is the group of professors/lecturer, followed by the groups of librarians, Ph.D. students and assistants. 
The strong university background of the participants cam also be seen by the distribution of the participants' employers in Figure 3. 
 
  Figure 3: Employers of the participants (total answers: 259 (on 130 completed forms), multiple entries, A) government: 42 answers, B) educational institution: 84, C) research institution: 85, D) library: 16, E) industry: 1, F) information provider: 3, G) publisher: 3, H) self-employed: 1, I) other form of employer: 0, J) non-profit organization: 23, K) for-profit organization: 1 ) 
 
Nearly 2/3 of the respondents belong to institutions, research or educational; 16 persons work in the central library and 8 persons in libraries are related to institutes (see Figure 3). 
The group of mathematicians who responded the EULER questionnaire can be seen Figure 4. 

Figure 4: Number of educated mathematicians (total answers: 120 (on 120 completed forms) A) yes: 83 (69%), B) no : 37 (31%) ) 

Of the 120 answers more than 2/3 we received from mathematicians; and 30% belong to other groups, e.g. librarians, theoretical physicists and computer scientists. The work fields for both groups are widely dispersed (see here). 
From these results we can conclude that the EULER questionnaire reached representatives of the expected  key target indicators:  

  • European 
  • mathematician
  • connected with university and/or  libraries 
The EULER system is designed for other groups as well, e.g. persons from  industry and information providers. But they were not well represented in the turnout.  

The completion rate was lower than expected. >From the 200 distributed paper versions of the questionnaire only 2 filled out from. Two forms came with no filled information but with information about the used browsers. If you are interested in the detailed completion rate, please look here. In the Figure 5 you can see which answers of the questionnaire have no representative completion rates
 

Figure 5: (Representative) completion rates to the questions. This figure shows the completion rate. The blue bars belong to none representative completion rates of the first part of the questionnaire, the violet bars to the representative completion rates of the second part of questions 1 to 9 of the second part of the questionnaire. The yellow bars show the none representative completion rates to  questions 10 to 14 of the second part of the questionnaire. The green bars show the none representative completion rates to different questions.  

Answers with none representative completion rates were left out of consideration since they were more than additional answers to another question. In the following analysis data with  none representative completion rates will marked in tables with an asterisk "*". A completion rate will be called  "none representative completion rate", if less than 2/3 of the group of all participants responded to the question. It is remarkable that from part II questions 10 to  14 only a half of the participants gave an answer. These results have  none representative completion rates. A possible reason could be the non standardized open question form which may have been too difficult to be answered without hints and comments. For this kind of interests, other methods like qualitative interviews could be used. Approximately 80 percent of the participants filled out their name, job title, organization, postal and email address. This shows the high interest in the development of EULER by the participants (see Figure 15). 

A methodical problem is the form of the questionnaire as a user or visitor questionnaire which could not be representative and could not find out the reason for a none usage (e.g. see Jaeger, 1997, p.13, quotation). 
Some other methodical problems, Jaeger ascertained in his study, may also account  for the problems of the EULER questionnaire: problems of communication, problems of representation (none participation), not completed answers, systematic failures and problems of the quality of the answers (see Jaeger, 1997, p.14-15). Problems of communication could be found for unknown terms like "proximity functions" or "Ariel" and not clearly formulated questions. Sometimes the completion rate could be seen as a hint for this kind of problems (see Figure 5). In the Figure 5 you can see the problem of not completed answers. A problem of representation could be e.g. the low participation of mathematicians from industry. A systematic problem in the EULER questionnaire is that persons who have no knowledge about computers could not take part in the questionnaire on the Internet. 

PLEASE NOTE: 
All percentage values in the following text refer to the number of persons who responded to the specific question, not to all participants. 
To compare the values for frequency and relevance is problematic because the significance of the values differs. Nonetheless, these values yield some useful and interesting hints. 

 


Analysis and Interpretations

The following analysis and interpretation will be presented according to the following subject question blocks used for the design of the questionnaire (see blocks of questions and the design of the questionnaire). 

If one is only interested in the results of special fields, one can use the following links: 

 

The use of the Internet by the participants and how their look for information 

(second block of questions PART I, question 4) 
 general services on the net:   frequency rank  
 (mean)
  relevance rank  
 (mean)
Internet services in general 1 (1.22)  1 (1.52) 
WWW 2 (1.28)  2 (1.62) 
Internet search engines 3 (1.95)  4 (2.04) 
library online catalogues 4 (2.46)  5 (2.05) 
bibliographic databases 5 (2.63)  3 (1.87) 
other online access * (2.90)  * (2.60) 
 
From this table, you can see that Internet services in general and the WWW are often used by the participants daily, Internet search engines and library online catalogues more weekly and bibliographic databases more monthly. About 34% to 40% of the participants use Internet search engines, library online catalogues, bibliographic databases and bibliographic/reference databases on the Internet every week. The participants are very familiar with the WWW; so the planned EULER user interface by the WWW is a good choice from this point of view. 
Other online access is not important for the group of all 132 participants. Only 11% of them answered the question of frequency of other online access with frequency at least ones a week. Only 13% of all 132 participants thought of relevance of other online access for their work. But it can be seen that for the people, who use other forms of online access, those forms are quite relevant (more than 57%). The participants mentioned for other online access: ftp, telnet, email, news, online databases and university libraries (see details). 
That means that  more general services will more often be used than special Internet services. This could be expected. The results show that the participants have experience with general electronic information systems on the Internet; that is above the average. E.g. Rutenfranz mentioned as one result of their empirical study that 64.3 % of persons belong to the group of mathematicians and scientists use their computer daily, 30 % more than ones week, 4.3 % one a week and 1.4% with a lower rate (see, Rutenfranz, 1997, p. 222). The results in this study are much higher. If you compare the order of frequency and relevance you can see that Internet search engines are used more often than evaluated relevant (see Figure 6).  
Figure 6: Frequency and relevance of Internet search engines (frequency (light blue bars) total answers: 127 (on 127 completed forms) A) 1/day: 47 (37%), B) 2/week: 50 (39%), C) 3/month: 22 (17%), D) 4/less: 5 (4%), E) 5/never: 3 (2%); relevance (dark blue bars) : total answers: 124 (on 124 completed forms) A) 1/highly important: 34 (37%), B) 2/important: 41 (33%), C) 3/no opinion: 31 (25%), D) 4/unimportant: 6 (5%), E) 5/not at all important: 4 (3%)) 

In the view of respondents bibliographic databases are more relevant than they were used. For the users' behaviour there could be different possibilities, e.g. the archived results were seen as not very good (perhaps by search engines) or the usage of information systems is too difficult. Another reason could be the access of the information systems. In the next table the access of bibliographic databases is analysed. 

Further details: 
57% of persons work at least once a week with library online catalogues. 73% of participants chosen "1" or "2" for the relevance of library online catalogues for their work. 52% of persons work at least once a week with bibliographic databases. 74% of participants chose "1" and "2" for the relevance of bibliographic databases for their work.  
See all Figures in the detailed annex. 
 

Access of bibliographic and reference databases by: 

 access  frequency rank  
 (mean)
 relevance rank  
 (mean)
access by Internet 1 (2.18)  1 (1.62) 
access by CD-ROM standalone 2 (2.87)  * (3.74) 
access by CD-ROM network 3 (3.83)  * (3.48) 
other online access * (4.32)  * (3.20) 
Access of bibliographic databases by the Internet is used rather weekly, access by CD-ROM standalone rather monthly and CD-ROM network access is used in most cases less than ones a month. The access by the Internet could be the most convenient way of access bibliographic databases. The low ranking of local access to databases CD-ROM differs from other studies 
(in the study by the Sachverständigenkommission (1995 p. 26) you could find the relevance for local databases with 60% for great relevance ("große Bedeutung"). ).  Also in contrast to the findings of the questionnaire for example in the University Göttingen we have a huge CD-ROM Net- about 120 CD-ROMs - which are used by the university and usage increased every semester. 
Further details: 
Over 50% of the participants never use CD-ROM network, CD-ROM standalone and other online access for bibliographic/reference databases. For 50% of participants the CD-ROM networks, CD-ROM standalone and other online access were not relevant.  Names of the used CD-ROMs  can be seen here 
See all Figures in the detailed annex. 
 

Usage of electronic full text resources: 

 electronic full text resources  frequency rank  
 (mean)
 relevance rank  
 (mean)
electronic preprints 1 (2.98)  1 (2.33) 
electronic journals 2 (3.18)  2 (2.42) 
electronic lecture notes/conference proceedings 3 (3.46)  3 (2.69) 
electronic delivery systems for print-only material 4 (3.91)  5 (3.30) 
electronic books 5 (3.98)  4 (3.19) 
electronic dissertations 6 (4.13)  6 (3.32) 
other full-text sources on the net * (3.91)  * (3.53) 
The participants use the following  electronic resources in this order: first preprints, second journals, third lecture notes/conference proceedings, fourth delivery systems, fifth books and  seventh dissertations. The average respondent uses electronic preprints, journals and lecture notes/conference proceedings  at least once a month and electronic delivery systems, books and dissertations less than once a month. A possible reason for this behaviour could be the actuality of mathematical knowledge in the different resources especially the relevance for the participants mathematical research work. Resources with high actuality were more often used by the participants. The high ranking for electronic preprints shows that an electronic library has a different centre  of resources than a "normal library"
The frequency of usage is similar to the meaning of relevance. Only for electronic delivery were lower evaluated and electronic books higher. The results for electronic books are astonishing because today there are a few electronic books available. The questions which electronic books the participants use should be examined further. 
 
Further details: 
Only 4% of persons work at least once a week with electronic books, 31% at least once a month and 70% less or never. 33% of participants chose "1" and "2" for the relevance of electronic books for their work and 44% "4" and "5" for unimportant. It could be seen that the relevance of using electronic books is divided.  
For the values of the group of librarians see annex to part I (click here). 27% of persons work with electronic dissertations at least once a month and 73% less or never. 31% of participants chose "1" and "2" for the relevance of electronic dissertations for their work and 45% wrote that they are unimportant with the answers "4" and "5". It could be seen that the meaning to relevance and the frequency of electronic dissertations of using electronic dissertation is very split. 67% of persons work with electronic preprints at least once a month and 32% less or never. 61% of participants chose "1" and "2" for the relevance of electronic preprints for their work and 20% wrote that they are unimportant with the answers "4" and "5". Electronic preprints are labelled to be very relevant for the work and are used very differently by the participants. The graphics for frequency and relevance do not correlate.  
50% of the persons work with electronic lecture notes / conference proceedings at least once a month and 50% less or never. 50% of participants chose "1" and "2" for the relevance of electronic lecture notes / conference proceedings for their work and 32% wrote that they are unimportant with the answers "4" and "5". This shows that the electronic lecture notes / conference proceedings are used very different from their assessment to be relevant.  
30% of persons work with current electronic delivery systems for print-only material at least once a month and 71% less or never. 37% of participants chose "1" and "2" for the relevance of electronic delivery systems for print-only material for their work and 49% wrote that they are unimportant with the answers "4" and "5". Electronic delivery systems for print-only material was labelled from nearly the half group of participants as unimportant and only a third use current electronic delivery systems for print-only material regularly.  
33% of persons work with other full-text sources on the net at least once a month and 67% less or never. 27% of participants chose "1" and "2" for the relevance of other full-text sources on the net for their work and 57% wrote that these are unimportant with the answers "4" and "5". It could be seen that more than half of the participants wrote that other full-text sources on the net are not relevant. The frequency of other full-text sources on the net is also low. 
See all Figures in the detailed annex. 
 

The technical equipment of the participants 

(third block of questions  PART I, questions 5 to 6 and thirteenth block of questions technical information) 
 
86% (113 persons with 3 multiple entries) of all participants of the questionnaire are connected to the Internet by the Netscape browser and at least 7% (9 persons with 1 multiple entry) by the Microsoft Internet Explorer. 8% of the participants did not specify the browser they use. The Internet Explorer is used almost equally in the versions 3 and 4. The Netscape browser is most often used in version 3 and 4 (see details). So it could be concluded that the participants followed the technical developments of web services (e.g. Java applications) could be used by most of the participants. But the technical standard for the EULER engine should be Netscape in version 3 and the Internet Explorer in version 3
   Figure 6: Connection to the Internet (Total answers: 173 (on 129 completed forms) A) campus: 116 answers, B) business/corporate: 16, C) home: 41, D) other forms: 0 ) 
At least 88% of all 132 participants of the questionnaire are connect to the Internet at their university office place
 

The expectations for a one-stop-shopping site 

(fourth block of questions PART II, question 1) 
 
 use of a one-stop information system  personal opinion rank  
 (mean)
search items to get the document 1 (1.49)
get an overview of the literature in some field 2 (1.74)
keep up-to-date with developments in special areas 3 (2.02)
get an overview of the publication activities of single authors 4 (2.20)
other * (2.30)
search items for verification of citations  5 (2.38)
 
The most important use for our respondents is to reach items in the information system in order to get the document, the second one is to get an overview about literature in some field, the third place there is the use to keep up-to-date with developments in special areas, the fourth place is to get an overview about the publication activities of a single author. And the last place is to search items for verification of citations. This rating means that the EULER system should provide a document delivery component by which the users have the possibility to get the document. The second usage rating to get an overview over the literature in some field and the third to keep up-to-date with developments in special areas. The subject-oriented access to EULER could satisfy this user needs. 
Further details: 
Using a one-stop information's system for getting the document is for 91% at least important and for 63% very important. See other usings of a one-stop-shopping site. 
See all Figures in the detailed annex.


The respondents' ranking of search fields and preferred form of search and navigation 

(fifth block of questions PART II, questions 2-3) 

The next Table tests the basic assumption that the user wants to search in the EULER user interface the same categories as in common library online catalogues. 
 

 preferred search field  personal opinion rank  
 (mean)
author 1 (1.44)
keywords 2 (1.72)
title 3 (1.84)
other fields to be included * (2.29)
Mathematics Subject Classification (MSC) 4 (2.48)
journal title 5 (2.79)
other classification * (2.91)
type of publication 6 (3.10)
date of publication 7 (3.13)
publisher 8 (4.02)
 
The participants want to search author (first), keywords (second) and title (third), followed by special classification MSC and journal title. This could be the possible order in the search interface.  It it very astonishing to see that the field 'date of publication' and  the field 'publisher' and  have the lowest rates of all fields, but we have to distinguish the different kinds of resources. The higher appreciation of keywords compared to classification is astonishing, as well. The result have  to be further investigated.  
Further details: 
90% of the participants chose the category author as their preferred search field and labelled this at least as important. 54% of the participants chose the Mathematics Subject Classification (MSC) at least as important for their search. Only 25% chose very important. 83% of the participants chose the category keywords as their preferred search field and labelled this at least as important. 51% chose very important. 29% of the participants chose the category date of publication as their preferred search field and labelled this at least as important. Only 6% chose very important. See other classifcations. See other fields to be included. 
See all Figures in the detailed annex.
 
  form of search and navigation -  
  search in the database 
 personal opinion rank  
 (mean)
menu-driven multi-field search 1 (1.44)
features for menu-driven multi-field search: "wild cards" 2 (1.63)
features for menu-driven multi-field search: selectable truncation 3 (1.94)
features for menu-driven multi-field search: clickable boolean operators 4 (1.98)
features for menu-driven multi-field search: proximity functions 5 (2.52)
one line simple search in all fields 6 (2.58)
command language interface 7 (2.97)
 
Further more they want a menu-driven multi-field search/navigation with the following features: first "wild cards", second "selectable truncation", third clickable boolean operators and last proximity function It can be recommended from this questionnaire that the search and navigation functions in EULER should be possible in a multi-field menu with the features: "wild cards", selectable truncation and clickable boolean operators. 
Further details: 
92% of the participants chose the form of menu-driven multi-field search as their preferred form of search and navigation and labelled this  at least as important. 68% chose very important. 
See all Figures in the detailed annex.
 
  form of search and navigation -  
  browse through different database fields 
 personal opinion rank  
 (mean)
index of author names 1 (1.91)
keywords 2 (2.01)
navigate through the classifications system tree 3 (2.30)
title index 4 (2.33)
other browsing fields * (3.44)
 
The participants want to browse the  author names first, second keywords, third the classification tree and fourth the title index. This is the participants' suggestion for the design order of the browsing fields. It is astonishing that proximity functions and command language interface are not rated very high. It is remarkable that browsing keywords is ranked higher than browsing the classification tree. A reason for this could be the common usage of library online catalogues where classifications are  not very often used by the users. Not all mathematicians are aware of  the difference between classification and keywords. EULER could explain this in help text fields and should also provide browsing by keywords. In recommendations "other browsing fields" e.g. journal title and series names indexes were given. 
Further details: See other browsing fields. 
See all Figures in the detailed annex. 
 

The most interesting information resources for the participants 

(sixth block of questions PART II, questions 4- 6) 
 
  important information sources  personal opinion rank  
 (mean)
bibliographic databases 1 (1.42)
electronic journals 2 (1.87)
preprints 3 (1.91)
online library catalogues and databases (OPACs) 4 (2.14)
proceedings 5 (2.16)
other sources * (2.36)
 
According to  the participants of the questionnaire, the most interesting information resources for EULER are bibliographic databases (first rate), electronic journals (second rate), preprints (third rate), online library catalogues and databases (fourth rate) and proceedings. It could be seen that the in the EULER project these information sources are represented. Publisher catalogues, citations and address directories were suggested by the participants. 
Further details: 
The respondents gave quite decisive answers about their ranking "highly important" of information resources: 1. bibliographic databases (84 answers), 2. preprints, 3. electronic journals (56), 4. OPAC library catalogues and databases (50) and 5. proceedings (47).  See other sources. 
See all Figures in the detailed annex.
 
  importance to restrict search  
 
 personal opinion rank  
 (mean)
restrict search on specific resource types  1 (2.76)
restrict search on specific information providers  2 (3.16)
 
This table shows that here is no important need to restrict search by resource types or specific information providers.  This is astonishing because the participants want to get the document on the one-stop-shopping site.  
Further details: 
 See all Figures in the detailed annex.

The most useful representation in the view of respondents 

(seventh block of questions (PART II, question 7) 
 
  
 useful result representation  personal opinion rank  
 (mean)
selectable long and short lists 1 (1.90)
refine search 2 (1.95)
selectable sorting by different fields 3 (2.19)
automatic relevance ranking 4 (2.70)
 
The respondents of the questionnaire want to get a result representation with selectable long and short list  first, then refine search and further selectable sorting by different fields. It is remarkable that the participants do not want to get automatic relevance ranking like they are used from common search engines. 
 
Further details: 
Regarding to the default or choices of result representation the respondents have given the following ranking for "very important": 1. refine search; 2. selectable long and short result representation; 3. selectable sorting of different fields; 4. sorting by automatic relevance ranking.  
See all Figures in the detailed annex.
The order of the found item should be presented as author, year, title and type; see Figure 8. 
Figure 8: Preferred order of found items (total answers: 203 (on 127 completed forms) A) by author: 90, B) by title: 36, C) by year: 52, D) by type: 15, E) by other: 10) 

It is astonishing that the participants evaluated the date as not so important but they want to get the date in the result representation as second item like in common form of citation in science. 
Figure 9: Preferred length of display (Total answers: 125 (on 125 completed forms) A) long: 33 (26%) ,B) short: 92 (74%) ) 

The participants prefer short display lists (Figure 9) with 20 hits per page in the result representation (Figure 10): 

Figure 8: Preferred hits per page (Total answers: 126 (on 126 completed forms), A) 10: 15 (11%), B) 20: 69 (55%), C) 30: 21 (17%), D) 40: 4 (3%), E) 50: 10 (8%), F) more: 7 (6%) ) 


The need for document delivery 

(eighth block of questions PART II, question 8) 
Figure 9: The need for document delivery (Total answers: 120 (on  120 completed forms) A) 1/very important: 43 (36%), B) 2/important: 28 (23%), C) 3/no opinion: 26 (22%), D) 4/unimportant: 19 (16%), E) 5/not very important: 4 (3%)) 

The answers to document delivery are divided. Automatic document delivery was ranked high by a majority of the respondents (59%; very important: 36% and important: 23%; mean value 2.28). 
Figure 10: Getting the document (Total answers: 125 (on 125 completed forms) A) scanned images (TIFF) by e-mail: 62 (50%), B) Ariel: 9 (7%), C) Fax: 7 (6%), D) copies via normal mail: 47 (38%))  

 EULER should provide a document delivery component via email or postal mail by the document delivery services of the partners. 


The need for multi-lingual features 

(ninth block of questions PART II, questions 9 to 11) 
 
 
  importance of multi-lingual features for  personal opinion rank  
 (mean)
user interface 1 (3.71)
main menu * (2.94)
help texts * (3.03)
search interface * (3.15)
thesaurus * (3.16)
classification * (3.55)
others * (4.08)
 

The multi-linguality in the user interface  was evaluated very low. This attitude to multi-lingual features could also be seen in the low rate of received answers. 

Further details (none representative completion rate !): 
The multi-lingual features are not very high ranked by the respondents. Only 20% of them saw it as important or very important.  
Of the 13 persons who gave the answer "very important" six persons come from France and two persons come from Germany. Further more one person from Mexico, one from Russia, one from Columbia and one from Sweden answered that multi-lingual features are very important. From the 26 people chose "very important" and "important" are  11 (42%) from France, 4 (15%) from Germany, 2 (8%) from Italy, 2 (8%) from Sweden, 2 (8%) without any e-mail address and 1 (4%) from each of the following countries: Mexico, Russia, Columbia, Italy and Switzerland.  65% of all French participants (17 persons) and 29% of all German participants (14 persons) of the questionnaire underlined the importance of multi-lingual features (13% of all 16 Swedish respondents, 13% of all 16 Italian participants). It could perhaps be conclude that the multi-lingual feature should be beside the English version designed in French and perhaps in German.  
Respondents, who are in favour of multi-lingual features ("very important") prefer them in the main menu (18 persons), the help texts (16 persons), the search interface (11 persons) and the thesaurus (11 persons).  See other desired multi-lingual features. 
See all Figures in the detailed annex. 
Figure 11: The choice of languages (Total answers: 100 (on 51 completed forms none representative completion rate) A) French: 34 (34%), B) Spanish: 10 (10%), C) Swedish: 5 (5%), D) Dutch: 6 (6%), E) German: 21 (21%), F) Portuguese: 3 (3%), G) Italian: 15 (15%), H) others: 6 (6%) ) 

The favour for Italian and German is as high as the representation of Italians and Germans/Austrians among the participants. More than twice than the French representation is the favour for French. Swedish and Dutch are requested less than their respective representation. 
 

Further details: 
Of  132 entries 71 have characteristics of languages (64 English [en], 5 French [fr] and 2 German [2]). Of the 71 entries 90% stand for English, 7% for French and 3% for German.  
 

Figure 12: The choice of languages in the Internet browsers (none representative completion rate because of 46% all questionnaires have no statements)


 

The usage of search engines and their evaluation by the respondents 

(tenth block of questions PART II, questions 12 to 13) 

There were several given reasons for the using of Alta Vista. The form of the question allows no easy interpretation. Some answers were put together under some points (problem of analysis): 

 
 reasons for usage Alta Vista problem of analysis ranking (possible answers for the usage of 37 answers)
number of  (relevant) hits  *1 (8)
easy usage  *2 (6)
not very selective/ wide range *3 (4)
fast *4 (4)
familiar  *5 (3)
Further more reasons were mentioned, e.g. search for French sites, possibility to refine the search or free selection of matches (see details). 
 
 
 reasons for usage MathSciNet/ Zentralblatt/MATH 
 problem of analysis
ranking (possible answers for the usage of 37 answers)
little garbage/relevant information * (5)
access in the institution  * (5)
fast * (3)
bibliographic information * (2)
wide ranging  * (2)
reliable * (2)
 

The reasons for usage of other search engines have very few statements from the respondents. The usage of the different search engines has in both cases the reason to get fast relevant information with wide ranging. But from the statements of the participants we could see two different views to get to get fast relevant information with wide ranging. One view is close related with a specialized subject-oriented access to the field of work. 
The question what makes a search engine interface design effective was answered only by a minority (only 29 persons filled out the text field). In most cases simplicity was the answer for Alta Vista and Zentralblatt/MATH. For details see here


Desired functionality of EULER 

(eleventh block of questions PART II, question 14) 

Here are some requests for  the functionality of the EULER engine: 

  • refine search:

  • "possibility to get lists of the database; possibility to refine a search" "1. The subscription price should be kept to a minimum. 2. Refined search capabilities on the output of the initial search are ESSENTIAL (a serious drawback of MathSciNet is their absence). 3. For bibliographical databases, it is important to have the information about translations, if any (often missing in MathSciNet and Math Reviews)." 
    See the question to refine search
  • help texts: 

  • "Good online help, especially on Boolean operators" 
  • links:

  • "many links to other information provider/systems, with a short description of what one would have to expect from the other provider/system." 
  • taxonomy:

  • "kind of mathematical taxonomy tree preferably better organized than AMS classification.", 
  • display:

  • "navigation with hypertext links (authors, keywords)in the results pages." 
    "I believe that it is convenient to have a possibility to select several items from results of (possibly) several search procedures and to save or print out them." 
  • MathML Display:

  • "personnel electronic shelf "Please take into account the upcoming MathML standard for display of mathematics via the web. I believe MathML will greatly facilitate the display *and* content-cataloging of math on the web. It will eventually allow us to read mathematics on the web directly via browsers." 
  • teaching:

  • "tree of categories, chains of information organised in something like tutorials (e.g. "Everything you want to know about trigonometry)" 
    "Overviews of Textbooks in Mathematics for Undergraduates, in particular new stuff. Listed by type of course e.g. "Basic Linear Algebra", "Mathematics for Biochemistry", (or whatever)" 
  • access to the EULER system:

  • "No, but show me how to start my searches from my own homepage! (See Q13.)" 

    For details see here
    Some desired functionality for EULER are: refine search, good help texts, many links, display selection for printing and saving search results, usage of MathML, looking for teaching material, special entrance points to the EULER search engine by personal homepages and the geographic distribution of resources (none representative completion rate) 


    Interest in the development of EULER by the participants 

    (twelfth block of questions personal information) 

    The following Figure shows the high interest of the participants of the EULER user questionnaire. The persons gave their email address to get further information about the project. 
    Figure 13: Respondents' email address rate for further information about the development of EULER 
     

 


Summary and Conclusions

Sample Group and their  Background 
The overall numbers of completed questionnaires should have been greater in comparison to the expected number of EULER users. Nevertheless, the 132 respondents represent a fairly distributed sample of the participating European countries. Most of them belong to different subfields of Mathematics or to the library scene associated with the mathematical world. The private and/or industrial sector is not represented at all. Therefore, if there is an agreement to continue the use of questionnaires, one should give more consideration on how to obtain responses from these sectors, too. Statistical analysis for specific subgroups of respondents was not done because the sample sizes were too small. 
The great Internet experience of the participants is remarkable, indicating that the respondents are qualified to make suggestions for the EULER engine. 
 
Expectations and Suggestions by the participants of the EULER user questionnaire  had to be EXTREMELY SHORTENED  AND SIMPLIFIED for this report. 

The use of the Internet by the participants and how they search for information  
More general services will be used more often than special Internet services. The participants are very familiar with the WWW; they access bibliographic and reference databases mostly via the Internet.  This confirms the correctness of the decision to use the WWW as the EULER user interface platform. 

Internet search engines are more often used than evaluated relevant. To some extent, this reflects that there are already more specific search options availabable for the needs of the mathematical community. The availability of integrated services such as EULER will further decrease the need for and use of traditional, non-specific search engines.  

The participants use the following electronic resources in this order: preprints, journals, lecture notes/conference proceedings, delivery systems, books, and dissertations (similar order for relevance).  This ranking will probably change as more resources and services are becoming available in electronic form. In particular, document delivery and alerting services are still in a prototyical stage and will gain importance in the future. 

Resources with high recency were more often used by the participants. This may well be considered a distinguishing feature of an electronic library as compared to traditional libraries. 

The technical equipment of the participants  
At least 88% of all 132 participants of the questionnaire are connected to the Internet at their university office place. The respondents were generally well equipped and able to use more recent Internet solutions such as Java applets. 

The expectations for a one-stop-shopping site  
The most important use for the respondents is to find and retrieve specific documents in the information system, the second one is to get an overview about literature in some field, third is the use to keep up-to-date with developments in special areas, fourth is to get an overview about the publication activities of a single author. Last comes the search for items to verify citations. This means that EULER should provide a document delivery component where the users could retrieve the document.  

The respondents' ranking of search fields  and preferred form of search and navigation 
The respondents want to search author (first), keywords (second) and title (third), followed by special classification MSC and journal title. This could be the possible order in the search interface. 
Furthermore, they want a menu-driven multi-field search/navigation with the following features: first "wild cards", second "selectable truncation", third clickable Boolean operators and last, a proximity function.  
The respondents want to browse the author names first, second keywords, third the classification tree and fourth the title index.  
This is the respondents' suggestion for the design order of the browsing fields. It is remarkable that browsing keywords is ranked higher than browsing the classification tree. So EULER should also provide browsing by keywords. As not all documents contain keywords, automated keyword generation mechanisms should be considered for this purpose, as they become available. Other browsing fields could be journal or conference titles. 

The most interesting information resources for the participants 
According to  the respondents of the questionnaire, the most interesting information resources for EULER are bibliographic databases (first), electronic journals (second), preprints (third), online library catalogues and databases (fourth) and proceedings.  There is no interest to restrict search to specific document types or information providers. 

The most useful representation in the view of respondents   
The respondents of the questionnaire want to get a result representation with selectable long and short list first and then the possibility to refine a search that has produced a too large outcome. 
The order of the found items should be presented as  1. author, 2. year, 3. title and 4. type. The participants prefer short display lists with 20 hits per page.  

The need for document delivery 
EULER should provide a document delivery component via email and postal mail. 

The need for multi-lingual features  
Among the respondents, there seems no special need to be felt for multi-lingual features (no representative completion rates). 

The usage of search engines and their evaluation by the respondents  
The usage of different search engines is motivated by the desire to get quickly to relevant information. 
Simplicity makes a search engine interface design effective (no representative completion rates). 

Desired functionality of EULER  
Some desired functionalities for EULER are: refine search, good help texts, many links, display selection for printing and saving search results, usage of MathML, inclusion of teaching material, special entrance points to the EULER search engine from personal homepages and the geographic distribution of resources (no representative completion rate). 
 

Special questions to ask in further questionnaires 
-basic index with free text for searching 
-which electronic books were used by the participants 
-rate of  satisfaction with search, display, design and results  for the EULER engine 
-advantages of searching  specific databases/search engines  
-disadvantages of searching specific databases/search engines 
-advantages of the EULER system 
-disadvantages of the EULER system 
 

Further strategies for new questionnaires: 
-the possibility to answer "not clear" 
-standardization of questions relating to search engines 
-get more participants from industry and other relevant groups 
-special questionnaire for persons with low Internet/computer knowledge 

Suggestions and comments e.g. for further user questionnaires are welcome.  
Contact: grosse@mail.sub.uni- goettingen.de 

 

References

Burton, Paul: 
Asking questions: Questionnaire design and question phrasing. In: Margaret Slater, Research methods in library and information studies, London: Library Association, 1990, p. 62-76. 

Jaeger, Marcus: 
Planung einer empirischen Untersuchung in Bibliotheken am Beispiel des Projekts "Kundenbefragung in der Bibliotheksregion Ostwestfalen - Lippe". 
Fachhochschule Köln - Fachbereich Bibliotheks- und Informationswesen (Hrsg.), 
Kölner Arbeitspapiere zur Bibliotheks- und Informationswissenschaft Bd. 2, 
Köln: Fachhochschule Köln, November 1997.  

Rutenfranz, Uwe: 
Wissenschaft im Informationszeitalter. 
Zur Bedeutung des Mediums Computer für das Kommunikationssystem Wissenschaft, 
Studien zur Kommunikationswissenschaft Bd. 19
Opladen: Westdeutscher Verlag, 1997.  

Sachverständigenkommission Elektronische Fachinformation (EFI) an den Hochschulen in Bayern (Hrsg.): 
Wissenschaftliche Information im elektronischen Zeitalter. 
München: Bayrisches Staatsministerium für Unterricht, Kultus, Wissenschaft und Kunst, Juli 1995. 
Especially chapters  2 (Fachinformation) and  3 (Umfrage an den bayrischen Hochschulen), p. 7-22 and 23-42. 

Slater, Margaret (Ed.): 
Research methods in Library and Information Studies. 
London: Library Association, 1990.  
 

 


      SUB Göttingen                    Version: 2.0                      Date: 10 February 1999
Many thanks to Aleksandar Perovic (TU Berlin/EULER) and Ralf Schimmer (SUB Göttingen).
grosse@mail.sub.uni- goettingen.de
Becker@mail.sub.uni- goettingen.de
http://harvest.sub.uni- goettingen.de/grosse/questionnaire/kurz.html