Full text of DOI:10.1108/EUM0000000007134

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
IMPROVING THE ‘DARKNESS TO LIGHT’ RATIO IN
USER-RELATED INFORMATION RETRIEVAL RESEARCH
NIGEL FORD
n.ford@sheffield.ac.uk
Department of Information Studies
University of Sheffield, Sheffield S10 2TN
Research into information retrieval (IR) cannot yet answer the basic
question of how we can design IR systems to help people search for
information with optimal levels of effectiveness. In relation to
human-system interaction, we have failed to develop any valid and
at the same time robust user models capable of driving practical sys-
tem development. If we strip away assumptions and over-optimism
relating to the generalisability of what are essentially sporadic and
fragmented research efforts, the great ‘darkness to light’ ratio char-
acterising our knowledge of human aspects of IR becomes apparent.
From a more critical and pessimistic (but by no means less realistic)
perspective, we are getting nowhere fast. A range of strategies is pro-
posed to improve the situation by supporting relatively ‘horizontal’
as well as ‘vertical’ knowledge integration. These consist of: greater
use of pluralistic research approaches; enhanced access to research
data; more multidisciplinary and multi-perspective integrative
reviews and conceptual mappings; and establishing a greater critical
mass of published research findings sufficient to support the gener-
ation of a less sparse and fragmented evidence-based knowledge
map. The potential of electronic publishing and data access for help-
ing achieve these goals is discussed.
1
.
INTRODUCTION
The ratio of darkness to light characterising our understanding of users, informa-
tion retrieval (IR) systems and the interaction between them increases sharply as
the interests of the IR research community expand to embrace the growing vol-
ume and heterogeneity of Internet users and content. Within this context, it will be
increasingly important to maximise the cumulative contribution of individual
studies to our emerging understanding. Such understanding entails both:
●
positive elements – that is, discovering the extent to which findings
➢
➢
may be generalised to other contexts; and
extend or confirm aspects of our existing cumulative knowledge;
●
and negative elements – that is, establishing
➢
the extent to which findings are bounded by their context and are
not generalisable; and
➢
the bounds of our cumulative knowledge.
Journal of Documentation, vol. 56, no. 6, November 2000, pp. 624–643
6
24
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
It is particularly important to map the contribution of individual studies onto
the emerging global picture – in negative as well as positive terms. This picture is
composed of different areas of knowledge characterised by differing types and
levels of supporting evidence. The nature and extent of such evidence derives
from qualities both internal and external to the individual studies themselves.
What may be thought of as relatively external qualities of a study relate to the
extent to which it confirms, supports, extends, questions or refutes other studies
thus contributing to the global picture. Internal qualities relate to strength of evi-
dential support offered by a particular study in terms of qualities such as accuracy,
validity and reliability.
With the increasing volume of published IR research, it may sometimes be
easy to concentrate on positive findings to the relative exclusion of negative ones.
It is often only at the level of holistic reviews that the extent and importance of
what we do not know are given prominence. Within the field of IR system-user
interactions, for example, we cannot yet claim to have generated any general
model of the ways in which users interact with systems which is at the same time
reliable, ecologically valid and practically useful.
Although an increasing range of research approaches is being used within IR
research and information science more generally [1, 2], it is still possible to per-
ceive two broad (by no means mutually exclusive) groupings which may be
described as essentially qualitative and quantitative [3]. Our current models are
limited by the fragmentary nature of much of our relatively narrow quantitative-
ly-derived knowledge, and by the objectively untested nature of much of our
more holistic qualitatively-derived knowledge. Indeed, Ford [3, p. 1151] has
drawn attention to the dilemma posed by these differences:
Overly-analytic states of knowledge are characterised by fragmentation –
at worst, isolated facts lacking integration into any coherent wider con-
ceptual picture. … As a result, much research in information science has
arguably provided highly reliable answers to highly meaningless questions.
The take-up of qualitative research approaches is now widespread in user-
oriented research. But without critical interaction with complementary
perspectives the increasing use of subjective analysis of introspections
using small samples of information users threatens to supply highly mean-
ingful questions with highly unreliable answers. Some balance and inte-
gration must be achieved between the two extremes.
The limitations associated with research may be thought of as a curtain pre-
venting us from viewing the reality beyond, that we seek to understand. Our exist-
ing knowledge ranges between two extremes, which to some extent mirror the
different research approaches discussed above. One may be characterised as scat-
tered pinpricks in the curtain, allowing clear and deep, but narrow and uncon-
nected views through to the reality beyond. The other may be characterised
as more extensive areas where the curtain is thinned, allowing complex, inter-
connected but hazy shapes to show through, inviting us to trace them onto the
curtain, elaborating their detail to represent what we imagine to be their reality.
It is arguably all too easy to ignore the extent of curtain and our consequent
lack of clear view, owing to a variety of (sometimes unconscious) forms of
6
25
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
over-optimism as to how securely and how widely we can generalise research
findings to form a reliable and relevant picture. Such forms of over-optimism
include, for example, the false equation of over-simplified research contexts with
more complex reality, and confusing internal coherence and plausibility with gen-
eralisability. Falsely equating constrained research contexts with more complex
reality is a form of over-generalisation most often associated with quantitative
research in which experimental control and accurate measurement of variables
are paramount. The danger is in assuming that what is observed in experimental
conditions also applies in non-experimental, more natural conditions [4–6].
The limitations of our knowledge relating to human-system interaction are
revealed when we consider the extent to which we are able to answer what to the
system developer and user are fundamental basic questions such as: how can we
help people search more effectively? What empirical research there is into users
tells us very little about how to design and improve systems. Complex IR systems
have been and are being built – but they are often based on relatively untested theo-
retical assumptions about users, not on any valid and robust empirically-based
understanding of them. Developments in IR system design increase strategic
options open to the user – but tell us nothing of how such options may be deployed
to achieve optimally effective searching.
Our desire to develop practically useful knowledge may encourage us to be
over-optimistic in relation to the generalisability of what are essentially very spo-
radic research efforts. This over-optimism may mask the scale of the darkness to
light ratio and the fact that we are falling very short in terms of the critical mass
required to support the generalisable knowledge we seek.
2
.
THE NEED FOR INTEGRATION
Fragmentation and isolation run counter to the process of generating a coherent
and generalisable knowledge map. Yet neither IR nor its parent information sci-
ence has been noted for its avoidance of such obstacles. More than twenty years
ago, Norman Roberts [7, p. 86] noted in relation to research in information
science generally:
Research is synonymous with narrow and compartmentalised forms of
study … separate research projects fail to convey the impression of con-
tributing to a common pool of ideas shaped by a consensus of professional
imperatives. … The current assumption appears to be that once a piece of
research is completed then it is completed forever. The consequence is that
much research is characterised by its static, snapshot, quality and, as such,
quickly relegated into irrelevance and history by the flux of circumstance.
More recent comments indicate the stubborn nature of the problem. Olaisen
[8, p. 260], for example, attributed fragmentation to a paradigmatic imbalance:
…
information science thought has been imprisoned by the dominant
quantitative empirical metaphors which have drawn the attention to some
quantitative phenomena while neglecting other more qualitative phenom-
ena. The result is a cumulation of trivial findings.
6
26
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
More specifically, Spink and Greisdorf [9, p. 272] have also drawn attention to the
lack of integration of different strands within information retrieval research:
A major problem for user-oriented researchers has been the lack of impact
of their research on actual IR system design as the systems and user-oriented
research has been operating largely on separate and unconnected tracks.
This view is echoed by Saracevic [10, p. 21]:
The mantra of [human-centered] … research is that the results have impli-
cations for system design and practice. Unfortunately, in most user-centered
research beyond suggestions, concrete design solutions were not delivered.
On the other hand, the systems side by and large ignores the user side and
user studies, and is even often completely ignorant of them.
Recent years have seen an increase in the adoption of qualitative and combined
research approaches. However, with the expansion of our research focus to reflect
the widespread availability and take-up of information retrieval tools by the vast
and heterogeneous Internet user population, and with calls to extend the scope of
IR research to take in contexts affecting it and in which it operates [1], there is a
need further to improve levels of integration. ‘Integration’ is used here to signify
maximising the cumulative evidential contribution of individual research studies
to an emerging coherent wider picture.
Such integration can be thought of as relatively vertical or horizontal. Both ver-
tical and horizontal aspects are necessary and complementary, either without the
other representing an essentially pathological research state. Based on a review of
cognitive research, Ford [3] has proposed a model in which understanding within
a field of research progresses via the building up of what Pask [11, 12] has termed
descriptions and procedures. These complement each other, and both are neces-
sary in the building of understanding. As Ford [3, p. 1146] noted:
Description building entails the construction of an overall conceptual map
–
a description of what may be known in a subject area. Operation learn-
ing relates to mastering procedural details – evidence and logical arguments
supporting for the larger picture. … [T]he growth of understanding in
a research area progresses via the building of descriptions and procedures.
Description-building entails … activities characterised by a relatively global
approach employing analogy recognition and relatively ambitious, tentative
concept development. Procedure-building entails … activities characterised
by logical, sequential and analytic progress establishing relatively well
defined concepts and building on them in a relatively sure, step by step way.
In terms of the integration of research processes and outputs, procedure building
represents relatively vertical integration, description-building relatively horizon-
tal integration, in the following sense. Procedure building entails deriving under-
standing of superordinate concepts from understanding of their subordinate
component concepts. To this extent this type of understanding may be thought
of as consisting of relatively vertical relationships. Description building, on
the other hand, entails other, non-derivative, relationships between concepts
including, for example, similarity, difference, illustration and elaboration. These
6
27
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
relationships represent relatively horizontal relationships in that they are not hier-
archically derived. In extreme form, they are characterised in terms of what de
Bono [13] has termed ‘lateral’ thinking.
To take an engineering analogy – when we build a bridge, we need both verti-
cal strength, deriving from the ability of the building material to be compacted,
and relatively horizontal strength, deriving from more complex multi-directional
interactions between structural components. In the case of a simple curved stone
bridge there comes a point when the essentially vertical stresses of the supporting
sides transfer to the horizontal stresses of the span. Whilst the vertical sides may
be constructed in isolation from one another, the placing of the horizontal stones
at the apex of the bridge requires a more holistic approach requiring interaction
between both sides.
Arguably, building the vertical pillars – in Pask’s terms ‘procedure-building’ –
has been the predominant focus of IR research, and indeed on balance within
information science more generally to date [1, 3, 4, 10, 14]. Interestingly, research
findings from cognitive science [3, 11, 12, 15] indicate that concentrating on ver-
tical procedure building at the expense of relatively horizontal description build-
ing characteristically results in fragmented knowledge. Conversely, emphasising
description building at the expense of procedure building results in unjustified
over-generalisation.
Since intellectual as well as physical resources are finite, depth is often bought
at the expense of breadth – and vice versa. Within this context, commitment to
particular research approaches may often centre on decisions as to how deep is
‘deep enough’. As Popper [16, p. 111] noted:
…
if we stop driving the piles deeper, it is not because we have reached
firm ground. We simply stop when we are satisfied that the piles are firm
enough to carry the structure, at least for the time being.
The point at which the depth of piles is sufficient to support a given structure is
likely to remain hotly disputed by researchers of different persuasions. Such per-
suasions may derive from experience in researching particular types of problem
to which particular approaches are best suited. However, it is also possible that
researchers may sometimes over-generalise their preferences on the basis of fac-
tors which do not derive exclusively from an objective mapping of approach onto
type of problem. As Marton and Svensson [17, p. 484] noted in relation to
research approaches:
What we can see from one point of view we cannot see from another. …
With one kind of observation certain aspects become visible: with another
kind of observation we see something else. We cannot arrive at a procedure
of observation which makes all the various aspects visible simultaneously.
Such differences may also to some extent help to explain the lack of integration
between human- and system-centred strands of IR research. Clearly, the IR com-
munity does engage in horizontal as well as vertical integration – for example in
the form of many excellent holistic reviews published in books, journals and annu-
al reviews. However, whilst conceding that such reviews do appear in a range of
publications, it must be noted that they are relatively thin on the ground. Although
6
28
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
certain publications are entirely devoted to reviews, treatment of a given subject
area may be infrequent. In the field of IR evaluation, for example, more than
twenty years elapsed between the most recent ARIST review and its predecessor.
It is interesting to note that, when attention is paid to the overall horizontal and
vertical picture – for example in reviews such as Harter and Hert’s [1] ARIST
review of IR evaluation – the over-riding impression is arguably still one of a lack
of horizontal integration. In the case of the latter review, this has resulted inter
alia in calls to reformulate the very focus of IR research, and to devise new, in
particular integrative, approaches to what appear to be relatively intractable
research issues. The resulting picture – particularly in relation to non-derivative
relatively horizontal concept relationships – is of a desperately high darkness to
light ratio. Indeed, when viewed from perspectives other than the highly vertical,
the darkness to light ratio is so great that there is an urgent need to redress the bal-
ance. We need to improve what have been referred to here as relatively horizon-
tal aspects of our knowledge of IR.
The development of an emerging knowledge map integrating both horizontal
and vertical aspects of understanding is dependent not only on empirical research
processes, but also crucially on patterns of access to the products of these
research processes – in the form of research data and results. The extent to which,
and the form in which, these are made accessible may significantly influence the
questions that can be asked of them by researchers engaged in secondary analysis
(for example, systematic reviews and meta-analyses). More pervasively however,
patterns of access to the products of research may also significantly affect the
nature of the research community’s foundation knowledge on which new research
ideas and proposals are built. The following four sections explore these issues in
more detail, discussing strategies by which we may be able to decrease fragmen-
tation. These strategies focus on attempting to improve the contribution of indi-
vidual studies to the global IR knowledge map via:
●
a greater use of methodological pluralism to integrate research
approaches;
●
greater access to research data, to enable researchers to ask more ‘hori-
zontal’ questions (i.e. questions more diverse than necessarily addressed
by the original researchers);
●
●
more multidisciplinary and multi-perspective integrative reviews and
conceptual mappings;
a greater critical mass of published research findings sufficient to sup-
port the generation of a less sparse and fragmented evidence-based
knowledge map.
3
.
GREATER USE OF METHODOLOGICAL PLURALISM
Methodological pluralism – in the sense here of the integration of qualitative and
quantitative research approaches – may be useful in the cumulative creation of an
integrated conceptual map, in that the application of multiple approaches allows
us to shed light on a phenomenon from different vantage points, thus bringing
into relief its different facets. As Entwistle and Hounsell [18, p. 361] noted:
6
29
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
The two paradigms … contain the tension of opposites – a thesis and
antithesis out of which a fruitful synthesis might be anticipated, but is still
far from being achieved. … Yet the methodologies of competing para-
digms could be used alongside one another, each providing distinctive yet
equally valid types of evidence.
Unfortunately in the field of study linking users and IR systems, we are arguably
still far from such a synthesis. Yet the two extremes of qualitative and quantitative
research approaches have much in common as well as differences. Qualitative
researchers are concerned with the generalisability of their findings as are quanti-
tative researchers – though the degree of generalisability, and the nature of evi-
dence deemed necessary to support it, may differ considerably. The point of
dispute between proponents of different research approaches often relates merely
to the starting point of such understandings, and the nature and extent of uncer-
tainty deemed sufficiently acceptable to allow understanding to be generalised to
other contexts.
For example, qualitative researchers may claim the value of understanding
even a single case study. However, this is not to deny that such value derives from
the capacity of that understanding to be generalised. To claim ‘understanding’ of
anything – including illuminative, phenomenological understanding of a single
case study – is to claim a degree of generalisability. The alternative to ‘under-
standing’ in cognitive terms is the reproduction of undigested data. The very act
of condensing individual pieces of data into fewer data items entails the applica-
tion of some underlying interpretative pattern which represents an attempt to
impose or test some more fundamental integrating concept. This is not to deny the
value of non-cognitive responses to phenomena – for example emotional respons-
es such as empathy, pleasure or admiration. Indeed, ‘understanding’ is often used
loosely to include general, not necessarily articulated, feelings of empathy
or agreement. However, it is used here to denote a cognitive state capable of
explicit articulation. As such, it implies a level of generalisation.
However, the way in which different research approaches may be deployed in
relation to one another may vary. Strategies for methodological pluralism include:
●
critical dialogue relating to a common phenomenon from different
paradigmatic perspectives [19];
●
●
the use of different paradigms within a single study [20];
the mapping of different paradigms onto different kinds of problem
[14];
●
methodological relativism, in which any paradigm is considered to be
as good as any other – drawing on Feyerabend [21].
The first and second strategies (which, of course, are not mutually exclusive) are
of particular value in the development of a coherent integrated picture of IR in
that IR is an area entailing study of the interaction of people and systems. As
such, it is well served by (indeed arguably requires) a degree of methodological
pluralism. The third strategy maps well onto specific individual components of
the field. The fourth strategy will be discussed in Section 7, which discusses the
epistemological basis of the present paper.
6
30
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
Just as funding agencies can encourage multidisciplinary research via specific
initiatives and refereeing criteria, so ‘pluralistic research’ and ‘integrative review’
initiatives could be further encouraged. However, methodological pluralism is by
no means universally welcomed. Burrell and Morgan [22], for example, suggest
that: ‘Contrary to the widely held belief that synthesis and mediation between
paradigms is what is required, we argue that the real need is for paradigmatic
closure’. There are certainly situations in which methodological singularity
makes sense – for example cases where:
●
the particular focus of investigation is more suited to one rather than
another research approach;
●
particular researchers or research groups are committed to a particular
approach over others in the sense that they believe it generally to pro-
vide superior evidential quality;
●
●
particular researchers or research groups wish to develop or extend
specialist expertise with particular approaches;
the perspectives afforded by different approaches are perceived as simply
too far apart to meet within the bounds of a particular research focus.
4
.
GREATER ACCESS TO RESEARCH DATA
The development of a coherent knowledge map is also dependent on research in
the form of secondary analyses and critical reviews. This and the following two
sections discuss the nature of this contribution. In each, it is argued that electronic
publication and data access have important roles to play.
Central to the process of integrating research outputs is subjecting them to crit-
ical scrutiny by the research community in order not only to explore their internal
properties (their accuracy and validity), but also their external properties (the
extent to which they fit into and contribute – positively or negatively – to the
emerging wider picture). Such explorations may take a variety of forms, includ-
ing peer review, systematic reviews, secondary and meta-analyses. Peer review
entails judging the suitability for publication or presentation of research papers
submitted to journals or conferences.
Systematic reviews entail reviewing a body of evidence in order to evaluate
and present current thinking relating to a research question [23–25]. Secondary
analysis entails re-analysing existing data (gathered as part of a prior study) in
order to generate an alternative perspective on a research question, or to pursue a
new research question. It has been applied to quantitative and qualitative data
[26–28]. Meta-analysis techniques [29, 30] have also been developed for qualita-
tive as well as quantitative data. Meta-analysis entails combining datasets from
distinct studies in order to increase the sample size in the case of quantitative
analyses [31, 32]. Meta-analysis of qualitative data also entails using data from
distinct studies in order to generate integrated findings across the studies [33, 34].
Such reviews and analyses require that sufficient detail of individual research
studies be rendered accessible in order to enable:
●
in the case of secondary- and meta-analyses, re-working of the data to
enable researchers to ask questions of the data which may differ from
6
31
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
those asked (at least in published accessible outputs) by the original
researchers. This will require access to original datasets and details of
research procedures and instruments;
●
the mapping of the research onto the emerging wider picture, for exam-
ple via systematic reviews (again requiring access to original data and
other research details, where such reviews entail asking questions not
directly addressed by the original researchers in their papers).
The potential for such horizontal activity is greater than ever before. As Floridi
35, pp. 267–268] noted:
[
…
the book era, providing a rigidly structured context, invited vertical spe-
cialisation. … The electronic encyclopaedia [i.e. the global encyclopaedia
of human knowledge], on the other hand, now promotes forms of hori-
zontal inter-disciplinarity and multi-disciplinarity which will probably start
bearing fruit in the next decade. The more flexible the space of knowledge
becomes, the more ‘diatopic’ our approach to it can be. It is difficult to
limit oneself always to the same confined space, if one can navigate so
easily across the disciplinary boundaries.
The rigid original organisation of print-based datasets, established by their creators,
determined what questions could be asked of the data to a far greater extent than is
the case with electronic datasets. Floridi [35, p. 268] went on to comment that:
It is now possible to query the digital domain and shape it according to
principles which are completely different from those whereby the primary
data were initially collected and organized. The structure of our particular
set of digital data can be modified to fit an infinite number of require-
ments, and hence provide answers to secondary questions which were not
meant to be answered by the original structure.
However, as discussed below, current access to research findings arguably falls
short of such requirements. The result is that the vast resource constituted by orig-
inal datasets produced by large numbers of researchers is underexploited. This is
understandable in a paper-based context. However, it becomes less justifiable with
the development of electronic forms of information storage and communication.
Paper-based journal and conference documents generally represent the tip of
the data iceberg. What is omitted – through concerns which are legitimate in the
context of ‘print on paper’ publication (e.g. conciseness and distillation) – may
often be the level of detail required to enable other researchers clearly to map the
published findings onto conceptual maps and research agendas which may be
broader than, and/or different from, those of the original authors. With limited
space, authors and editors may often also wish to emphasise positive findings,
eschewing the failures and negative results. However, these may, in the context of
mapping out the broader emerging picture and of the research agendas of other
researchers, be of great interest and relevance [36]. In the field of medicine, for
example, Smith and Egger [37] argued that:
The dissemination of failed reviews is an important task, which is
neglected by traditional journals. … This is increasingly being taken on by
6
32
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
the Cochrane Collaboration, along with the dissemination of many other,
conclusive reviews.
Print is hardly a suitable medium for giving access to large sets of machine-
manipulable data. However, checking, analysis and re-analysis often require not
only access to full datasets, but also to code books, detailed information about
research instruments and study design, and other voluminous documentation.
The notion of electronic data archiving and data sharing is by no means a new
phenomenon. Indeed, Ginsparg [38] noted in 1996 that the e-archives in some
fields of physics had: ‘already supplanted traditional research journals as convey-
ors of both topical and archival research information’. The archiving of data
allowing networked access by researchers is relatively well established in a vari-
ety of disciplines [39–41]. Such archives often contain data gathered as part of
large scale studies, and represent centralised, highly selective and quality con-
trolled resources. They may consist of qualitative as well as quantitative data [42].
However, the Internet provides an infrastructure for widespread distributed
access, to datasets produced by large numbers of research individuals and groups,
on a less formal basis. Such access represents data sharing as opposed to data
archiving. Archiving entails a range of proactive functions geared to preserving
access to stored data despite changes in hardware, software and standards, and
ensuring security [43]. However, much can be achieved via relatively informal
data sharing.
Increasingly, instructional texts are published with accompanying web sites. It
would be useful if research papers could link to web sites allowing access to full
data and other research documentation. An increasing number of journals are
offering searchable electronic archives of past issues. Less common is the provi-
sion of access to data and other research documentation. Clearly, there is nothing
to stop authors of print on paper articles from publishing web addresses linked to
original data and other research documentation. However, a number of journals,
for example the Journal of Digital Information [44], plan themselves to provide:
support for interactive data manipulation and comparative analysis, …
interlinking with other quality sources from e-print archives to other jour-
nals and Web databases. … [offering authors] plenty of scope to use more
of your data in its native form.
Similar moves are taking place within information science with the development
of JASISWeb. As Brooks [45, p. 1031] noted:
JASISWeb … would include journal articles … as well as background
datasets (e.g. TREC datasets, results sets, etc.), examples of working
algorithms, software programs illustrating classic information science
theorems …, chat rooms with noted information scientists, and classified
sets of links to Web sites of interest to information scientists.
However, it is by no means standard procedure for researchers to make original
data freely available to the research community (except by explicit collaborative
agreements between consenting researchers). Rockwell [43] noted some extreme
cases in the social sciences generally:
6
33
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
In some of the more famous cases of data that are closely held, one must
join the research team in order to have access to the data. Often, ‘joining
the research team’ means bringing financial support to the project. There
have also been a few cases in which the data collector somehow believed
that it was impossible to do a correct analysis of the data without his or
her direct supervision, and sometimes that it was impossible to do a cor-
rect analysis more than ten miles from the data collector’s campus.
IR research is not devoid of similar, if not as extreme, criticisms. As Saracevic
[10, p. 24], for example, noted:
Since it started in 1992, TREC resulted on one hand in a large number of
technical reports, but on the other hand, only in a surprisingly small
number of articles or other publications in peer reviewed literature.
Consequently, TREC results are more or less limited to TREC partici-
pants and to those in the know. The results have not trickled out. They are
not widely known in information science, and even less outside of it.
TREC results are not being used or evaluated outside of TREC. It is
almost as if TREC is there for the sake of TREC.
Indeed, a number of factors militate against individuals and research groups offer-
ing the wider community access to research data and documentation. These
include competitive research environments, reluctance freely to open data and
methods to public and possibly hostile critical scrutiny, and ethical considerations.
5
.
MULTI-PURPOSE REVIEWS AND CONCEPTUAL MAPPINGS
Reviews have an important contribution to make to the development of an inte-
grated knowledge map. The ARIST model is excellent, entailing reviews by spe-
cialist information researchers for specialist information researchers. However,
specialist reviews such as those provided in ARIST are unlikely to be readily
available to researchers other than information specialists unless they belong to
institutions with specific LIS research and/or teaching interests. Just as increas-
ingly IR researchers find it useful to draw upon cognate areas of research, so
increasingly IR literature has a potentially important contribution for researchers
other than IR specialists. However, even when physically accessible, there may
be intellectual barriers to accessibility insofar as reviews produced by specialist
researchers for specialist researchers may address different agendas, and commu-
nicate assuming different types and levels of knowledge, from those with which
researchers and practitioners from other fields approach the research literature.
Researchers from a range of different disciplines need to be able to access IR
research in ways which will enable its use in relation to their own problems and
perspectives. They need to be able to find relevant IR research findings in
response to searches formulated in terms of their own particular problems and
perspectives. Such multi- and inter-disciplinary searches would benefit from
a range of ontologies designed to map such requests onto more specialised IR
terminology. However, as well as a range of heterogeneous reviews and onto-
logical mappings of search terminologies, we also need multi-perspective and
6
34
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
multi-purpose conceptual mappings more complex than those provided by
ontologies. Such mappings could include:
●
negative mappings – namely, what Smith [46] has termed ‘open prob-
lem’ papers charting what we do not currently but would particularly
like to know;
●
hyperlinked reviews enabling readers to engage in vertical and hori-
zontal navigation of both positive and negative aspects of our knowl-
edge of IR.
The latter type of review could enable the reader to drill down vertically and
move horizontally to access further mappings, annotations, documents and ulti-
mately raw data. Some form of ‘authority navigation’ would be needed to over-
come the fact that, as with the ‘raw’ web (as opposed to sites offering high quality,
critically evaluated sources) at present, anyone could attach material (papers, data
and annotations) including the naive and erroneous. Such a system is predicated
on expanded free access to papers and data of the sort discussed in Section 6.
This model is in conflict with the view of writers such as Smith [46] who,
rather than advocating more reviews, suggests a more vertical approach whereby
reviews would be excluded from JASIS on the grounds that they are covered by
ARIST. However, it must be acknowledged that she does recommend increasing
emphasis on ‘open problem’ papers. We need plenty of secondary and tertiary
mapping activity to chart the territory – to map out hooks onto which we need to
hang evidence.
There is also scope here for innovative approaches such as the machine sup-
port for scientific discovery across cognate but distinct subject areas described by
Valdes-Perez and Swanson’s computerised techniques for teasing out knowledge
that may be inferred from the interaction of complementary but previously non-
interactive literatures. As Swanson and Smalheiser [47, pp. 183, 201] noted:
An unintended consequence of specialization in science is poor commu-
nication across specialities. … Bringing together two complementary but
noninteractive literatures from different specialized areas of research can
reveal undiscovered public knowledge – unnoticed implicit relationships
not apparent in the two literatures considered separately.
They went on to describe interactive software and database search strategies that
support this process. Valdes-Perez [48, 49] described work to evaluate machine
learning systems across a range of disciplines, and attempted to distil general
techniques capable of supporting scientific discovery.
6
.
CRITICAL MASS
In order to generate coherence and plausibility, researchers may often be forced
to make big, possibly unjustified, assumptions – especially when a relatively
extreme vertical view is taken (i.e. one with an extremely narrow and specialist
focus), and when human as well as system aspects are concerned [5, 6]. Sarajevic
[10, p. 17], for example, noted that:
6
35
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
We can consider IR as a branch of computer science. … But in doing so
we … make and accept without questioning some HUGE assumptions,
particularly about users. In fact we assume users and everything that goes
with them, such as ‘meaning,’ ‘understanding,’ ‘relevance’, and a host of
others. We avoid dealing with them.
The result is an empty response to system designers’ appeals to human-centred IR
researchers for guidance [10, p. 21]: ‘As to design, the stance is “tell us what to
do and we will”. But nobody is telling’.
The problem is that human-centred IR research has nothing much to tell them
(as opposed to suggesting tentative hypotheses), because translations of such
research into practical guidelines are so infused with assumption that they have
little predictive power outside the specific and often artificially constrained con-
text in which the data were obtained.
When we view human-related, as opposed to purely system-related, aspects of
IR theories and models from a more critical, less assuming perspective, arguably
the darkness to light ratio is so great – and decreasing so slowly – that urgent action
is required. We desperately need to replace assumptions (based on desire – for
coherence and integration) in our models and theories with scientifically based
hypotheses, and to replace these hypotheses with evidence (based on data).
However, to do this we need vast quantities of research. We are currently trying to
build edifices with finely worked marble – but we also need vast quantities of stone.
In this context, quality (beyond a certain basic level) at the expense of quanti-
ty represents a pathological research state in which it will take us too long to get
anywhere meaningful. Whilst acknowledging that an even less desirable patho-
logical state is quantity without quality (since the likely destination of this
approach will not be worth arriving at), arguably we have an imbalance toward
the first pathology. However, the present paper is not advocating the latter. Rather,
we need to avoid both by attempting to ensure an optimal blend of quality with
quantity. It is important to note that there is no reason to suppose that IR is alone
in exhibiting such a pathological research state. An overly analytic and frag-
mented approach arguably also characterises other research fields – especially
those entailing the study of people interacting with systems.
A number of factors relating to the publishing of research data and results have
the effect of promoting the pathological research state which, it is argued here,
characterises much past and current IR research. Researchers are often under
pressure to publish in what are held to be top rated journals in their field. In the
case of library and information science many of these journals – even if offering
electronic access – adopt the traditional approach of relatively slow, highly selec-
tive quality control, expensive profit-motivated subscription, and restrictive copy-
right practice in relation to authors, with unwillingness to countenance web-based
pre-publication of papers. There are, however, an increasing number of journals
which, whilst still offering high quality refereed papers, do not fit this mould –
in our own field, for example, D-Lib [50] and Information Research [51].
Whilst students and newcomers to the field may benefit from access to research
outputs that are highly filtered and quality controlled, many experienced
researchers may find it patronising if, and to the extent that, it inhibits alternative,
less restrictive, forms of access. This is not to deny the undoubted value of quality
6
36
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
control by recognised authority figures in the field [52]. Peer reviewing is an inter-
active process whereby papers may be greatly improved as a result of referees’
comments. But the value to be derived from such a process is multifaceted and
variable. Students in an area of research may benefit greatly from one type of qual-
ity filter. Experienced researchers coming from other fields wanting to integrate
research findings into multidisciplinary research may benefit from other types of
quality filter.
Yet the quality criteria adopted by many journal editors via their reviewers are
not necessarily multipurpose. Rather, we need a variety of quality filters aimed at
a variety of needs, and produced from a variety of perspectives. As soon as phys-
ical space becomes a less significant constraint, then arguably other models
become viable, including the notion of multi-level quality control. Quality filter-
ing via peer review is clearly a valuable procedure – for certain purposes.
However, it is not mutually exclusive to the notion of multi-level quality control,
which is valuable for other purposes.
A number of writers have addressed this issue, some proposing radically dif-
ferent models of review and publishing. In Varian’s model [53], for example,
authors would, in addition to the normal abstract and full paper, submit a five-
page summary of their paper. Only this summary would be reviewed by the jour-
nal’s editorial board – and this review would focus only on level of interest as
opposed to accuracy. Readers would rate papers online for interest, and only those
papers reaching a threshold would progress to being reviewed for accuracy.
The present author takes a position which differs from that of Varian in that
although still arguing for the notion of ‘good enough’ reviewing, this would relate
to soundness as opposed to level of interest. ‘Soundness’ is defined here as the
degree to which the research presented is free of basic errors (for example relat-
ing to sampling, data analysis techniques used, etc.), and uses variables which are
valid and unambiguously defined. In this view, publishing of papers which are
judged to be sound, but not necessarily well presented, important, or even of great
interest in the view of the journal’s reviewers, should be greatly expanded to
complement existing traditional publishing.
In a multi-level quality control system, at the lowest level, unrefereed papers
could be published voluminously, subject to basic quality control (i.e. being of
‘good enough’ quality as defined above) to filter out patently bad research.
Referees’ comments, and progressive revisions of the paper, could all appear as
they are produced without delay. This may be particularly instructive to novice
researchers and authors. Provision would also need to be made to index papers by
quality level, to enable those readers who wished to search for only fully refereed
papers, as in the traditional journal model. One might envisage further levels of
quality control between the two so far described, in which critical commentary
from readers would be linked to versions of papers.
A number of electronic journals are moving in this direction. Information
Research is an electronic journal which publishes both fully refereed papers and
unrefereed working papers. The working papers make research findings rapidly
available – for example, initial results from an on-going PhD study. Clearly,
despite such papers not being subjected to full refereeing, editorial selection does
entail a level of quality control. Plans for JASISWeb are closer to the idea of multi-
level, as opposed to single or dual, quality control [45, p. 1031]:
6
37
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
Each ‘article’ in JASISWeb should be a separate address where one could
find earlier versions, author reflections, reader comments, author responses,
and so forth. The accumulation of traffic at the site of an ‘article’ would
indicate its importance.
Software to support publishing of this type, entailing progressive critiquing and
discussion, is freely available from the Open University’s Knowledge Media
Institute in the form of their D3E (Digital Document Discourse Environment)
system [54], along with examples of its use [55, 56].
7
.
EPISTEMOLOGICAL ISSUES
This paper has presented criticisms of the state of IR research and has suggested
strategies by which they may be addressed. But on what evidence does the author
justify the criticisms and the strategies suggested to ameliorate them? Why should
we think, as proposed here, that progress would somehow be better if we were to
adopt different approaches? Even if it is indeed true that IR research lacks inte-
gration, as argued in this paper, this may be the necessary price to be paid for
obtaining scientifically sound knowledge.
The basic epistemological stance on which the nature of both its criticisms and
suggested strategies is based is broadly in line with what Dick [57] has termed
‘holistic perspectivism’. Based on his doctoral research [58], he draws attention
to the plurality of possible epistemological perspectives relevant to library and
information research and practice, and to the eccentric swings into and out of
vogue of particular perspectives – sometimes accompanied by ‘paradigm hos-
tility’ [3, 59, 60].
Dick argued the value of what he refers to as ‘perspectivism’, essentially a
meta-epistemological stance capable of accommodating both individual perspec-
tives and the interaction and integration of multiple perspectives. ‘Holistic’, as
used by Dick, relates to the focus of research, encompassing both micro-elements
(for example, the individual) and complex wholes formed by the interaction
between multiple micro-elements (for example, different user groups). The pre-
sent paper advocates increased levels of integration, both of research approaches
(
(
qualitative and quantitative) offering different perspectives, and of different foci
users and systems), based on the critical view that such integration is lacking in
IR research. Both criticism and envisaged strategic solutions thus implicitly assert
the value of a holistic perspectivist approach.
There would seem to be two possible sets of grounds on which to attempt a jus-
tification for such an approach. The first, and in the author’s view weaker, repre-
sents a position of extreme logical relativism, based on the views of Feyerabend
[21]. This view posits that we simply have no logical way of assessing the rela-
tive merits of different views, it being impossible to prove that any one is better
than any other. The implication of this view is that the position advocated in this
paper is likely to be as good as any other. If this is so, then it has as much chance
as others of generating benefits. Insofar as the integrative approaches advocated
in this paper have not been widely adopted, their expansion may generate fresh
approaches, findings and solutions. Equally, however, this may not be the case,
time being the only arbiter.
6
38
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
The second position is stronger and more specific in its claimed value, but
weaker logically, being based on reasoning by analogy. However, to some, a posi-
tion based on a compelling analogy (if they find it so) may have more appeal than
the somewhat hapless position predicated on extreme logical relativism. Ford [3]
argued that to a significant degree there are analogous relationships between the
growth of understanding in the individual as studied in cognitive science, and in
a research field such as IR.
Research into human cognition suggests that high levels of understanding
require the integration of relatively holistic, and narrower analytic, perspectives.
Emphasising one or the other perspective up to a point represents cognitive style
offering differing but equally effective routes to high levels of understanding.
However, beyond that point, emphasis on one perspective to the relative exclu-
sion of the other represents a learning pathology resulting in inferior levels of
understanding. In the context of the present paper, the implication is that under-
standing that is based on a greater integration of horizontal and vertical, system-
centred and human-centred research is essentially a less pathological research
state than that in which we currently find ourselves.
However, reasoning by analogy in this way is intrinsically a more horizontal
than vertical intellectual activity, entailing relatively holistic and intuitive, as
opposed to analytic and logical mental processes [61, 62]. The problem is, then,
that the argument presented here for a greater integration of research evidence
cannot itself call upon an integrated base of supporting research evidence.
Viewed negatively, the argument is thus essentially self-referential and as
such, must remain mere polemic until supported by the very type of vertical (ana-
lytical and logical) research evidence to which, in view of its horizontal (holistic
and intuitive) nature, it is particularly unsuited. Viewed more positively, however,
it represents what Olaisen refers to as a ‘sensitising’ as opposed to a ‘definitive’
concept. Sensitising concepts emphasise relatively intuitive as opposed to more
logical analysis. These relatively tentative and speculative concepts:
…
offer a general sense of what is relevant and will allow us to approach
flexibility in a shifting, empirical world to ‘feel out’ and ‘pick one’s way’ in
an unknown terrain. … In sum, the on-going refinement, formulation, and
communication of sensitizing concepts must inevitably be the building
block of our exploratory theory [8, p. 254].
REFERENCES
1
.
Harter, S.P. and Hert, C.A. Evaluation of information retrieval systems:
approaches, issues, and methods. Annual Review of Information Science
and Technology, 32, 1997, 3–94.
2
.
.
Powell, R.R. Recent trends in research: a methodological essay. Library and
Information Science Research, 21(1), 1999, 191–119.
Ford, N. The growth of understanding in information science: towards a
developmental model. Journal of the American Society for Information
Science, 50(12), 1999, 1141–1152.
3
6
39
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
4.
5.
6.
Ellis, D. The dilemma of measurement in information retrieval research.
Journal of the American Society for Information Science, 47(1), 1996,
3–36.
Landauer, T.K. Relations between cognitive psychology and computer sys-
tem design. In: Carroll, J.M., ed. Interfacing thought. Cambridge, Mass.:
MIT Press, 1987, 1–25.
Landauer, T.K. Let’s get real: a position paper on the role of cognitive psy-
chology in the design of humanly useful and usable systems. In: Carroll,
J.M., ed. Designing interaction: psychology at the human-computer inter-
face. Cambridge: Cambridge University Press, 1991, 60–73.
2
7
8
.
.
Roberts, N. Forum: comments on ‘The developing cognitive viewpoint in
information science’ – Brookes. Journal of Informatics, 2, 1978, 84–86.
Olaisen, J. Pluralism or positivistic trivialism: important trends in contempo-
rary philosophy of science. In: Nissen, H.R., Klein, H.K. and Hirschheim, R.,
eds. Information systems research: contemporary approaches and emergent
traditions. Amsterdam: Elsevier, 1991, 235–265.
9
.
Spink, A. and Greisdorf, H. Users’ partial relevance judgements during
online searching. Online and CDROM Review, 21(5), 1997, 271–280.
1
0. Saracevic, T. Users lost: reflections on the past, future, and limits of infor-
mation science. SIGIR Forum, 31(2), 1997, 16–27. (1997 ACM SIGIR
Gerard Salton Award for Excellence. Acceptance address.)
11. Pask, G. Final report of S.S.R.C. Research Programme HR 2708. Richmond,
Surrey: System Research Ltd., 1979.
1
2. Pask, G. Learning strategies, teaching strategies, and conceptual or learning
style. In: Schmeck, R.R., ed. Learning strategies and learning styles. New
York: Plenum Press, 1988, 83–99.
1
1
3. de Bono, E. Lateral thinking. In: Gregory, R.L., ed. The Oxford companion
to the mind. Oxford: Oxford University Press, 1987, 428–429.
4. Olaisen, J. Pluralism or positivistic trivialism: important trends in contem-
porary philosophy of science. In: Nissen, H.E., Klein, H.K. and Hirschheim,
R., eds. Information systems research: contemporary approaches and emer-
gent traditions. Amsterdam: Elsevier, 1991, 235–265.
1
1
1
1
1
2
2
2
5. Pask, G. Conversation theory: applications in education and epistemology.
Amsterdam, Oxford: Elsevier, 1976.
6. Popper, K. The logic of scientific discovery. New York: Harper and Row,
1
968.
7. Marton, F. and Svensson, L. Conceptions of research in student learning.
Higher Education, 8, 1979, 471–486.
8. Entwistle, N.J. and Hounsell, D. Student learning in its natural setting.
Higher Education, 8, 1979, 359–363.
9. Weaver, G.R. and Gioia, A.D. Paradigms lost: incommensurability versus
structurationist inquiry. Organisational Studies, 15(4), 1994, 565–589.
0. Wildemuth, B.M. Post-positive research: two examples of methodological
pluralism. Library Quarterly, 63(4), 1993, 450–468.
1. Feyerabend, P. Against method: outline of an anarchist theory of knowl-
edge. London: New Left Books, 1975.
2. Burrell, G. and Morgan, G. Sociological paradigms and organisational
analysis. London: Heinemann, 1979, 397–398.
6
40
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
2
2
3. Ohlsson, A. Systematic reviews – theory and practice. Scandinavian Journal
of Clinical and Laboratory Investigation – Supplement, 219, 1994, 25–32.
4. Popay, J., Rogers, A. and Williams, G. Rationale and standards for the
systematic review of qualitative literature in health services research.
Qualitative Health Research, 8(3), 1998, 329–340.
2
5. Sandberg, S. Systematic reviews of diagnostic tests – a new challenge for
laboratory medicine. Scandinavian Journal of Clinical and Laboratory
Investigation, 57(5), 1997, 369–372.
2
2
2
6. Grinnell, R.M. Social work research and evaluation: quantitative and qual-
itative approaches. 5th ed. Itasca, Ill.: Peacock, 1997.
7. Heaton, J. Secondary analysis of qualitative data. Social Research Update, 22,
1998. http://www.soc.surrey.ac.uk/sru/SRU22.html (visited 17 April 2000).
8. Hinds, P.S., Vogel, R.J. and Clarke-Steffen, L. The possibilities and pitfalls
of doing a secondary analysis of a qualitative data set. Qualitative Health
Research, 7(3), 1997, 408–424.
2
3
3
3
3
3
3
9. Smith, J.T. Meta-analysis: the librarian as a member of an interdisciplinary
research team. Library Trends, 45, 1996, 265–279.
0. Trahan, E. Applying meta-analysis to library and information science
research. Library Quarterly, 63, 1993, 73–91.
1. Hasselblad, V. and Hedges, L.V. Meta-analysis of screening and diagnostic
tests. Psychological Bulletin, 117, 1995, 167–178.
2. Irwig, L., Macaskill, P. and Glasziou, P. Meta-analytic methods for diag-
nostic test accuracy. Journal of Clinical Epidemiology, 48, 1995, 119–130.
3. Glass, G.V., McGaw, B. and Smith, M.L. Meta-analysis in social research.
Newbury Park, Calif.: Sage, 1981.
4. Noblit, G.W. and Hare, R.D. Meta-ethnography: synthesizing qualitative
studies. Newbury Park, Calif.: Sage, 1988.
5. Floridi, L. Internet: which future for organized knowledge, Frankenstein or
Pygmalion? International Journal of Human-Computer Studies, 43, 1995,
2
61–274.
3
6. Naylor, C.D. The case for failed meta-analyses. Journal of Clinical
Evaluation and Clinical Practice, 1, 1995, 127–130.
3
7. Smith, G.D. and Egger, M. Meta-analysis: unresolved issues and future
developments. British Medical Journal, 316(7126), 1998, 221–225. Also
available at: http://www.bmj.com/cgi/content/full/316/7126/221 (visited 17
April 2000).
3
8. Ginsparg, P. Winners and losers in the global research village electronic
publishing in science. Invited contribution for Conference held at UNESCO
HQ, Paris, 19–23 Feb 1996, during session Scientist’s View of Electronic
Publishing and Issues Raised, Wed 21 Feb 1996. http://xxx.lanl.gov/blurb/
pg96unesco.html (visited 17 April 2000).
3
9. Mochmann, E. Data access & conditions. European Cooperation in
Social Science Data Dissemination. 1998. http://www.ifdo.org/archiving_
distribution/eudd_bfr.htm (visited 17 April 2000).
4
0. Radcliffe Institute for Advanced Study at Harvard. Murray Research
Centre. Mental Health Archive. 1999. http://www.radcliffe.edu/murray/
data/mentalhe.htm (visited 17 April 2000).
6
41
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
JOURNAL OF DOCUMENTATION
vol. 56, no. 6
4
4
1. Harvard University. Astrophysics Data System (ADS) Data Archives. 1999.
http://adswww.harvard.edu/ads_archives.html (visited 17 April 2000).
2. Corti, L. and Thompson, P. Are you sitting on your qualitative data?
Qualidata’s mission. International Journal of Social Research Methodology,
1(1), 1998, 85–89.
4
3. Rockwell, R.C. Data sharing [and archiving] for research, teaching and
development. The opening address prepared for the Workshop on Data
Sharing for Research Capacity Development, Pretoria, South Africa, 9 July
1998. http://www.icpsr.umich.edu/ICPSR/sada.html (visited 17 April 2000).
4
4
4
4. About the Journal of Digital Information. http://ojfpc.ecs.soton.ac.uk/
about.html (visited 17 April 2000).
5. Brooks, T.A. Postmodern information science and its ‘journal’. Journal of
the American Society for Information Science, 50(11), 1999, 1030–1031.
6. Smith, L.C. Journal of the American Society for Information Science
(JASIS): past, present and future. Journal of the American Society for
Information Science, 50(11), 1999, 965–969.
4
4
7. Swanson, D.R. and Smalheiser, N.R. An interactive system for finding
complementary literatures: a stimulus to scientific discovery. Artificial
Intelligence, 91, 1997, 183–203.
8. Valdes-Perez, R.E. Principles of human computer collaboration for knowl-
edge discovery in science. Artificial Intelligence, 107(2), 1999, 335–346.
Also available at: http://www.cs.cmu.edu/~sci-disc/Abstracts/aij99.html
(visited 17 April 2000).
4
9. Valdes-Perez, R.E. Discovery tools for science applications. Communications
of the ACM, 42(11), 1999, 37–41. Also available at: http://www.cs.cmu.edu/
~sci-disc/Abstracts/cacm99.html (visited 17 April 2000).
5
5
0. D-Lib Magazine. http://www.dlib.org/ (visited 17 April 2000).
1. Information Research. http://www.shef.ac.uk/~is/publications/infres/ircont.
html (visited 17 April 2000).
5
5
2. Harnad, S. The invisible hand of peer review. Exploit Interactive, 5, 2000.
http://www.exploit-lib.org/issue5/peer-review/ (visited 17 April 2000).
3. Varian, H.R. The future of electronic journals. Journal of Electronic
Publishing, 4(1), 1998. http://www.press.umich.edu/jep/04-01/varian.html
(visited 17 April 2000).
5
4. Sumner, T., Buckingham Shum, S.B., Wright, M. and Story, M. D3E: Digital
Document Discourse Environment. 1999. http://d3e.open.ac.uk/ (visited 17
April 2000).
5
5. Journal of Interactive Media in Education. http://www-jime.open.ac.uk/
(visited 17 April 2000).
5
6. Special Issue. World Wide Web Usability. International Journal of Human-
Computer Studies, 47(1), 1997, 1–222. Also available at: http://ijhcs.open.
ac.uk/ (visited 17 April 2000).
5
5
5
7. Dick, A.L. Epistemological positions and library and information science.
Library Quarterly, 63(3), 1999, 305–323.
8. Dick, A.L. Towards establishing an epistemological position for library and
information science. PhD dissertation. University of Cape Town, 1991.
9. Allen, D. and Ellis, D. Beyond paradigm closure in information systems
research: theoretical possibilities for pluralism. In: Proceedings of the 5th
6
42
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com

Journal of Documentation, Vol. 56, No. 6, November 2000
Aslib, The Association for Information Management.
All rights reserved. Except as otherwise permitted under the Copyright, Designs and Patents Act
©
1
988, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means, electronic, mechanical, photocopying or otherwise without the prior
written permission of the publisher.
November 2000
DARKNESS TO LIGHT
European Conference on Information Systems. University College Cork,
Ireland, June 19–21, 1997, 737–759.
6
6
6
0. Bradley, J. and Sutton, B. Reframing the paradigm debate. Library Quarterly,
63(4), 1993, 405–409.
1. Holyoak, K.J and Thagard, P. Mental leaps: analogy in creative thought.
Cambridge, Mass.: MIT Press, 1996.
2. Ford, N. Information retrieval and creativity: towards support for the orig-
inal thinker. Journal of Documentation, 55(5), 1999, 528–542.
(Revised version received 19 April 2000)
6
43
Aslib, The Association for Information Management
Staple Hall, Stone House Court, London EC3A 7PB
Tel: +44 (0) 20 7903 0000, Fax: +44 (0) 20 7903 0011
Email: pubs@aslib.com, WWW: http://www.aslib.com