Beyond citations

Part of every scientist’s job is to publish about their work. If you publish many papers in highly ranked academic journals and your papers are cited often, your academic impact is considered to be high. This makes you a well respected academic, and your academic impact also reflects well on your research group and the institution you work for. This traditional research evaluation model based on citations, impact factors and the H-index of researchers has been in place for a long time. However, since 2010 there are increasing calls for alternative ways to measure impact, which go beyond citations. Alenka Princic, Head of Research Support at the Library explains why. And Professor Maarten van Ham shares his experiences with publishing outside the academic world.

Alenka Princic

Alenka Princic

Alenka Princic: “It is an assumption that a high number of citations reflects the quality of your work, as others are clearly interested in what you are saying. The more publications and citations, the better the performance. However, this measure of impact factor is unfair, as there are many differences among research disciplines, e.g. in the number of journals to choose from, or the ‘citation culture’ within a particular field. Besides, there are more ways to evaluate your research output. Therefore a new, additional set of metrics was proposed by a project group called ImpactStory in 2012. They started to look at publications in a broader context. They provided a web-based tool to help researchers measure and share the impacts of all their research outputs – from traditional ones such as journal articles, to alternative research outputs such as blog posts, datasets, and software. Altmetrics in general capture various parts of ‘impact’ a paper or work can have. They refer to any communication outside the traditional academic publication. Think of performance on blogs, LinkedIn, videos, Wikipedia, newspapers, etc. Measuring your activity and the mentions on these communication channels may say something about your relevance to and engagement with the wider public. At, another provider of an altmetric tool, you can find a collection of data which includes attentions, i.e. citations to your work in social media, public policy documents, mainstream media and newspapers, and much more. It gives you a sense of who’s talking about your work.

Impact on society
However, what do all these mentions mean? And aren’t they equally difficult to compare as the differences between research fields? A number of challenges surround the use of altmetrics. Some subjects are simply more popular for a wider audience than others. There are also geographical and language limitations and the coverage is not well defined. The data that feed altmetrics are not yet transparent enough, so interpretation needs to be done with caution. Perhaps the most prominent challenge is the ease with which (alt)metrics-based evaluation systems can be gamed. There is still a long way to go. Some fear that altmetrics will introduce a new form of competition not based on scientific quality. There is positive potential though. Social media are all about interaction and engagement. Therefore the current agreement is that rather than the impact on scientific progress, altmetrics show the influence and engagement of a researcher beyond the borders of the academic world and the impact on society. It reflects the endeavours of a researcher to share knowledge, making science more accessible.

Accessibility and knowledge sharing go, of course, hand in hand with the principles of Open Science. There is an EU expert group that is looking into altmetrics as part of Open Science and how it can support disseminating knowledge beyond academic boarders . Institutions could stimulate their researchers to think beyond citations. The other way around, with altmetric data a researcher can provide extra information to the board of their research group. Moreover, a healthy online buzz can be an early indication that your paper is of wider interest. Citations through the traditional publications take a lot longer to appear. This could particularly help the early-career researcher, or bring to light the less cited disciplines.

Most institutions do not yet have protocols for making an impact with social media, but we are looking at what altmetrics could mean for scientists at TU Delft and how you can follow the echo of your particular output. Currently the added value of the Altmetric data is yet to be proven as the metrics remain dependent on publications.  Although the idea of measuring something else than citations-based impact is noble and exciting, it is precisely what major altmetric tools fail to do. Hopefully the future endeavours will focus on counting mentions of scientists in relevant media such as national newspapers or policy documents rather than solely counting mentions of the publications.”

Looking at the wider picture

Prof Maarten van Ham

Professor Maarten van Ham

Maarten van Ham, professor of Urban Renewal and Housing in the Faculty of Architecture and the Built Environment, is very much in favour of looking at the wider picture. “Citations and your H-index are still important within the academic world. You need them for funding, for example. But the reach of your work beyond the academic borders and its impact on society is equally important. Guest lectures, policy reports, non-scientific papers do not count in the traditional metrics and do not show up in web of science, but they can deliver influential insights to the public domain. I am proud to have written for an EU/UN report which resulted in questions in parliament. It shows that if your research can reach policy-makers, you can actually make a change.

Snowball effect
Another good example of the snowball effect that a non-sicentific publication can have is when I wrote a book on Social Economic Segregation in European Cities. It describes how the spatial gap between the poor and the rich is widening in capital cities across Europe. I decided to send out a press release, which was eagerly picked up by Dutch media. A website and Youtube video explaining the subject matter followed and the international press became interested as well. Articles appeared in more than 30 European and US newspapers. British newspaper The Guardian included the video in their news feed and their article hit more than 200.000 views within days, reaching a wide international audience. This book had more impact than any journal article could have. It also led to new research projects and funding. As a spin off it resulted in many requests for guest lectures, and still I’m often asked as an expert in this field, because the media attention is visible through Google.

For me it is a conscious strategy to spread knowledge as wide as possible. Therefore I fully support the Open Access policy. I do not mind putting in the time and effort even though that does not necessarily guarantee public or media attention. Sharing my insights can benefit the way people think and act. That is highly rewarding.”

More information: and
Photography: Marcel Krijger –

Author: Marieke Hopley TU Delft Library – m.j.hopley@tudelft.nlPublication date: July 2017

Don’t want to miss the next edition? Subscribe here!

TU Delft Library