Plausibility test of PEW reports on restrictions of religion: Cross-sectional country comparison for the years 2007 to 2014

Monday July 18th, 2016 / Articles, ProMundis Blog, Religions, Religious Freedom / shortURL

Thomas Schirrmacher (translated by Christof Sauer)

This is a plausibility test of the “Restriction of Religion” Reports of the PEW Forum of Religion and Public Life published 2009-2016

Introductory notes

Between 2009 and 2016, the Pew Research Center’s Forum on Religion & Public Life published five global reports on religious freedom. They include comprehensive rankings, the development of which were compared in different variations.

A plausibility assessment at an early stage is standard procedure regarding all statistics and scientific research outcomes. It has the potential to uncover weaknesses in investigations early on without having to say where the implausible results stem from.

The categorization of individual countries in the PEW reports did not appear very plausible to me. Countries with rather liberal religious freedom received very poor evaluations, and countries with serious restrictions received positive ratings. For instance, in 2013 Germany received a Social Hostilities Index (SHI) rating of 4.5, worse than Saudi Arabia’s rating of 3.6. Additionally, a number of countries fluctuated significantly over the six years examined, during which, from my point of view, there were no notable changes standing out regarding the situation. Furthermore, there were significant differences in the classification of countries when compared to the results of other researchers.

I thus began to compare countries by looking at a number of countries over time. I then looked at countries reported on for a single year period, and then over time, and compared countries with each other. The outcome of this sample examination can be found below and raises considerable doubts regarding the reliability of the results.

One possible reason for such results could lie in the fact that the data base for the reports is very thin. There is actually no on-site research underlying the PEW religious freedom reports. Likewise, there is no elaboration by experts within the prospective countries or by external experts on those countries. [That is unusual since the Pew Foundation otherwise conducts comprehensive surveys and research on religion, etc., in many countries.] The results emanate exclusively from encoding other reports. Again, among those reports there is none which results from on-site research. Additionally, all 18 reports are highly dependent upon each other.

Reports, abbreviations, scores

PEW report 2009 = status mid-2007
PEW 2011 = mid-2009
PEW 2012 = mid-2010
PEW 2013 = end 2011
PEW 2014 = end 2012
PEW 2015 = end 2013
PEW 2016 = end 2014

All dates given below refer to the status date and not to the years in which the PEW reports were published.

GRI = Government Restrictions [on Religion] Index

SHI = Social Hostilities Index

GRI: „Very high“ = 6.6 or higher; „High“ = 4.5 to 6.5; „Moderate“ = 2.4 to 4.4; „Low“ = 0.0 to 2.3

SHI: „Very high“ = 7.2 or higher; „High“ = 3.6 to 7.1; „Moderate“ = 1.5 to 3.5; „Low“ = 0.0 to 1.4

Testing the scores on Germany

a) Chronological cross-section

Germany GRI scores: 2007: 3.1; 2008: 3.2; 2009: 3.5; 2010 4.0; 2011 3.5; 2013: 4.5; 2014: 3.4. Has there really been a worsening in government restrictions on religious freedom from moderate in 2007 to high in 2013 and then back again? No. The score was already too high in 2007. In addition, there is no proof whatsoever that legal restrictions on religious freedom have strongly increased during this six-year period.

Germany SHI scores: 2007: 2.1; 2008: 2.5; 2009: 3.3; 2010: 5.3; 2011: 5.0; 2013: 4.3; 2014: 2.5. This set of scores suggests considerable fluctuation, whereas in reality the situation is quite stable. The scores for 2007 to 2009 and 2014 may be considered valid, the scores for 2011 and 2013 are much too high.

The GRI should be quite low and the SHI much higher in Germany, but PEW says that in 2007 the SHI was by far the lower of the two scores, and in 2013 it was still slightly lower, in opposition to reality.

b) Cross-national comparisons with Germany for 2013

Germany GRI 2013: 4.5:
Countries with comparable scores (+/– 0.2) included Libya, Palestinian autonomous regions, Ethiopia, UAE, Greece, and Nepal
Countries with a slightly better score than Germany included Nigeria and Liberia
Countries with a much better score than Germany included South Sudan, Serbia, Hungary, Italy, and Cambodia.
That is really a strange result.

Comparison between Germany and the geographical area of Europe on the 2013 GRI:

Only three countries scored worse than Germany: Belarus, Bulgaria, Russia. Countries scoring better included Austria, Bosnia-Herzegovina, Georgia, Italy, Malta, Serbia, Greece, and Hungary. Accordingly, Germany would place near the bottom in matters of governmental restrictions of religious freedom within Europe. That is far from reality.

Testing other countries

Bulgaria SHI: 2007: 2.2; 2009: 4.0; 2010: 2.2; 2011: 4.7; 2012: 4.4; 2013: 3.6; 2014: 2.8. The score constantly fluctuates, even though the situation on the ground is stable.

Indonesia GRI: 2007: 6.2; 2012: 8.3; 2013: 8.5; 2014: 7.9: Allegedly increasing significantly, although in reality the problems are decreasing. Compare Iran GRI: 2007: 7.9; 2012: 8.6; 2013: 8.3; 2014: 7.7. Is the situation in Indonesia as bad as or even a little worse than in Iran? I think that is nonsensical.

Mexico GRI: 2007=2008: 4.7; 2009: 4.2; 2010: 3.5; 2011: 3.6; 2012: 3.9; 2013 3.4; 2014: 4.5; SHI: 2007: 5.5; 2008: 4.7; 2009: 5.1; 2010: 3.6; 2011: 3.2; 2012: 6.7; 2013: 3.7; 2014: 4.2: Did the situation really improve decidedly in 2013? Is a jump in the SHI score from 6.7 in 2012 to 3.7 within one year plausible? And is there any evidence for such movement?
Also see the SHI: the 2012 and 2007 scores are very bad; 2010. 2011, and 2013 are much better. But there is no evidence for that.

Brazil SHI: 2007: 0.8; 2013: 3.7; 2014: 3.5. There is no evidence for such a dramatic worsening from one of the best countries to a country with real problems, while the GRI is one of the best all those years.

Iraq SHI: 2007: 10.0; 2012: 9.0; 2013: 7.4; 2014: 8.5. Has there been such an improvement regarding social hostilities in Iraq for 2013? The same could be asked about Syria.

Saudi Arabia: SHI: 2011: 6.5; 2012: 6.4; 2013: 3.6; 2014: 3.6: There is no evidence for such a strong improvement of the SHI, which has fallen to the same level as for Germany.

Hungary GRI: 2007: 0.3; 2011: 1.0; 2012: 2.4; 2013: 2.9; 2014: 2.8; SRI: 2007: 1.0; 2011: 2.0; 2012: 2.7; end of 2013: 2.4; 2014: 2.2: In reality there has been a considerable worsening of the situation, which is not reflected in the scores. Hungary here scores considerably better than Germany, whereas religious minorities see the situation the other way around.

Various countries received a very low SHI score (below 0.5) in 2013 when in fact a deeply rooted hatred of other ‘races’ or social groups and a strong antagonism between the respective religiously diverse groups exist. Examples: Jamaica SHI 2013: 0.4; Cambodia 0.4; Cuba 0.0; Kazakhstan 1.0; Taiwan 0.0; Turkmenistan 0.4; Oman 0.1; Western Sahara 0.0; Congo 0.0.

Examination of the sources (18 reports)

Only 4 out of the 18 reports deal specifically with religious freedom. All other reports deal with discrimination in a broader sense or with other topics.

It is explicitly stated that the second source is the main source and that all others are consulted in addition only when there are discrepancies (Jan 2014: p. 40). This means e.g. that Germany owes its poor scores mainly to the very negative comments in source 2.

I choose the report for 2013. The report for 2014 deletes one report on religious freedom (Hudson, no. 6) and the reports by Amnesty International (no. 11) and replaces them by the Global Terrorism Database (now no. 10), which weakens the direct religious freedom data further.

All reports that mention tensions are relevant for the SHI, which reports on tensions due to racial and other reasons. But none of them gather data specifically focused on discrimination carried out by the majority population due to their religious adherence. Therefore, the SHI for Germany rather makes a statement about the relationship of Germans to foreigners than about their relationship to adherents of other religions. However, this problem is nowhere addressed by PEW.

None of the said reports is striving for exhaustiveness by their nature. They tend to report what information they receive or learn through the media. Does this mean that if there is a lot of negative reporting about a country, the score rises, whereas if there is little reporting at all, the score falls? Or how does the score evaluate the quantity and quality of instances in relationship to the size of a country’s population?

Furthermore, the reports assess quite varying time frames, and they often make statements that go beyond the previous 12 months. Nevertheless, they are still attributed to a fixed one-year period by PEW.

None of these reports aims to determine any kind of ranking or scores. The PEW methodology simply makes use of the events listed in those reports. If any statement in the PEW reports gives the impression that they are based on value judgments found in other reports, such a statement is misleading.

Many of the source reports, such as USCIRF, report only on selected countries, usually 36 or fewer (source no. 17 for 2012 = 0; source no. 12 for 2012 = 29). Only source no. 2 is reporting on all countries (except the USA). This means that a given country may be mentioned in only half of the 18 reports. No methodology is given that would prevent those countries that are mentioned in almost reports from faring worse than those covered in only a few of the sources.

The reports are in part strongly interdependent and point to each other as sources. This makes them unsuitable for a scholarly ranking. For example, the reports by the US State Department quote most of the other 17 reports of human rights organizations (e.g. AI, HRW). In turn, these other reports frequently quote the reports by the State Department or the US government in general.

Reports no. 2, 3, 4, 14 and 16 are political and official government reports from within US government, but not scholarly reports. Reports no. 9 and 10 emanate from other governments or networks of governments.

No scholars in the countries themselves are being consulted by PEW, not even for a plausibility assessment of the results.

None of the reports contain original research on or in the countries (as PEW often does it, or as is done by the World Watch List [WWL] which focuses on Christianity only9.

Comments on the 17 reports used as sources by PEW (latest report)

1. Country constitutions

This information rarely changes from one year to the next and says little about the actual situation. Iran, for instance, has a good statement on religious freedom in its constitution.

2. US State Department annual Reports on International Religious Freedom

Their content is anecdotal, meaning that only events are included which have been reported to the State Department. It does not represent systematic global research. The contents are strongly dependent on positions of US foreign policy (see e.g. the report on Saudi Arabia) and on stereotypical views of other cultures.

The reports always start with what is listed under no 1. above and frequently refer to the reports named below as sources.

The points where Germany is scoring negatively on State Department reports are its alleged system of state-acknowledged religious bodies and alleged persecution of Scientology.

3. US Commission on International Religious Freedom annual reports

Annually covers between 10-15 countries of varying composition. Not significantly dependent on positions of US foreign policy, but the main task is to recommend “countries of concern” to the US Department of State.

4. UN Special Rapporteur on Freedom of Religion or Belief reports

Annually only reports on very few countries, changing from year to year based on what countries have been visited by the rapporteur. This means there is no comparative reporting on the same country year after year.

5. Human Rights First reports in first and second years of coding; Freedom House reports in third, fourth and fifth years of coding

This is a general report on democracy, which does not specifically cover religious freedom.

6. Hudson Institute publication: “Religious Freedom in the World” 2008 (Paul Marshall)

This is a very good report. But since it appeared only once, it cannot be used for year-to-year comparisons.

7-17. All these are human rights reports which do not specifically deal with questions of religion and only provide anecdotes.

7. Human Rights Watch topical reports

8. International Crisis Group country reports

9. United Kingdom Foreign & Commonwealth Office annual report on human rights

10. Council of the European Union annual report on human rights

11. Amnesty International reports

12. European Network against Racism shadow reports

13. UN High Commissioner for Refugees reports

14. US State Department annual Country Reports on Terrorism

15. Anti-Defamation League reports

16. US State Department Country Reports on Human Rights Practices

17. Uppsala University’s Uppsala Conflict Data Program, Armed Conflict Database

18. Human Rights Without Frontiers, “Freedom of Religion or Belief” newsletters

This is a very good newsletter, but it (1) almost exclusively forwards news from other sources, and (2) forwards only what is readily available. Therefore, some countries are almost never covered, whith others being reported almost on a daily basis.

PEW iirf_report_2016_2

PDF-Download

Tags: Brasilien, Bulgarien, Germany, IIRF, IIRF-Bulletin, IIRF-Reports, Indonesien, International Institute for Religious Freedom, Irak, Mexiko, Pew Research Center’s Forum on Religion & Public Life, PEW-Berichte, Plausibilitätsprüfung, Religious Freedom, Saudi Arabien, Ungarn