All studies are equal….

All studies are equal…

…but some are more equal than others.  At least, this is what the Jamaican public is being asked to believe.  The recent decision of Minister of Education Ronald Thwaites to table the report of a preliminary study of inmates in Jamaica, as well as defend his actions raises critical questions about the way that information is used to inform social policy.

Whenever I read articles in Jamaican newspapers about this or that study and its egregious claims about controversial issues – education, women’s sexuality and crime are often the foci – I admit that usually choose not to respond.  I tell myself that the effort that it will take to do so just seems to give more credibility to whatever was said, even as I worry about the damage that such misinformation is likely to do to a society that is increasingly self-referential and where asking questions is treated as an invitation for a put-down rather than respectful engagement.

This time, however, not only did the newspapers scream “Non-traditional high schools breeding criminals, police say” (Jamaica Observer) and “Prison Schools – Gov’t Study Says Poor-Performing Institutions Produce Most Inmates” (Jamaica Gleaner).  Minister Thwaites chose to throw the full weight of his authority and office behind the claims made by the study by presenting it to the House of Parliament as if it were legitimate, reliable and credible knowledge.  This cannot be happening, I thought.  But it was.

The study in question was conducted by the Research, Planning and Legal Services Branch of the Jamaica Constabulary Force.  The members of the research team have not been publicized.  The study consisted of administering questionnaires via 5-minute interviews with 851 men and 43 women who are currently incarcerated in Jamaica’s prisons.  The interviews focused on the inmates’ education, family and criminal history.  At best, that’s exactly what this study is: exploratory, useful only for generating questions and identifying areas that need to be investigated in a more thorough way.  The rush on the part of the legislators to turn a single, poorly designed, preliminary study into public policy is both hasty and uninformed.  I am wondering if the standard for the quality of usable information is so low that anything that presents itself as “research” and with quantitative figures – tables, percentages, standard of error – meets the bar.  Otherwise, what could make this study so different and compelling that Ministers Thwaites and Bunting, both legislators and policymakers, are actively defending the findings of the study as if it offered fool-proof and incontrovertible evidence about a direct and causal connection between criminality and which schools one attends in Jamaica?  That no such relationship emerged from, or was indicated in the data presented in the report seems beside the point.  That the report itself also acknowledges serious limitations of the information gathered also seems to be irrelevant to their cause.

Among other things, the study set out to produce a list of schools that inmates attended prior to being incarcerated. And that it did. Now what?  What the study did not and cannot offer, based on its limited design and non-existent analytic framework, is how one variable (school attended) is related to the other (involvement in criminal activity and subsequent incarceration).  And yet, this is exactly the analytic chasm that legislators, pundits and the public have leapt into, echoing reckless claims about schools being “breeding grounds” for criminality, and indicting the teachers, administrators and students who are associated with those schools, all in search of the quick fix.  So, I am left to wonder: is research only useful to policymakers when the findings confirm what we already [want to] believe? Is this really the way that we want to approach making policy about youth, or any other group?

Let me take a step back.  If research is to be taken as the cornerstone of public policy regarding schools and communities, and I certainly champion such approach, then legislators and citizens need to ensure that the best possible quality of data is being used to inform the decision-making.  But it must be good research, solid evidence that was gathered with the intention of producing the best knowledge possible.  By research, I mean the systematic process whereby one identifies an issue, develops a question that can be answered, devises a method for gathering information that can be used to answer the question, identifies patterns in the data, and comes up with a variety of credible explanations for those patterns. To have good quality data – whether in the form of numbers or words – the research must be designed in a way that ensures that the information that you receive is unbiased and free from “noise” i.e. interesting bits but unrelated to the question you are asking. Especially important to research is the collective process of creating knowledge.  For the work to be credible and reliable – and this is what distinguishes ‘good’ research from ‘bad’ research – it must seek to provide unbiased explanations of the data, rather than use the data to support pre-existing assumptions.  The work being undertaken must also recognize, and be evaluated on, its limitations.  To the extent that the limitations of the research are so significant that the data cannot be used to answer any question, or the data is so specific to the study that it is meaningless beyond the instance in which it was done, there is nothing to do with all that work except throw it away and start over.

Far from what the Gleaner editorial (January 26, 2014) disparages as “peevish whingeing”, there are many legitimate questions to be raised about both the prison study and the way in which the information is being put to use. The conditions under which the study was conducted violate basic ethical guidelines of research.  Yes, there are guidelines that experienced researchers do follow if they want the data they collected to be treated with any seriousness.  Participation was not really voluntary; the report claims that the inmates gave oral consent, but there’s no way to verify that.  The fact of having police interviewing prisoners in lock-up means that the prisoners actually have very little choice about whether they can participate; the possibility of retaliation is real.  The report notes that all information is self-reported and that there was no verification of information given by the inmates.  If the data is to be treated as reliable and valid, it must be verifiable.  The researchers did not take this critical step.  There is no way to know how accurate any of the information is. As such, the veracity of the information is suspect.

The study’s design reinforces the existing male-centered view of prisons and crime in public discourse and little does help us to understanding how and why young women end up in jail. Information about the total number of men and women in the prisons is not disclosed. The design does not adjust for the fact that the majority of the prison facilities in Jamaica is built to, and, house men.  As such, a simple random sample cannot produce enough information on women inmates that would make the study’s findings representative or meaningful.  And so, when the vast majority of the prisoners in the study are men (19 men to every 1 woman), and the data is lumped together to find an ‘average’ prisoner, it’s a no-brainer what the result would be.  The study essentially treats women inmates as if they don’t exist and don’t matter, making them invisible in the study as well as the subsequent report and its recommendations.  This is particularly problematic given the growing number of women and girls who are getting caught up in correctional system (the Armadale tragedy brought this into full view), and the hotly debated issue around housing youth in the adult prisons.

It does not seem as if much research went into the making of the study. Only one unpublished working paper (dated 2001!) was cited, when a basic google search would have shown that the paper was later published in 2003.  There’s also no data in the report to compare the education or family profiles of the inmates to the general population.  For a report that is purportedly intended to influence policy, the researchers don’t seem to think it relevant to provide context for understanding what they found.  Even if the focus is only on men, how is the “typical” inmate similar to or different from men of the same age who are not imprisoned?  If the majority of children born in Jamaica since 1838 (Emancipation) are born to or raised in mother-headed families, what do we make of the study’s claim that boys raised in such families are more likely to end up in prison? Given that most of the inmates interviewed either dropped out or were pushed out of school, it would certainly have been useful for the researchers to note whether there are existing studies that link either of these factors to young men’s (and even women’s) subsequent involvement in criminal behavior. Instead, readers are left to fill in the gaps with their own ideas about what is or is not true.  And that’s exactly what is happening.

Other basic issues which are relevant to the interpretation of the findings but were not mentioned in the report or the subsequent media analyses:  First, the kinds of criminal activity that are actively policed (and labeled as ‘crime’) is not randomly distributed throughout Jamaican society.  Second, the criminal justice system in Jamaica is not exactly one that is either fair or immune to manipulation; who ends up in prison is not simply a result of what they did, but also of their inability to beat the system through high-cost legal representation and other methods. Third, legal and policy changes directly affect the make-up of the prison population i.e. who ends up in prison and what they are charged for. Fourth, to my knowledge, the majority of Jamaican youth who have any secondary education would have attended a ‘non-traditional’ or upgraded school anyway. Fifth, access to schooling in Jamaica is not randomly distributed.  Jamaican youth do not get to choose what schools they attend; they go where they are placed, and, since placement is based on academic achievement, and academic achievement is related to socio-economic status, this means that children from impoverished and working class communities are more likely to go to school with each other than with non-poor children.  From existing research, we know that premature ending of secondary education is directly related to socio-economic factors that exist outside the school, and produce various dispositions and experiences within school settings.  Sociological research also shows that schools tend to reproduce existing social arrangements in the society.  In Jamaica, this is no different: most children from poor and working-class backgrounds do not have adequate resources to support their education at home, tend to go to poorly resourced schools that operate more like holding pens, from which they are channeled into dead-end jobs and under-employment etc. etc.  If Minister Thwaites is aware of this, he does not indicate as much.  Instead, individual schools are being targeted for what is essentially a structural problem in the entire educational system.  Curiously, he did not present any other studies to Parliament that show the factors that lead to a healthy school environment and to positive outcomes for students, regardless of socio-economic background.  What does all this mean? It is impossible to deduce anything about the relationship between schools and criminality from this single, preliminary study.  All we have are more questions, and few answers.

The researchers seem unaware of the various caveats that need to be made when presenting a study that purports to “explaining the educational background of prison inmates” and “describe the characteristic features of the typical inmate” [sic].  While they are convinced that what is little more than descriptive statistics can be used to inform policy in education and prison systems, in fact, the study has explained nothing.  Such an omission could be forgiven if this was a project completed by high school students or even college students taking their first research methods course. This oversight is unforgivable however, when coming from a statutory body, given the complexity of the issues involved.

To my mind, these oversights reflect the biases built into the study itself.  The study was designed and authored by the Jamaica Constabulary Force, a body which has a direct investment in the outcomes of the study.  That the Jamaican police are having a difficult time in curtailing criminal behavior concerning guns, drugs, money laundering and gangs is not news.  From the news reports, we know that the problem is especially acute in the heavily populated areas of Kingston, St. Catherine and St. Thomas, as well as rapidly urbanizing towns in Clarendon and St. James.  These are also spaces of incredible economic and social inequality, and where poverty has spawned a host of social problems, involving people of all ages, especially youth.  Instead of speaking to the larger structural issues that affect the types of criminal activities that ensue, the JCF uses the study to take direct aim at the national government’s attempts to expand secondary education for poor and working-class children, and declares it, a failure at best, and contributing to the involvement of youth in criminal activities. Indeed, the report claims: “it is widely felt that a significant contributing factor to the high involvement of youths in criminal activities in Jamaica is the worrying state of the country’s educational system.” (p. 5).  There is no indication of who feels this way, why they would feel this way, or whether there is any evidence to support this claim.  Here, I suspect that this is the opinion of the JCF being passed off as a fact.

On this matter, the report is certainly clear: the study was designed on the premise that schools mattered, and so the researchers set out to identify which schools the inmates attended.  They wanted to show that there is a causal connection between school and history of involvement in criminal behavior.  The data was gathered in a way that merely affirmed what the researchers already believed: that some schools ‘harbored’ or ‘breeded’ criminals more than others. From this perspective, which schools showed up on the list was almost not as important as the production of the list itself.  No one has said it better than Commissioner Ellington:”We wanted to provide some evidence of a correlation between issues in the school community and the risk factor for individuals going into criminal activity,” (Gleaner, “Thwaites’ Presentation Of Prison Report Doesn’t Paint Complete Picture – Commish”).  Set aside the truism that correlation is not the same as causation.  Such a startling and bare-faced admission leads me to wonder whether Commissioner Ellington understands how damning his statement was. In effect, he revealed that the study is guided by a fundamentally unethical research practice of collecting and reporting data in such a way that it proves what you already believe to be true.  Or, perhaps neither Commissioner Ellington nor the legislators care about whether they are using ‘good’ research or not.  As long as it’s called research, and the information says what they want it to say, no more questions need be asked.

Equally problematic is that many voices – some with more sway than others – have been raised in support of the public disclosure of the problematic report, claiming that no harm was intended to the schools and that disclosure is a necessary part of addressing the problem.  Airing the dirty laundry is a good thing, they say. But intention has little to do with effect. The implication of such vociferous support is that it doesn’t matter how the information was collected or why: if there is evidence that there is something particular about the schools themselves that actively enhance and even produce criminality among youth, then action should be taken.  Except, the study didn’t provide any evidence of such. Further, what action and to what end? If there is a school-to-prison pipeline, and there probably is one, how exactly does public shaming of the individual schools (and those associated with them) as well as directing specific and limited resources towards those specific schools address the problem?  That there is no definitive information about direction of association does not seem to make any difference.  The police and legislators believe that the school is at fault; the public is certainly encouraged to believe so. I suspect that the principals and students of those schools might see things a little differently.  That the study does not provide any answers should have dissuaded Minister Thwaites from embracing the study so quickly and uncritically, until corroborating evidence had been gathered to make their case.  That would have been the prudent, responsible and informed course of action to take.

One implication of shifting the locus of police attention to schools is this: when specific schools are labeled as the centers of, or at least conduits for, criminal behavior – “breeding ground” is the pejorative language used in recent discussions – it is only a matter of time before police are given greater control of and authority within schools.  Given the public record of police conduct and interaction with youth, especially young men from non-wealthy backgrounds, we should all be worried about this.  And yet, studies like this one, flawed as it is, operate as the perfect justification for a whole range of “interventions” that the Ministry of Education and Justice did not feel they had a mandate to take before.  The rushed response to “intervene” – the militarized metaphor does not escape me – is both telling and wanting.

That is, the same low value placed on students from working-class and impoverished backgrounds that make their educational experiences less than productive in the first place is merely being reconfirmed. Basic resources like social workers and conflict resolution programs are only being brought into the picture as a way to isolate and treat the ‘bad’ students (proto-criminals?) on the list of ‘bad’ schools.  All schools, and the students who attend them, are mostly left at the mercy of structural conditions that create the problems in the first place.

However, the reception of the study tells me that it’s not only the specific schools and the individual students that need ‘intervention’.  Legislators and policymakers need to learn how to properly assess and interpret the information that is being fed to them.  That may be difficult for many of them to accept, but the consequences of not doing so may be more than our citizens can continue to bear. Social policy can only be as strong as the evidence that supports it.  At this moment, the basis for policy seems to be emotion, prejudice and raw opinion, with a poorly designed study or two thrown into the mix to support particular agendas and points of view.  Maybe this is how public policy is usually made in Jamaica.  I hope not.  However, if the current reality of youth’s lives and experiences in schools and communities is any indication, this approach is a recipe for disaster and cannot be allowed to continue unchallenged.