Tuesday, November 24, 2015
Monday, November 16, 2015
Sunday, November 15, 2015
Friday, November 13, 2015
The Painful Necessity of Replicating Research
The Painful Necessity of Replicating Research
A landmark study published this summer in the journal Science estimated the "reproducibility" of psychological research. It rightly received massive media attention, much of it centered on questions of whether research in psychology should be trusted. But the research in that field is not alone in being questioned.
Academic research, especially in the social sciences, is undergoing a profound change today that is born of a moment of crisis about the trustworthiness of research findings. There has been increased scrutiny over when we "know" what we think we know. Such scrutiny includes questions about whether a single study can or should serve as a definitive answer to a question, as well as on how statistics should be used and interpreted. As a colleague once asked us, "Would you set national policy based on the results of a single study?" We would not, nor should anyone else.
Education isn't ignoring these issues, but such questions do not yet dominate our education research discussions. Our 2014 paper in Educational Researcher, ”Facts Are More Important Than Novelty: Replication in the Education Sciences,” spurred significant discussion, but more than discussion is needed.
—Getty
Education can make a singular contribution to the evolution of how social-science research is conducted and interpreted. This is because the field has vast experience in an area highly related to replication: program evaluation. For example, despite generally being conducted with the best of intentions, some replication attempts are being met by a considerable and growinganti-replication backlash, accusations of bullying, and even concerns about possiblegender or race bias in replication. These are exactly the issues that often come up in discussions of education program evaluation. So we can pull lessons from evaluation to make sense of the growing paradox surrounding replication: How can something almost universally acknowledged to be valuable be so often reviled and controversial?
As researchers who have been involved in both replication research and program evaluation, we believe that if replication is viewed as a special case of evaluation, members of the education community can (and should) lead the charge on using replication to improve scientific research. What follows are some lessons we've pulled from years of evaluating education programs, or, more to the point, lessons about the psychology of program evaluation. They are not meant to be exhaustive, but rather a springboard for more discussion about replication within the education sciences.
• No One Likes to Be Evaluated.
Everyone tends to be a fan of evaluation ... until their work is the focus of those verification efforts. Replication is no different. When one of us was involved with an evaluation of a federal agency, most of the staff members were helpful and congenial; ironically, the unit within the agency tasked with promoting rigorous program evaluation was the most resistant to being evaluated. It's just not human nature to welcome an external evaluation with open arms, and replication is no different.
• A Weak Defense Is Often Worse Than No Defense.
Because of the aversion to evaluation, a common response by someone whose work is being evaluated is, "But we've already been evaluated!" These previous evaluations, upon closer inspection, often tend to be self-evaluations, evaluations conducted with or by close colleagues, or those based on satisfaction surveys (that is, a low level of evidence). This defensiveness weakens one's arguments from the start and should be avoided. As replications slowly become more common in the social sciences, we have observed similar knee-jerk responses (such as complaints that "my study has already been replicated," when, on closer examination, that proves not to be the case). The best compliment for anyone's research can be found in multiple, independent replications of the original study. That may not be fun for the researcher, but that's science.
"There has been increased scrutiny over when we 'know' what we think we know."
• Don't Be a Jerk.
The motivation behind the vast majority of replications we've seen is to conduct sound scientific inquiry. However, any expectation on the part of replicators that the replicatee will be thrilled to have his or her work evaluated is probably naïve, especially if that researcher is approached in a manner that could be interpreted as hostile. An evaluator whose goal is to prove someone wrong is not one who will be well received, but an evaluator seeking to understand what is (or isn't) happening, in an open and fair manner, will be much more welcome. Yet, in other fields, there have been instances of poor judgment, in which replicators have discussed their largely negative results on blogs in unfortunate tones. To paraphrase The Dude from the movie "The Big Lebowski," they're not wrong, they're just jerks. Honest, rigorous evaluation is an essential component of academic research. Being a jerk, gloating, and bragging do not need to be part of the research process, and can work against the effectiveness of a replication attempt.
• Replication Isn't Easy.
Just as there are best practices when conducting a program evaluation, standards for conducting replications should be established. Replication procedures, however, are still in their relative infancy. Many scholars, including the economist and Nobel laureate Daniel Kahneman, have recently proposed a new etiquette for replications, suggesting that replicators must make a "good-faith effort to consult with the original author" and then report this correspondence along with the final manuscript, so that reviewers can integrate it into their process of assessing the replication. Original authors who are not responsive or helpful cannot tank the replication, and replicators who don't accurately replicate the original methods are identified before publication.
These suggestions, in the main, make sense to us. But we aren't convinced that a formal partnership is necessary. We are both in the process of conducting replications of major studies within our fields of interest. In both cases, we approached the original authors to let them know we loved their studies and wanted to replicate them—not out of any sense that they are wrong or fraudulent, but because their results, if replicated successfully, are potentially very important. Both sets of authors responded enthusiastically. If one treats an evaluation as an aggressive exercise, things will not go well. Replication is no different.
• Don't Judge a Book by Its Cover.
Others have suggested that the relative inexperience of a researcher could be associated with failure to replicate original findings, with the implication that graduate students and junior faculty members should not conduct replications. Experience can be helpful, to be sure, but casting aspersions on entire groups of researchers is the type of argument social scientists typically spend their careers fighting, not propagating. There are plenty of weak evaluators out there, and it stands to reason that there are also plenty of weak replicators. But those least entrenched in a field can often be the best at identifying potentially fatal flaws in research findings. And what better way to learn methods than to replicate seminal studies?
MORE OPINION
• Results Are Rarely Appreciated at the Time.
The results of a program evaluation are often underappreciated when the study is concluded. This is especially true when the report contains constructive criticism and recommendations for significantly improving the program in question. But after a period of time—weeks, months, or even years—people gain emotional distance from the recommendations, take them much less personally, and view the suggestions for improvement in a new light. The same will likely be true with replications.
Replication is a critical, if underused, part of the scientific process. It has become both more popular and more controversial recently, but we should not allow the controversies to outweigh the many benefits for education. Because inaccurate findings pollute the scientific environment, the goals of a good replicator should be to identify these pollutants so that they can be removed from the environment, but with the tacit admission that one person's pollutant may be another person's life's work and passion. We hope other education researchers join us in our fight to change the research climate to one that encourages clean and kind replications.
Sunday, November 8, 2015
So just what should that school resource officer have done?
So just what should that school resource officer have done?
Nathaniel Cary, ncary@greenvillenews.com
Clear the room.
That’s how multiple local law enforcement officials say they would have handled an incident caught on videotape in a Columbia-area high school where a school resource officer forcibly removed a female student from her desk and threw her to the floor across a classroom. Videos of the incident quickly went viral and the officer was fired.
But many law enforcement officials in the state said there’s no specific protocol for how to handle an uncooperative student who refuses to get up from a desk when confronted by an officer.
The state SRO training doesn’t cover the tactical moves, but focuses more on theory, and officers at different schools may handle the situation differently based on each police department’s protocol.
But every official who spoke to The Greenville News about what they would do differently said the school resource officers’ goal should always be to de-escalate a classroom situation.
The best way to do that, they said, would be to clear the room of students.
“I had a situation like that where a kid wouldn’t get out of his desk,” said Tony Koutsos, a SRO at Mauldin High School. “So do you know what I did? I took everybody out of the classroom. As soon as you change the environment and take the audience away, there’s no problem.
“As long as you understand who you’re talking to, that they’re still just kids. No matter how mature they are, they still have the mind of a kid,” he said.
School officials, police and trainers each said the main priority for school resource officers is to provide safety. Their second priority is to enforce the law, which in South Carolina can sometimes bring an officer into a classroom setting.
And it takes training on how to handle students and situations that could occur in a school setting, officials said.
Most school resource officers who work in Greenville County schools have taken training classes specifically offered to SROs by the South Carolina Criminal Justice Academy or the National Association of School Resource Officers.
Officers are not required to take the additional training course prior to working in schools, but many do.
All but three of the 36 SROs in Greenville County Schools have taken one of the SRO-specific courses, according to information provided to The News by each law enforcement agency in the county.
The three who have not taken the 40-hour course are new to their positions and are scheduled to take it soon.
Experienced officers who show a desire to work in schools are chosen to fill those roles, and more awareness of the need for officers to provide safety became evident in the wake of dozens of school shootings across the United States in the last 20 years, local officials explained.
The role and actions of SROs were placed under a microscope in the days following the Oct. 26 incident at Spring Valley High School in Richland County that was caught on camera.
However prior to that, the Greenville County School District formed agreements with law enforcement agencies to provide officers at all middle and high schools. In the weeks after a shooter killed 26 people at Sandy Hook Elementary School in Newtown, Connecticut, agencies began to rotate patrols through elementary schools.
There are 17 middle school SROs and 19 high school SROs in Greenville County Schools, said Oby Lyles, school district spokesman. Each school has one SRO except Mauldin, Hillcrest and Woodmont high schools, which have two.
The SROs are employed by their agency, not the school district, and their primary responsibility is to investigate crimes and make arrests as warranted, Lyles said.
The schools are the officers’ jurisdiction - “that’s their city,” said Mauldin Police Chief Bryan Turner.
In their “cities” they are trained to act more like community patrol officers – building relationships with students, giving support when needed, but also investigating crime and making arrests when needed.
Disturbing-schools law
South Carolina is one of the few states where “disturbing schools” is a state law. Officers can be called into the classroom more frequently than in other states, said Mo Canady, executive director of the National Association of School Resource Officers.
Disturbing schools is a misdemeanor charge that carries up to a $1,000 fine and 90 days in jail in South Carolina. Students can be charged if they “interfere with or to disturb in any way or in any place the students or teachers of any school" or "act in an obnoxious manner."
Disturbing school was the third most-frequent offense associated with juvenile cases in 2013-14 with 1,189 cases, according to the state Department of Juvenile Justice. It ranked just below assault and battery and shoplifting.
South Carolina’s law “changes the game a little bit” and likely results in more officers in the classroom confronting students like what happened in Columbia, Canady said.
State law leaves it up to the SRO to decide whether a student should be charged for a matter that in most states would be handled as in-school discipline.
The national organization trains officers in general to step back and let school administrators handle most situations, Canady said.
“One of the things that we want to do is de-escalate situations,” he said. “Where is the line between it being a matter that needs to be handled just by the school administration and where is it that the school resource officer should step in?”
Greenville County Schools’ policy says “SROs may provide assistance to the administration in discipline matters such as assisting with a student who is being disruptive,” Lyles said.
SROs can question students about any matter “pertaining to the operation of a school and/or enforcement of its rules,” the district policy says.
The policy says the SROs questioning should be done “discreetly, and under circumstances which will avoid, to the extent practical under the circumstances, unnecessary embarrassment to the person being questioned.”
Koutsos, the SRO at Mauldin High for nine years, said the vast majority of the time he’s called to a classroom, the student cooperates immediately upon his arrival.
Most of the time, he will take the student to his office to talk.
“The majority of cases are kids just needing to vent,” he said. “'My mom and dad are going through a divorce.' 'I just broke up with my girlfriend or boyfriend.' 'I’m pregnant.'”
Sometimes, if it’s a repeat offender or the student remains uncooperative, he will file charges.
“I really, really, really try not to arrest a kid because when you do that, it’s on your record,” he said.
Teachers handle 90 percent of classroom situations, said Scott Rhymer, Mauldin High's principal. If it gets out of hand, an administrator is called to the classroom. SROs are a last resort.
What it’s like
Each morning at Mauldin, Koutsos and another SRO, Becky Sulligan, stand in “The Circle,” an area that leads from the school’s large atrium to the classroom hallways.
They watch for arguments that could flare up into fights and they’re making sure students don’t roam the halls before class.
Students walk up to Koutsos and give him a fist-bump or a hug. One girl brought him a bag of doughnuts and asked boyfriend advice.
“Smile” he tells one girl passing by. “Take off your hat, please” he tells a boy. A few minutes later he yells out “thank you!” as the boy -- now hatless -- walks past again.
He teases one student about dancing at lunch and another about her hair.
Koutsos is loud and commanding, but also approachable. Several students notice his new haircut and tell him so.
The entire time he’s interacting with students, he scans the hallways.
“Sometimes they’re loud because they’re loud,” he said. “Sometimes they’re loud because they’re about to fight.”
After the morning rush, he walks the perimeter of the building to make sure all doors are closed and locked.
“I keep them safe from the public. I keep them safe while they’re in here. And I’m a resource if they need to vent. That’s what it boils down to,” he said.
Purpose
Schools in a few states began to employ school resource officers in the 1950s and 1960s, but it wasn’t until the late-90s that SROs took off nationwide, according to the state Criminal Justice Academy’s SRO training manual.
Greenville County Schools began to employ SROs in 2000, adding them to schools as local police budgets allowed, Lyles said.
They came with the goal to add safety, especially after mass shootings at Columbine High School in Colorado and elsewhere.
At Mauldin, Greenville’s largest high school, officers are needed to enforce the law and provide safety to nearly 2,500 people every day, Rhymer said.
The officers are visible and provide a link between anything that happens outside the school that could affect students inside the building, he said.
Administrators meet weekly with the SROs to discuss issues and ways to improve safety. This semester, Koutsos and administrators have been discussing the need for a new camera system to monitor the crowded cafeteria.
Meanwhile, the incident in Columbia has sparked discussions among SROs and school officials as to how to handle incidents in classroom settings.
Greer Police Lt. Jimmy Holcombe said he’d discussed the incident with the department’s four SROs. Mauldin’s Rhymer said administrators and the SROs had talked there about how better to handle a similar situation.
But each said their concern is that the incident would cloud the public’s view of the purpose of having school resource officers: to provide safety.
“The way things are going right now -- social media, the news -- we’re the bad guys all of a sudden,” Koutsos said.
Some students have been pushing the envelope at Mauldin in recent weeks, trying to draw a reaction from him, he said.
That, he said, is why it pays for SROs to get the training they do and grow from experience.
Thursday, November 5, 2015
Wednesday, November 4, 2015
Tuesday, November 3, 2015
Monday, November 2, 2015
Subscribe to:
Posts (Atom)