Timeline of the alleged ISCA 2019 peer-review incident
TL;DR
See these two links on the tragic case of Huixiang Chen:
A student committed suicide due to alleged peer-review fraud at ISCA 2019.
ACM SIGARCH/ IEEE TCCA ran an "investigation" run by a team with
ISCA conflicts-of-interest and found that no fraud occurred.
Further evidence was found that showed the student had
access to the full un-redacted comments for all papers from ISCA2019
and from HPCA. This is a clear violation of double-blind review.
After much feet dragging ACM SIGARCH re-opened the investigation, but it
took them over a year and a half to find any wrongdoing.
In the meantime ISCA 2020 and ISCA 2021 were unwisely allowed to continue.
After two years the ACM finally released the name of the primary
perpretrator in ISCA'19 and ASPLOS'17 fraud,
but the various collaborators
have seemingly escaped justice.
Minimal (if any) changes have been made to the peer review process that
would prevent this from happening again.
Full Timeline
The lead up to ISCA'19 and the horrific aftermath
- 7 December 2018
- 22 February 2019
- ISCA 2019 rebuttals due (responses to comments raised by
anonymous reviewers. ISCA is double blind so authors
are not supposed to be able to see identities of
reviewers, or the internal discussion of the papers).
- 15 March 2019
- ISCA 2019 Author Notification (told if paper got in or not)
- 22 May 2019
- ISCA 2019 camera ready deadline (when the final version of the
paper to appear in proceedings is due)
- 13 June 2019
- University of Florida student Huixiang Chen committed suicide.
- For more details see
this article
- Allegedly there were irregularities with the peer review and
problems with the paper methodology.
The student wanted to withdraw the paper but allegedly
his advisor would not allow it.
- 15 June 2019
- The advisor allegedly threatened legal action against those
reporting on the incident.
- For more details see:
this article
- 26 June 2019
- Despite the suicide, Chen's paper "3D-based Video Understanding Acceleration by Leveraging Temporal Locality and Activation Sparsity"
was still presented at ISCA.
- It still appears in the proceedings, it can be viewed at the ACM here:
3D-based video recognition acceleration by leveraging temporal locality
-
The paper claims support from millions of dollars of US government NSF
grants, including grants
1900713, 1822989, 1822459, 1527535, 1423090, and 1320100.
Taxpayer money supported those behind these alleged events.
- 3 July 2019
- The University of Florida starts an investigation after requests
from the Graduate Assistants United group.
- Article
- 10 July 2019
- A memorial service for Huixiang Chen
is held at the University of Florida.
- Article 1
- Article 2
The Failed First Investigation
- ?? June 2019
- The ACM SIGARCH and IEEE TCCA chairs post
an initial statement saying they will let
the University of Florida investigate and that
they will follow their respective processes.
- 5 August 2019
- 26 September 2019
- A bug report is filed by SIGARCH against the HotCRP
conference submission system to add extra logging.
Before this change it was possible for downloads
of paper data without being logged (this data
could be used to break the peer-review process).
- Apparently someone at SIGARCH was aware of this issue,
but officially SIGARCH claimed to be unaware of
any sort of ISCA'19 data breach until the end
of January 2020.
-
twitter discussion and the
relevant git commit
- 26 November 2019
- Paper deadline for ISCA 2020
- 1 December 2019
- Josep Torrellas of UIUC announces both on twitter and on the SIGARCH
website the outcome of the "investigation".
- Link:
Outcome of investigation on ISCA 2019 paper
- Twitter Link: Beginning of Torrellas Statement (the threading is broken)
- This announcement conveniently happened just after the ISCA 2020
deadline so it was too late for submitters to know the outcome
of the investigation before submitting.
- The final conclusion in the announcement:
"The committee has determined that there was no evidence of misconduct as part of the paper review process."
- To quote him: "The committee which conducted the investigation included members from the IEEE Computer Society TCCA and ACM SIGARCH, as well as members of the ISCA Program Committee."
-
This committee seemingly had a large number of conflicts of interest.
Many of the people involved (ACM policy is the members of this
committee are kept secret) apparently have deep ties to ISCA and it
would reflect poorly on their careers if any ill news came up
about the conference.
- 66% of ISCA'19 papers had PC members as authors, which might
have affected their objectivity as they reviewed the case.
- Torrellas himself was a member of the ISCA'19 PC as well as
had three papers accepted there, so he had obvious
conflicts of interest in investigating whether there were
irregularities with the peer review process.
- 4 December 2019
- The various ACM SIGARCH and IEEE TCCA office-holders are
contacted about the issue via e-mail. No response
is received.
- The ACM president (Cherri Pancake)
sounded concerned, but due to obscure
procedural rules within the ACM she is not allowed
to get involved in any sort of investigation.
- A first attempt is made to contact the ACM Ethics board
(upon the recommendation of the ACM President) but
no response is received.
- 9 December 2019
- An ethics complaint is made with IEEE about how the
investigation was run (EthicsPoint Complaint #50).
- The lawyer who runs the IEEE code-of-ethics enforcement
says he called the people involved and he thought
they made a convincing
argument that they didn't have any conflicts of interest
because "several" of the members were not past ISCA PC members.
Case closed.
- 11 December 2019
- Various attempts to get a response from the IEEE failed
at all levels
(from president down to local regional representatives).
A senior IEEE member at a local University had the
position that the only way to change things is to join up and
rise through the ranks for a few years and maybe then
someone might care.
- The IEEE president-elect responded
but she was only willing to discuss off-the-record
via phone call.
- 12 December 2019
-
An attempt is made to contact individually each
program committee member from ISCA2019
and make sure they were aware
of the alleged irregularities at ISCA2019.
The hope is maybe they will press internally to do something
about the issue.
-
Of the large number of people contacted (ISCA'19 PC was pretty
large) only two e-mail responses were received.
One inquired if there was any evidence.
The other said the first author of a paper has
no right to withdraw an article due to methodology concerns
and that if an advisor says publish, they should just publish
without complaint.
-
A few more responses came in where the PC members only wanted
to discuss off-the-record via phone. This was declined.
-
One phone message was left by a prominent member of SIGARCH
with a checkered past saying
they had been planning to eventually bring the topic up for
discussion at the ISCA
business meeting (meaning, in 6 months, June 2020).
- A reason for the PC members' reluctance turns up later,
when it turns out that 66% of ISCA'19 papers
have PC members as authors (the number for ISCA'20 is 48%).
- 3 January 2020
- After a month the
ACM ethics board finally responded (their reason for their
delay was they were overwhelmed with other reports of
ethical misconduct?)
- Their response was that "conflicts of interest"
are not something mentioned in the official ACM Code of Ethics
and thus they were not going to do anything.
- As an aside, reporting an ethics problem to the ACM was nearly
impossible. You have to search the website for contact info,
then sort of guess who to e-mail. Then you get no confirmation
and you might have to bug them again to get a response.
The IEEE, for all their flaws, has an issue-tracker system that
works much better for this.
- 24 January 2020
- The ACM director of publications reported by e-mail that
the makeup of the investigation board was being kept secret
as part of a general policy (to avoid retribution/harassment).
He personally vouched for the team and said he was sure if
they had been able to find any evidence of misconduct
they would have done something about it.
- Possibly related to that, the same day the SIGARCH and TCCA
chairs (Babak Falsafi and Josep Torrellas) posted
an announcement meant to clarify how the investigation was run
Information about the ISCA'19 investigation
it is made clear that a lot of insiders were deeply involved
in the investigation.
The Release of More Evidence and start of Second Investigation
- 28 January 2020
- An article is posted
Evidence Puts Doubts on the IEEE/ACM's Investigation
that allegedly shows there was peer-review
compromise at both ISCA'19 and possibly HPCA'19.
-
Full reviews from the conferences, with full unredacted names, were
found on Chen's hard-drive. With double-blind peer review,
no one should have access to the full unredacted reviews,
especially not someone in the process of submitting papers.
- 29 January 2020
- Eddie Kohler, author of the HotCRP submission software used
by ISCA, confirmed that there was a breach and that the full
paper review info was downloaded. However at the time of the
breach there was no logging to tell who obtained the files.
It is also unclear how early they had access to these files.
- This is documented in a
twitter thread.
- confirmation that the audit logs were not enabled for ISCA'19
- Back in September, after the time of the breach
(but before the breach was made public),
code was added to HotCRP to enable proper
logging of downloads like this.
In what might be a co-incidence, the request to add this
extra logging support was a request from members
of SIGARCH.
twitter discussion and the
relevant git commit
- 29 January 2020
- The top-level HPCA'19 organizers are notified about the issue, but
they deferred to let Torrellas/TCCA handle things.
- The top-level HPCA'20 organizers are notified.
The PC Chair of HPCA'20 responds that any peer-review issue was
with HPCA'19 and can't possibly have any impact on HPCA'20.
- 8 April 2020
- 14 May 2020
- The various candidates for the upcoming ACM election were
contacted to see their take on the ISCA issue.
The few that replied were unaware of the issue.
- 29 May 2020
- Despite the alleged issues with peer review in the past,
ISCA 2020 is happening as usual (or as usual as it
can in the lockdown year of 2020).
- 1 June 2020
- Margaret Martonosi of Princeton and head of the NSF CISE
directorate gave a keynote speech at
ISCA'20. She also is author of 3 papers published there.
- In December she had given encouragement to everyone who had
contacted her that she cared deeply about the ISCA'19
issues and that we should keep investigating.
Apparently she did
not care enough to withdraw her papers or refuse the keynote.
- She was also a founding member of
SIGARCH/SIGMICRO CARES
- 1 June 2020
- José F. Martínez of Cornell University
(General Chair of ISCA'20, on Board of Directors of SIGARCH)
is of the opinion that actions at ISCA19 should
have no bearing at ISCA20, as if somehow there isn't a huge
overlap between the people involved in the peer review
process of both conferences.
- Twitter link
- 2 June 2020
- An attempt is made to contact all first and last authors of
papers in the remaining two days of ISCA, to encourage
them to cancel their presentation out of solidarity
with Huixiang Chen.
- Not a single response was received.
- 4 June 2020
- SIGARCH wants everyone to be patient with their investigation
that they started in February 2020.
-
An update from Joint Investigative Committee (JIC)
- What they failed to mention is the investigation should
have started in June 2019 but Josep Torrellas wasted 6 months
dragging his feet on a conflict-of-interest riddled
(he was a PC at ISCA'19 and had 3 papers in) "investigation"
that found nothing wrong. The JIC is the second
investigation after the first one turned out to be a farce.
- ISCA'20 should really have been cancelled until all of these
issues were resolved.
- 8 June 2020
- 15 June 2020
- Emily Reisbaum the, lawyer in charge of the
Joint Investigative Committee (JIC) investigating
ISCA'19, sent us an e-mail saying
"that there are allegations - even troubling ones -
about the integrity of
one aspect of the ISCA 2019 conference,
does not warrant punishing all of
those doing good work by cancelling ISCA 2020".
- There are a lot of assumptions in that statement, especially
from someone supposedly running an unbiased investigation.
- Reisbaum seems to be of the same opinion as
Martínez, that alleged academic fraud at ISCA'19
cannot have any possibly bearing on ISCA'20, as if there
isn't significant overlap in the communities behind
both conferences.
- In addition, Reisbaum asked us to refrain from calling for
the cancellation of ISCA'20.
A bit odd, as it's a bit late for that.
Also we feel we had every right to have called
for ISCA'20 to be cancelled
until the ISCA'19 allegations had been fully and properly
investigated.
- Also, why is the supposedly unbiased JIC investigator sending
out e-mails defending the honor of those she's supposedly
investigating? We have not received one single response
from anyone we e-mailed asking to show solidarity with
the dead student. Apparently instead the ISCA authors
complained about us to the JIC, compelling them to
try to silence us. The JIC is supposed to be investigating
the ISCA authors, not sending e-mails on their behalf.
- If you are one of the ISCA'20 authors complaining to the
JIC about us: we don't think asking to cancel ISCA'20
in solidarity with the dead student is the equivalent of
some sort of harsh group punishment. Send your work somewhere
else instead. MICRO for example. Unless you think MICRO
has peer-review issues (in which case, contact the JIC).
Or maybe you think your paper would get preferential
treatment at ISCA, and if that's the case then you should
probably also report that to the JIC.
- We do not want a rushed investigation, we just feel it was
irresponsible to hold ISCA'20 before the issues at
ISCA'19 were properly investigated. If you're going to
be mad at anyone over calls for ISCA'20 to be cancelled,
it's ACM SIGARCH, IEEE TCCA, and the chairs of ISCA'20
who pushed things through in spite of protests.
- 23 June 2020
- SIGARCH posted a message
Remembering Huixiang Chen through our Actions
- The timeline they post conveniently leaves out the
failed first investigation that found
"no evidence of improper action [found] during the peer review process".
A thing to note here is that Josep Torrellas
(who lead the first investigation) was the PhD advisor
of José F. Martínez who is a member
of the SIGARCH Executive Committee.
- The "Actions" they took did not involve cancelling
or delaying ISCA'20 until a full investigation
could be concluded. This is probably
because that would have affected the publication
count of too many important people (ISCA'20 was 48%
papers by the the Program Committee, meaning the
people with final decision oversight on which papers
got in somehow made the conference nearly half
papers that they wrote themselves).
- Many of the "actions" they claim to have taken
seem to be them saying
yes, ISCA might have problems,
but we also found a bunch of
stuff going on at other conferences too.
- The article is signed by the
SIGARCH executive committee.
Please let them know how you feel about their
decisions.
- 26 June 2020
- We have called for the members of the SIGARCH executive committee
to resign their positions due to their lack of
handling of this whole incident.
- José F. Martínez (Board of Directors of SIGARCH,
General Co-Chair of ISCA'20 and
ISCA'21 and Associate Dean of Engineering
at Cornell University)
posted a rebuttal in this
(twitter thread)
which was hard to read as he blocked us immediately after
posting it so we were unable to respond.
He says that "the overwhelming majority of our research
community" does not agree that ISCA 2020 should have been
cancelled until the investigation finished, and that by
contacting the various presenters and PC members to make
sure they were aware of the ongoing investigation we were
engaging in harassment.
- 29 June 2020
- 8 July 2020
- An article is published at Nature Index:
Probe into leaked papers submitted to leading engineering conference
- The author investigated some interesting things, including the
current situation of the advisor in Florida.
- Odd that Jose Renau seems to be one of the few people
willing to speak up on the issue.
For those of you who think we have some sort of anti-UIUC
bent, we might as well point out that Renau got his PhD
under Torrellas at UIUC too.
- 6 November 2020
- It looks like they are planning to push through having
ISCA2021 despite the investigation still being
underway.
- Any bets they are withholding the outcome of the investigation
until November 25th (the day after the papers are due)
just like last year?
- 12 November 2020
- Another sad, but not surprising
article from T. N. Vijaykumar where anonymous e-mails
allege more collusion fraud at ASPLOS and with NSF reviews.
- Sad because while the general feeling is this is indeed a widespread
problem, none of the profs involved in running
the various conferences
are willing to do the right thing and do something about it.
- The even sadder part is most computer architecture researchers
know who that bad actors are. Get them talking off the
record at a conference, and they'll tell you which cliques
play peer-review games, which high-profile computer architecture
researcher was kicked out of multiple universities for having
affairs with students, and which computer architects are
known for harassing women at conferences. For whatever
reason though they all feel powerless to do anything, and
fear for their career if they speak up. Which means they
are cowards at best and complicit at worst.
The Release of the JIC Report
- 8 February 2021
-
The JIC finally released the results of their investigation
- It took them 1.5 years and multiple stumbles to possibly
confirm the events at ISCA'19 that initiated all of this.
- While investigating they also found evidence of of misconduct
at other high-profile Computer Architecture conferences
(HPCA'19, ASPLOS'17).
- It's not even 100% clear that the people punished or the paper
involved was the one that led to Huixiang Chen's death
as apparently the names will not be released by the ACM.
- As of 8 February 2021
the paper involved in Huixiang Chen's death is
still
available for download on the ACM website
by the 9th it was finally marked as being retracted,
retroactive to 26 January 2021
- This was all too little, too late, and it raises questions
about what other incidents might be happening where the
people involved were a bit better at hiding things.
- Also remember that various higher-ups in the ACM and IEEE
(the people running the recent ISCA conferences,
as well at the IEEE TCCA and ACM SIGARCH board members)
dragged their feet against a second
investigation and did their best
to dismiss the allegations.
-
Jonathan Aldrich reports:
"Apparently it has been longstanding ACM policy to keep
names/institutions of those disciplined confidential.
The reason given is to avoid creating 3rd party conflicts
or having external groups imposing additional penalties.
- By keeping names a secret
It's unclear how the ACM is going to ensure those punished
do not serve as reviewers or external reviewers, and also
they provide no protection to other organizations such
as IEEE or USENIX from accidentally using them as reviewers.
Also, these reviewers will possibly still be able to compete
for grants and faculty positions with the ACM hiding their
findings.
- For all we know those guilty remain high-level members of the ACM
as the findings makes no mention of them being expelled.
Apparently a different branch of the ACM (ACM COPE)
would handle the non-publication related issues
and that probably takes time, but the ACM is definitely
not being transparent here.
- Boycot ISCA! Cancel your ACM and IEEE Membership!
- Don't belong to an organization that provides privacy protections
to those found guilty of academic fraud!
- 9 February 2021
-
The paper still has not been retracted by the IEEE.
- It's a bit unclear if/when the IEEE will act on things. Some
of the co-authors on the paper withdrawn by the ACM
are fairly high-ranking IEEE members.
- The authors on the retracted paper also have a few more papers
that got published after the ISCA'19 paper. Is the
15-year publication ban retro-active, or do they get to
keep those?
- 14 February 2021
- After a brief burst of interest on this issue, it's now
all been forgotten again. Swept under the rug, as I'm
sure most IEEE/ACM members prefer.
- At some point as many as 30 people were talking about it on
twitter, amazing (sarcasm).
But due to twitter's idiotic
threading most people didn't see each other's comments.
- Samira Khan posted a few comments that might be relevant.
link1
link2
- 15 February 2021
- Everyone is excitedly posting their ASPLOS papers, which are
hard to take seriously, as the JIC report claims there has
been peer-review issues at ASPLOS recently too.
Were the peer-review miscreants involved with the most recent
ASPLOS reviews? We can't know unless ACM ever releases names.
Should this mean that all ASPLOS publications should live
under a cloud of suspicion? Probably, but none of the authors
seem to care as long as they get to put another publication next
to their name.
- It's about time for the ISCA'21 PC meeting. Will any of the members
be willing to show some honor and boycott things?
- The PC Chair for ISCA'21 is Lizy K. John, who was the PhD advisor
for the author of the retracted ISCA'19 paper,
and thus grand-advisor to Huixiang Chen.
Will she allow ISCA'21 to continue?
- 16 February 2021
- Still think ISCA should be cancelled until there's a major
change of leadership in both SIGARCH and TCCA.
- Sent reminder e-mails to much of the ISCA'21 Program Committee,
just so they can't pull the whole "oh, I never heard about
any of this" excuse they sometimes use to avoid
responsibility. As usual, no responses.
- Really, I think most of them probably feel like the finished
investigation is some sort of validation that it was
a few bad apples, rather than the tip of the iceberg of
the toxic atmosphere present at the conference.
- Why do we care? Why don't we just let them be to sit around
pretending they are relevant and accepting each others papers
into their conference?
Many of us did try that, just letting them fade into
irrelevance. And after the good people left, a student
died in a horrifying manner. And no one at the ACM, IEEE,
SIGARCH, or the ISCA PC seems to care,
or at least they don't care enough to put their publication
count into jeopardy. So the least we can do is try
out best to keep any other students being dragged into
this quagmire.
- 22 February 2021
- SIGARCH posted
a new blog post signed by the SIGARCH board
[Babak Falsafi (Chair),
Natalie Enright Jerger (Vice Chair),
Karin Strauss (Treasurer),
Sarita Adve (Past Chair),
and Board of Directors members
Joel Emer, Boris Grot, Martha Kim and José F. Martínez.]
- They call out the ACM for a lot of the ACM's nonsense.
This letter is much different in tone than the usual ones
from the SIGARCH board, making us suspect it was written
by someone different than normal.
- Before we go rah-rah-rah behind them,
remember this is the group that was in charge
when Torrellas ran the first failed investigation
that delayed this whole thing by months, and they
refused to do anything about that situation.
- 1 March 2021
- There was finally some news on the University of Florida
side of this whole incident:
Article at WUFT
- The revelations give a pretty grim outlook on what it's like
being a grad student in academia today, especially if you have
the kind of advisor who thinks publishing in ISCA is a good
idea.
- And despite all that has happened, the University appears to
still be dragging its feet on doing anything useful
about the whole situation.
- 3 March 2021
- 16 March 2021
-
Retraction Watch published an article
- It's always laughable seeing how the ACM director of publications
continues to be useless.
- Also, it's frustrating how all of the chairs of ASPLOS, ISCA, etc,
like to claim how powerless they are to do anything about
peer review fraud. It almost makes you wonder why they bother
pretending the peer review process is anything but
a sham.
- 25 March 2021
- 15 April 2021
- 26 April 2021
- It is reported that IEEE Fellow Tao Li (Huixiang's advisor)
resigned from U of Florida effective 15 April.
(Gainesville Sun)
(WUFT)
- By resigning, he possibly escapes any further investigation.
- 28 April 2021
- U of Florida releases to GAU the
results of their investigation,
which found Huixiang's advisor had "insufficient evidence"
of a poor work environment.
A separate report INV-21-07 found the advisor
erased a large amount of potentially relevant
evidence before the investigators
had a chance to investigate.
- 1 August 2021
- The ACM has
revoked Tao Li's membership
- It was nice of the ACM to name at least one name and also to
give a lot more details, the peer-review fraud was a lot more
involved than you might think. It also raises issues about
how much went on that this very superficial investigation
missed.
- The ISCA people will probably consider the issue settled now,
an isolated incident. If you read the article though
there were various collaborators (possibly even in
the peer committee) in the peer-review fraud
both at ASPLOS'17 and ISCA'19 and we have not been
told what their fate is. They're possibly the people who
refused to co-operate mentioend in the JIC report.
- It is unclear if Tao Li is still an IEEE Fellow.
Back to the cancel ISCA page