Sourced from: http://ivory.idyll.org/blog/2019-themes-caos-open-science.html
The “Critical Assessment of Open Science” meeting, or CAOS, was
convened by Sage Bionetworks in New Orleans in early February. About
30 open science practitioners and advocates were invited by Sage to a
day long meeting in New Orleans to consider the last 10 years of
progress and failures in open science. The meeting was attended by
scientists, policy experts, funders, and others. While the emphasis
was on the biosciences, many themes were discussed in a broader
context of all of science.
You can read more about the motivation for the meeting, and see a
series of summary blog posts,
here.
This post is my attempt to summarize the entire meeting, based on notes
I took during the meeting.
The meeting was organized in a series of
“call and response”
engagements, in which two participants “called” for 5 minutes to one
of five broad themes, and then a responder summarized, contextualized,
and responded to their call. There were multiple such calls &
responses in each session, for about 5 sessions. Audience
participation was lively!
The meeting was held under
Chatham House rules,
so below I am reporting my takeaways without reference to specific
individual comments or revealing details. There should be some form of
publication output in the future so you can see who attended and get a
more global view of the meeting; I’ll link to that below when it is
out.
Thank you to Sage Bionetworks for coordinating this meeting & inviting me!
Main themes that emerged (for me)
We hoped that open science would lead to new and better practices; what
we too often got was practices that fed into the same broken system.
As the value of analytics and data becomes ever more apparent, there
is ever more interest by commercial interests in capturing that value
in closed systems. Often, the data creators and/or owners seem to be
unaware of this capture, especially when the data is secondary to
their primary mission (e.g. in universities). This lack of awareness
Has Consequences.
Governance and sustainability of open institutions (especially open
source projects) is on a lot of people’s minds. Sage has a large
team focused on this! (John Wilbanks says “call me!”)
We talked a fair bit about the challenge of convincing individuals and
groups that increased opportunity for unpredictable serendipity
was worth giving up predictable (but smaller) gains in
fame/power/money.
The invisibility of successful “open” came up repeatedly – the modern
data science ecosystem is built on R and Python, preprints in the life
sciences, open & FAIR data, and open source especially. That
successful open practices achieve near instant adoption is wonderful;
that they are not highlighted as successes of open in the open science
community is unfortunate; and their invisibility means that their
sustainability is often not strongly considered.
(You can see a longer blog post by me on this topic, here.)
It was great to see multiple statements about how the idea of one
consortium/community building THE platform for analysis in an area was
a non-starter. Functional interoperability, collaboration, and
ecosystem thinking within and across platforms is seen as critical,
even by the most senior researchers.
In concert with that, I see that every functional system is a
compromise between various requirements and design
considerations. Therefore building multiple differently functioning
systems is a good ecosystem bet.
Several different people referred to the increased attack surface
that open practices offer: e.g. by making your methods and data open,
you increase the ability of others to attack your conclusions. While
this is an important aspect of open science, it is also something that
discourages everyone, with disproportionate negative impact on already
marginalized populations. Sharing within “club” structures, or gated
communities, was seen as one possible solution.
We noted the need for & challenge of placing “do no harm” restrictions
on use and reuse of data; community codes of conduct were discussed as
one example of a governance structure that (combined with
not-entirely-open communities) could enforce such restrictions.
Diversity and inclusion was a frequently mentioned topic. Lack of
diversity in communities can be seen as empirical evidence of missing
structure in communities that is not clearly visible from within; I
think this is important when it comes to formal governance discussions
that can externalize internal culture (hopefully accurately).
Another interesting theme was the extent to which some saw that
grassroots communities of practice could be an antidote to the
“monkey’s paw” or “shitty genie” of requirements generation. Often,
engineers building infrastructure want detailed use cases and
requirements specification, which then leads to the wrong thing being
built (and the associated blame), while if the engineers are brought
into the community of practice they are more likely to build the right
thing due to shared understanding and iterative/continuous
participation.
The challenge of analyzing all the interesting data sets was
frequently mentioned. While not discussed at the meeting, in my view,
training is a way to bring prepared minds and hands to tackle the
analysis of interesting data sets. This training needs to be built in
rather than bolted on to projects, however.
My own POV: the critical role of communities of practice
Again and again, I saw that communities of practice presented a key
ingredient to solutions for problems in governance, training,
infrastructure, methods, etc. Communities of practice bring the people
to the problems! Fundamentally, I think open systems do not work
without a community of practice underpinning them.
Creating, growing, and sustaining these communities is, I think, one
of the most important tasks to be tackled. More on that as I have
time to write.
Concluding thoughts
One of the organizers closed out the meeting by asking everyone to
highlight one theme that surprised and/or dismayed them. This was a
productive if depressing way to extract essential takeaways!
“The cavalry isn’t coming.” One of the more sobering conclusions from
this part of meeting was that, given the seniority of the people in
the room, we had no one but ourselves to blame for failing at open in
the next decade. If we couldn’t figure out how to coordinate and
incentivize open, then it was unlikely that someone else would step in
to help us out. We are the cavalry. (And existing, closed,
institutions are more resilient than we realized.)
Consumers are often very happy to trade data for convenience. This is a
challenge for open!
Open science can be weaponized by opponents of science, e.g. reproducibility
challenges can lead to the conclusion that all science is wrong; there
are many politicians eager to attack science. The dangers of further
deligitimizing science in the eyes of the world are real!
While scientists always start in and often revert to competitive mode,
they can also switch to cooperative mode with ease, given the proper
incentives and structure. (I personally recommend reading Kathleen
Fitz’s book Generous Thinking, which focuses on this issue!)
A generational (?) concern was that DIY biology will eat all of biology,
and that this meeting could be viewed as a bunch of PDP-11 engineers
discussing the intricacies and importance of time sharing system design.
I personally think millenials are more sophisticated about data ownership,
more invested in sharing (and more sophisticated about its tradeoffs), and
are likely to seriously upset current apple carts, but I’m an optimist :).
There was a repeated concern that open biomedical science has to
translate into better outcomes, and a shared concern that open science
is an ideology built on practices that don’t really work 80% of the time.
My own (depressing) conclusion was that it is not possible for open to
be truly open, and that completely open institutions are extremely
vulnerable to attack (for my previous thoughts on this in open source
projects, see
“How open is too open?”). There
are gates that must be kept (hodor)! I’ll expand on this theme in
another blog post when I have time!
In general, I’m happy to expand on themes as time permits, if people
have questions!
Immediately after writing this, I happened to revisit Denisse
Alejandra’s article,
“Reimagining Open Science Through a Feminist Lens”,
and I was encouraged by the overlap and relevance of a lot of what was
discussed at the CAOS meeting to this reimagination!
–titus