Providing scentists easy access and good services to JGI's large datasets
Scientists all over the world come to the JGI data portal to look for and download data for their research. We interviewed 114 of them individually. Among them were PIs (principal investigators), bioinformaticians, biochemical engineers, geneticists, biologists, professors, lab researchers, and PhD students.
These individuals may use the data to assist them in various fields of research, such as climate and insect interaction, to wildfire analysis, sorghum production, observation of the Yellowstone environment, and even the comparison of 2 microbial genomes that can convert carbon dioxide into chocolate.
As a project of the US Department of Energy, JGI is set to to advance genomics in support of clean energy generation and environmental characterization/cleanup. Operated by UC Berkeley, JGI's mission is to provide the scientific community at large with access to integrated, high-throughput, high-quality sequencing, DNA design and synthesis, metabolomics and computational analysis capabilities that enable systems-based scientific approaches to their challenges.
JGI's projects involve many important multicellular organisms, microbes and communities of microbes called metagenomes related to the DOE mission areas of bioenergy, which help us to understand global cycles such as the carbon cycle, and biogeochemistry.
We kicked off the project with 6 weeks of discovery and framing right before the COVID-19 lockdown. So the team conducted workshops, interviews, and design sprints both remotely and in person at the Berkeley Lab campus. After this, ongoing research efforts such as card sorting, prototype testing and usability testing continued to refine the understanding of the problem space and continue yielding evidence and data for decision making. These are the top pain points we discovered:
Analytic tools like Google analytics and Honeycomb revealed that the dropout rates were high. User interviews showed that users often gave up and left the portal because they couldn't easily find the data they were looking for.
Nearly all the data on the Genome Portal was available to the public, but not all of which was available to download immediately. Downloading large amounts of data via command line was especially challenging for even the most experienced researchers.
Reliance on NERSC (National Energy Research Scientific Computing Center), the current infrastructure, also resulted in unexpected downtime, not just maintenance down time.
Data provided to the Genome Portal reflected the structure of JGI. Users needed to understand the organizational structure of JGI to be able to get around in the portal.
The datasets were messy. Users often found themselves having to clean them up in MS Excel after the download. They also weren't structured to support newer scientific approaches to research.
Search often happened on kingdom portals, then users came to the Genome Portal to download, which was extremely frustrating.
JGI's main kingdom portals include IMG (Integrated Microbial Genomes and Microbiomes), Phytozome, PhycoCosm, MycoCosm, and GOLD (Genomes Online Database). Data Portal is the unified access to all JGI genomic databases and analytical tools we're building to replace the current Genome Portal.
To ensure the genome data is available to the public and the entire scientist community, so that they can view and download multiple organisms at a time, even across multiple datasets. The current system only supported single organism download.
Search functionality was the first thing many users asked for. Searching across multiple kingdoms didn't exist, neither did we have the foundation work to support it until phase 1 wrapped up. With better structured datasets and clean metadata, now we had the infrastructure ready.
View, refine, & download multiple organisms, including plant, fungi, algae, microbe & metagenome + baseline metadata work
Fair (Findable, Accessible, Interoperable, Reusable) is what JGI promised to its users. Search addressed the first 2 points.
Starting from reviewing high school biology on Khan Acadamy, we tried to fit our tiny feet in the shoes of the scientists who came to JGI for its genome data, and we tried to understand what they were looking for and what stopped them from their work, and how we might help them.
Started with single organism download, we created robust "filter and refine" process prior to downloading. Displaying all the available plant genomes in a tree made sense at the beginning.
With CI/CD and feature flags, we were able to test-in-production, conducting usability tests early and frequently. After multiple iterations, we became familiar with the underlying metadata and their strengths & weaknesses. We started to make suggestions, request updates, and share feedback with different kingdom teams.
We also realized that the Phylogenetic tree design would not scale to 1,000 fungal or 15,000 microbial datasets. The findings and insights from usability tests also supported us to switch from the tree view to the list view, for it reduced the number of steps scientists needed to view or download.
In user interviews, we heard that users had to click too many times before they could locate and download files. This can be attributed to the Genome Portal's files being organized in a complex folder structure.
Try it yourself
An important insight that came out of our discovery activities was that JGI’s presentation of projects and proposals did not match the mental model of users. They didn’t know what the various IDs meant, and they didn’t understand how the records were organized. A hypothesis that came out of this insight was that re-orienting our presentation around organisms would resonate more with the users. We validated this hypothesis through user interviews and research.
Try it yourself
During research, we heard about challenges in understanding file names and what the files or organisms might contain. Nesting files under each organism helped to provide the much-needed context, which also progressively disclosed the detailed info (without overwhelming them upfront).
The simple color palette can help the users focus on the selection they made, the most important actions, and the availability of the data.
We used Figma to create interactive prototypes for usability testing, exploring interactions and transitions, and to simulate the real-life experience we're aiming to provide.
Try it yourselfThe project sailed rapidly and smoothly at this point. The product and the database had matured enough that we could finally address the question almost every scientist user was asking, why can't we search?
Remember we had filters from Phase 1? Filters are important when dealing with huge sets of data. However, user feedback indicated that the filters took up too much room; but when we tucked them under a filter bar, users sometimes missed them. So half jokingly, we started with a strong visual statement to make sure the filter bar stood out, a rainbow! From there, we explored 20+ different ways to make the bar visually stand out.
We also needed to provide a way for users to enter a new search or modify an existing one. We wanted the users to be able to access both search and filters anywhere on the page, even on a long scroll. With limited space at the top of the page AND on the sticky header, the team agreed to try what we called a bento-box. The bento-box combines all the elements from both search and filters into compartments holding more discoverable info.
The new homepage is focused on search, while everything else is secondary. It's a fully responsive web page: users who are here for genome data can have a good experience with mobile devices. (Due to the complexity of genome data though, we couldn't extend this approach to the nested genome page.)
Try it yourself
An old Chinese proverb has it, "If you give a man a fish, you feed him for a day. If you teach a man to fish, you feed him for a lifetime.” Teaching the JGI team how to fish was part of our deliverables. We demonstrated how Lean Agile and Human Centered Design work, and we made sure we fostered a sustainable team. The project was very productive and happy. With the proper expertise represented on the team, it met and exceeded the targeted goals.
The JGI Leadership Team was just amazing. They doubled down AND advocated for Human Centered Design and iterative software development. They are the dream client that any team would want to work with.
The team had been paying attention to CRISPR, gene editing, and the work Jennifer Doudna and Emmanuelle Charpentier were doing before they were awarded the 2020 Nobel Prize in Chemistry. We're thrilled with the news.
Many non-designers on the team claimed design studio was their favorite activity. We involved everyone and always had a blast. Insights & Action came in second, which happened after a run of either user interviews or usability tests.
We built a design system in Figma and a component library in Storybook within a few weeks.
Check it outThe biggest win was the partnership between the "client" and the "vendor". There were no "client" or "vendor" in the team, only people who worked together, trusted one another, and tried to deliver the best possible products and services.
To be honest, we didn't have a strong frontend presence on the team, some of the implementations were far from state of the art. The JGI team got it, they promised to carry on the torch and keep pushing the products to be better and better.