Guest post by Ian Lamb, Senior Developer at NYU Health Sciences Library
Early in the development of the data catalog at NYU, one of our team members floated the idea of publishing the dataset records in a Linked Open Data (LOD) format. As the developer on the project, the implementation of this idea would fall to me, so I did what every good developer does in this situation: pretend to know what’s going on, and then Google it when the meeting’s over.
At the time, LOD had not seen much uptake in the tech world, which would explain my ignorance, but libraries, museums, and other institutions were interested in the idea. The phrase refers to data that is “linked” to other data, in a machine-readable way, so that machines can begin to understand what our webpages are really about. For instance, if we have a dataset that was published by the US Department of Energy, we probably already state that in a human-readable way, i.e. in a sentence on the page. But it takes a great deal of effort for a computer to figure out what that means, requiring the use of natural language processing and other artificial intelligence algorithms. LOD allows us to explain this relationship to a computer in a much more direct fashion.
Using a standardized data interchange format (such as JSON-LD), along with a formal schema such as those from Schema.org, we can explicitly link this dataset to the Department of Energy. And when this dataset becomes linked to the US Department of Energy, it also becomes linked to a lot of other information – where the Department of Energy headquarters is, who runs it, other types of research it may produce, where its funding comes from, etc. A machine can follow these relationships and start to form a picture of what the Department of Energy really is, and how our dataset fits into the big picture.
This network of structured, linked data underpins the Semantic Web – a proposed new evolution of the web in which the whole internet would be imbued with meaning in this way. The Semantic Web was first proposed in 2001 in an article by Tim Berners-Lee and colleagues, and the phrase Linked Open Data followed a few years later. But while these concepts have been around for decades, there are still relatively few companies or institutions that publish LOD, and even fewer that utilize it. The Wikipedia article on the Semantic Web has a whole section explaining some reasons for this. Indeed, the most mainstream use of JSON-LD up to this point has been to improve search engine rankings and alter the appearance of a search result in Google.
So imagine my surprise recently, when Google quietly rolled out a search engine for datasets that relies almost entirely on data they’ve harvested from repositories and data catalogs using JSON-LD. Apparently Google has not given up on linked data. They’re just taking their time with it.
Dataset Search, as it is humbly called, is still in beta, and there do appear to be some quirks in the metadata it displays. Take the Asian American Partnership in Research and Empowerment dataset, for example. Here is the record as it appears on our data catalog, and here it is in Google’s search. To figure out why the search result appears so sparse, I fed it into Google’s Structured Data Testing Tool, which shows us the dataset as Google sees it based on our linked data output. (The JSON-LD itself is embedded in the page source, and you can see it by scrolling the left-hand frame down to line 39.)
The tool shows that Google is receiving lots of metadata about the dataset itself, including authors, keywords, associated citations, file formats, and more. It also shows the various relationships we’re specifying for it: the dataset record isPartOf a data catalog, the provider of which is the NYU Health Sciences Library, which has a parentOrganization of the NYU School of Medicine, and the url for that entity is…etc. But out of all this data, only the authors and the description currently appear on Google’s search result. There is a link to some citations there, but they’re not the ones we’ve specified – they appear to simply be searching for the name of the dataset in Google Scholar. Similarly, Google lists the dataset as provided by New York University, which, while correct in a sense, is not the entity we’ve specified as the provider or even the parent organization of the provider.
Some of this discrepancy is because we’ve chosen slightly different field names than Google has. For example, we are listing the file format as an encodingFormat attached to the main dataset entity, but using another Google tool, I’ve found that they expect any encodingFormats to be listed under a distribution, which would look like this:
“name”: “Asian American Partnership in Research and Empowerment”,
"description": "Project AsPIRE (Asian American Partnership in Research and Endowment) was a community-based…",
Other fields, such as Keywords, don’t appear to be supported at all. At this stage we can’t know if they’re going to be adding any additional fields or how much of this is going to change, but we have identified a few small alterations we can make to our JSON-LD output that should improve our results.
So there are some kinks to be worked out for sure, and it remains to be seen how much Google is going to invest in this effort and how aggressively they plan to publicize it. But it does seem to represent a significant move into this space. And if it does pick up steam, there is a very substantial upside for any institution using our catalog code: instant inclusion in a worldwide, highly-visible dataset search engine.