The Legacy of Arecibo for Data Science

If you have been following the science news, you are probably aware that this month an important era in radio astronomy ended, first with the announcement of the decommissioning of the Arecibo Observatory, and second with the collapse of its’ dish.

We cannot do science without instruments and tools that help us collect, analyze data and get results. Biologists have microscopes, astronomers have telescopes, data scientists have computers. But some tools are more ubiquitous than others, and some data can only be collected with very special instruments. One of a kind such instrument is Arecibo.

I had the pleasure of talking with Dr. Sean Marshall, Observatory Scientist at Arecibo and University of Central Florida, and also my former colleague at NASA Frontier Development Lab. Dr. Marshall has been with the Observatory for three years, working on asteroid shape modeling and with asteroid data.

The telescope has been consistently collecting data from for the past 57 years. Some of these are stored on servers at the Observatory, while other data are being stored or backed up by third parties, such as the University of Arizona or University of Central Florida (which is also managing the Observatory together with the National Science Foundation). Depending on the project or range of observations (radio astronomy, ionosphere, asteroid data, etc.), the data is constantly collected and backed-up. In fact, there are currently even 20+ years old datasets collected on tapes that still reside at the Observatory.

On a normal day, the Observatory used to collect in the range of a few hundred KB/sec to 1-2 MB/sec, depending on the type of observation. On the Observatory’s servers, on site, there are currently ~1 petabyte of data and lately data scientists were brought on board with the sole purpose of storing, maintain and archiving Arecibo’s data.

While the collapse destroyed the main dish, parts of the dish are still usable. In the case of asteroid data, there was faster incoming data that it was possible to process and model in real time, therefore the scientists on site will be continuing the work of processing, analyzing the publishing results from these truly astronomical data.

If you are interested in browsing through the data from Arecibo, this is a good place to start.

Currently there are also preliminary efforts to rebuild the Observatory in a new form. If you want to help these efforts, here is a petition that you can sign.