7 Large Data Sets

Contributors: Margaret O’Brien, Corinna Gries, Mark Servilla

7.1 Introduction

Data entities are kept offline when they are too large to be handled easily by the HTTP protocol, are expected to be rarely requested, and can be mailed on an external drive. If you suspect your data fall into this category, contact EDI for advice (). Below are recommendations for the EDI repository’s handling of data packages that have an offline component.

7.1.1 Background

Standard practice is to handle data entities (both upload and download) via the HTTP protocol, using a URL. However, for very large datasets HTTP can fail due to physical limits. The limit for “too large” is somewhat subjective; EDI’s current limit for datasets that are “too large for HTTP” is 100GB (all data and metadata).

7.2 Recommendations for data packages

7.2.1 Physical Storage

  • The use of a Solid-state Drive (SSD) is strongly recommended for all offline data storage. The SSD should be formatted using one of the following file systems: 1) exFAT, 2) NTFS, or 3) ext4. Each of these file systems can accommodate individual file sizes greater than 1TB.

  • Add data to external drive in native (non-compressed, non-tarred, non-zipped) format, deliver to EDI (e.g., by physical mail).

  • EDI will store three copies, one external hard drive each in New Mexico and in Wisconsin, one copy in general EDI backup cloud storage.

  • Please mail one copy each to:

Attn: Mark Servilla UNM Biology, Castetter Hall 1480 MSC03-2020, 219 Yale Blvd NE Albuquerque, NM 87131-0001

Attn: Corinna Gries University of Wisconsin Center for Limnology 680 North Park Street Madison WI 53706-1413

7.2.2 Data package

  • The external hard drive should contain at least two entities: the data (which will be offline) and an inventory or manifest that describe the contents of the external hard drive.
  • Content of the manifest (inventory of holdings) would be dictated by the type of data entity. The manifest will be available as an online entity (through the EDI Data Portal) so that potential requestors can evaluate the offline resource before requesting it.
  • Suggested columns are:
    • Filename(s)
    • Format (netCDF, tabular csv, etc.)
    • Start_datetime
    • End_datetime
    • Location_lat
    • Location_lon
    • (other params the PIs may feel are essential)
    • Checksum

7.2.3 Package Metadata

(in EDI metadata template and converted to EML - generally, as for any data package)

  • Abstract: describe the collection generally. If individual files require specific software to read, provide the name of that software.
  • Creators
  • Contact (will be responsible for sending out copies as requested.) positionName: EDI Repository Manager Email:
  • Methods - detailed collection/generation methods for the offline data entities. Detailed information for re-using the data. (May instead be included in the manifest table if different for different offline files.)
  • Data Entities
    • Offline Entity:
      • Describe as you would for an online resource. Restate the software needed to read the individual files if this is important to a user. See Table 1 and Sample XML.
    • Manifest (inventory of the offline holdings)
      • Column descriptions as for any data table

7.2.4 EML

In addition to basic resource-level metadata, at least two entities should be described:

  • Manifest (inventory) should be a tableEntity: will be the online entity and described as all
  • Offline entity:
    • Fill out high-level fields as for an online resource. Restate the software needed to read the individual files if this is important to a user.
    • Distribution node will be offline (See Table 1, code block) Table 1. Three required fields for an offline distribution

physical/objectName As for any entity, this is the name of the file or data object
dataFormat/ExternallyDefinedFormat/formatName The name of the format the data object is in. If there is a special compression applied, list it here.
distribution/offline/mediumName Instead of a data URL, you will have an offline distribution node. The name of almost all offline media is “external drive”, because that is how you will deliver the data to a requestor.

7.3 References

7.3.1 EML documentation


Look for the PhysicalDistributionType

7.4 Potential Issues

  • SSD formatting (eventually, whatever we use, it will become unusable).
  • Even with cloud storage, eventually a binary format will become unusable.