DOKIPY handles datasets and information on datasets. A dataset may be one or more files.
The services offered by DOKIPY differs between the nodes. All nodes (IMR, NPI, METNO) handles metadata and makes these searchable. IMR and METNO do also handle the actual datasets. Currently only METNO offers online access to the datasets. Access to the datasets is controlled by the associated metadata. Some datasets are freely available, while others are protected by username and password for a period. Metadata are synchronised between the DOKIPY nodes facilitating searching through all Norwegian IPY datasets at any of the nodes.
Metadata can be inserted using forms at NPI and METNO nodes. Handling of the datasets differ between the nodes at IMR and METNO. METNO has automated the upload of datasets and monitors the upload process. This require the datasets to be uploaded using standardised file formats (NetCDF/CF). Conformance of files can be tested interactively at the web portal (http://dokipy.met.no/submission.html). Direct upload of files can be done at http://dokipy.met.no/r2/upl/index.php. Contact IMR using the contact details of the node list to receive information on how to upload files to the IMR node.
Metadata are data about the data. Metadata describes who measured what, where and when. Many Many different standards for metadata exist, most data centres use one of the international standards. IPY has chosen Global Change Master Directory (GCMD) Directory Interchange Format (DIF). This is compatible with ISO19115 which INSPIRE require. All metadata for Norwegian IPY datasets are also available in Dublin Core.
More information on metadata are available at:
Funding agencies, projects and scientists all need an overview of who measured what, where and when. This information increase the benefit of observation campaigns, promotes interdisciplinary science and avoids duplication of efforts. It is also required for data management purposes, especially when concerned with archiving datasets. The intention of IPY data management procedures is to ensure that the data collected during this IPY are available to the scientists of the next IPY. Metadata ensures that IPY will be more than individual projects, it connects projects and scientists across projects, disciplines and nations.
A data archive is a data centre with a long term mandata for data management. A data archive ensures that datasets can be tracked and used by future scientists ensuring proper credit is attributed to the scientists actually generating the dataset. Data archives have procedures and systems that ensures secure management of datasets independent of technological and personnel changes.
DOKIPY ensures that datasets entered into the DOKIPY system are handled in accordance with the IPY data policy and its requirements.
Data delivery can be done in two different ways. If you only have a small dataset, you can upload it interactively. If you have a dataset that is regularly updated (e.g. every month, day, week, ...) a dedicated delivery interface can be set up for you. In both situations, only registered users can upload data to the data management system. Furthermore, this system is heavily automated and require preparation of datasets before upload.
Datasets delivered has to be formatted following the NetCDF/CF convention and to contain the global attributes described in the submission page. Files can be uploaded as:
product_nameetc to separate products within a dataset.
While waiting for the updated version of the METAMOD portal solution which
will include a shopping basket feature,
wget may be used
directly towards the THREDDS Data Server.
A statement like the following:
wget -nc -r -l2 -A.nc -I /thredds/fileServer/,/thredds/catalog/
will fetch all files ending with
.nc (-A.nc) recursively (-r)
up to to links away from that page (-l2). The -nc (no-clobber) option
makes sure, existing files are not fetched twice in case of an abort and
restart of wget.
The URL to specify is the address of the file-listing in thredds (catalog.html is optional). This will download all the .nc files, plus the catalog-files (which are not needed, but are small). The URL can be copied from the search results found.
E.g. to download all ITP2 products from this DOKIPY node use:
wget -nc -r -l2 -A.nc -I /thredds/fileServer/,/thredds/catalog/ 'http://thredds.met.no/thredds/catalog/data/met.no/itp02/catalog.html'