Working with netCDF files

For anyone using netCDF files (e.g from GMT software), Scilab can readily now read and write such data using the HDF5 functionality. Use h5dump to se what datasets are present.

A short code snippet to help show how to read data in and get variables built:

clear

NC=h5open(‘faa_test_1m.grd’); // open netCDF file and assign an ID
latitude=h5read(NC,‘/lat’);
longitude=h5read(NC,‘/lon’);
zdata=h5read(NC,‘/z’);
h5close(NC)

xmin=min(longitude); xmax=max(longitude);
ymin=min(latitude); ymax=max(latitude);
zmin=min(zdata); zmax=max(zdata);

a=gcf()
a.color_Map=jet(64);
a.figure_size=[800 800]
Sgrayplot(longitude,latitude,zdata,zminmax=[-50 50])
colorbar(-50,50)
isoview on

colbar1 = gce();
ylabel(colbar1, ‘Free-air gravity anomaly’)
title(‘mGal’)
colbar1.title.font_size = 3;

Hope this is useful. Tested with grid files generated from GMT v6.5.0

Lester

2 Likes

Hello Lester,

This example is worth becoming a demo don’t you think (there is currently a single item in demos/HDF5). This would be even better if the faa_test_1m.grd could be retrieved with http_get.

S.

1 Like

Hi Stephane,

I will work out an example demo for this as netCDF is a file format that is common in the scientific community, and not greatly different from HDF5.

Need to work the output back to a readable netCDF structure, which is a little fiddly as it is a compound object with attributes and dimensions to assign and link to the data itself.

The file 'faa_test_1m.grd’ isa one I generated from GMT software. I can e-mail this to you if you know a place to locate it for a demo?

Lester

Is the original data source under netCDF format or are you processing this data in order to create a netCDF file that is later on read in Scilab ?

S.

Hi Stephane,

The file 'faa_test_1m.grd’ is the file generated in GMT (v6.5.0) and then read into Scilab via HDF5. The file is not processed, however it can be manipulated within Scilab.

It is important to show that one can go the other way from Scilab and create a “netCDF” compatible type structure via h5write etc. The filename type seems to only allow the output to be like filename.sod, it does not accept (as far as I can see) any other extension eg. filename.grd or filename.nc

HDF5 and netCDF are very close in terms of file structure when looking at h5dump(‘filename’).

Lester

Ok in that case we can host the grd file in the same folder as the demo file. Please have a look at scilab/modules/hdf5/demos · main · scilab / scilab · GitLab. The best way to add a new demo would be to create a merge request adding the demo script, the data file and the modification of hdf5.dem.gateway.sce. You can start by editing and adding files in your local Scilab application tree in

SCI/modules/hdf5/demos/

S.

Getting the data written out to a grid structure format that is readable as HDF5/netCDF is tricky. The datasets and attributes can be written and all of that information is stored in the output structure.

The challenge is to link the longitude (x) and latitude (y) datasets to the z-dimensions, which requires the need to construct a compound reference list; at the moment it is not clear how this works.

Cannot see a way to replicate this structure:
Longitude dataset (as an example)
ATTRIBUTE “REFERENCE_LIST” {
DATATYPE H5T_COMPOUND {
H5T_STD_REF_OBJECT “dataset”;
H5T_STD_I32LE “dimension”;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): {
DATASET 4111 /z,
1
}
}
}
Zdata has a similar issue:
DATASET “z” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 1200, 1200 ) / ( 1200, 1200 ) }
STORAGE_LAYOUT {
CHUNKED
SIZE 2973243

       ATTRIBUTE "DIMENSION_LIST" {
        DATATYPE H5T_VLEN { H5T_STD_REF_OBJECT }
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): (DATASET 817 /lat), 
        (1): (DATASET 480 /lon)
        }
     }

Running the code will generate a file with three datasets corresponding to longitude, latitude and zdata. Getting netCDF data into Scilab is the easy part; I guess one could eithe ESRI ascii raster format or some image structure (need to think on that one). If anyone is familiar with HDF5 that would be good!

Lester

The code so far:

NCout=h5open('test.sod', 'w');

// Root level global attributes
h5attr(NCout,'/','Conventions','CF-1.7')
h5attr(NCout,'/','_NCProperties','version=2,netcdf=4.8.2-development,hdf5=1.10.5')

// write datasets to the root group
// x-axis
h5write(NCout, '/lon', longitude);
h5attr(NCout,'/lon','long_name','longitude')
h5attr(NCout,'/lon','units','degrees_east')
h5attr(NCout,'/lon','standard_name','longitude')
h5attr(NCout,'/lon','axis','x')
h5attr(NCout,'/lon','CLASS','DIMENSION_SCALE')
h5attr(NCout,'/lon','actual_range',[xmin xmax]')

// y-axis
h5write(NCout, '/lat', latitude);
h5attr(NCout,'/lat','long_name','latitude')
h5attr(NCout,'/lat','units','degrees_north')
h5attr(NCout,'/lat','standard_name','latitude')
h5attr(NCout,'/lat','axis','y')
h5attr(NCout,'/lat','CLASS','DIMENSION_SCALE')
h5attr(NCout,'/lat','actual_range',[ymin ymax]')

// Data
h5write(NCout, '/z', zdata);
h5attr(NCout,'/z','_FillValue',%nan)
h5attr(NCout,'/z','actual_range',[zmin zmax]')

// Close the file (ID)
h5close(NCout)

OK, digging deeper into this, it would appear that the HDF5 implementation in Scilab is not complete as far as writing data out and does not have the ability to create complex objects.

h5ref - needed to generate internal dataset references to lon, lat and z
There is no clear workaround other than output to some other format.

Lester

Hi Stephane,

I have prepared the demo file for reading the netCDF file and making the plot. Updated the hdf5 demo list (all works) and put the files in place to test locally. All works apart from the colormap is the matplot default, it seems to ignore the jet scheme I have chosen.

Not sure what is wrong as it is fine when run as a normal script! I have tried with and without absolute path definitions and the same result.

Lester

hdf5_demo_output

// Scilab ( https://www.scilab.org/ ) - This file is part of Scilab
// Copyright (C) 2024 - Lester Anderson
//

function netCDF_demo()

filename = 'Read_netCDF.dem.sce';
demo_viewCode(filename);

// open netCDF file (read-only)
// Keep original file format intact
NC=h5open('faa_test_1m.grd','r');
//NC=h5open(SCI+'/modules/hdf5/demos/faa_test_1m.grd','r');

ncfile = NC.root;
Datasets = ncfile.Datasets; // read datasets present 

latitude=h5read(NC,'/lat'); 
longitude=h5read(NC,'/lon');
zdata=h5read(NC,'/z');

h5close(NC)

xmin=min(longitude); xmax=max(longitude);
ymin=min(latitude); ymax=max(latitude);
zmin=min(zdata); zmax=max(zdata);

a=gcf()
a.color_Map=jet(64);
a.figure_size=[800 800]
Sgrayplot(longitude,latitude,zdata,zminmax=[-50 50])
gce();
xlabel('Longitude')
ylabel('Latitude')
colorbar(-50,50) // the actual data range differs
isoview on

colbar1 = gce();
ylabel(colbar1, 'Free-air gravity anomaly');
title('mGal');
colbar1.title.font_size = 3;

endfunction

netCDF_demo();
clear netCDF_demo;

The updated demo listing:

// Copyright (C) 2011 - DIGITEO - Sylvestre Ledru
//
// This file is released under the 3-clause BSD license. See COPYING-BSD.


function subdemolist = demo_gateway()

    demopath = get_absolute_file_path("hdf5.dem.gateway.sce");
    _("HDF5");  // lets gettext() harvesting it
    add_demo("HDF5", demopath + "hdf5.dem.gateway.sce");

    subdemolist = [_('Ring resonator (HDF5 data source)')     ,'resonator.dem.sce' 
                   _('Read and plot data from a netCDF file') ,'Read_netCDF.dem.sce' ]

    subdemolist(:,2) = demopath + subdemolist(:,2);
endfunction

subdemolist = demo_gateway();
clear demo_gateway;

The structure of a Netcdf4 file in Scilab HDF5

Netcdf_HDF5_structure.txt (10.0 KB)

Same data using netcdf 4.9.2 tools:

ncdump -h faa_test_1m.grd
netcdf faa_test_1m {
dimensions:
lon = 1200 ;
lat = 1200 ;
variables:
double lon(lon) ;
lon:long_name = “longitude” ;
lon:units = “degrees_east” ;
lon:standard_name = “longitude” ;
lon:axis = “X” ;
lon:actual_range = -10., 10. ;
double lat(lat) ;
lat:long_name = “latitude” ;
lat:units = “degrees_north” ;
lat:standard_name = “latitude” ;
lat:axis = “Y” ;
lat:actual_range = -10., 10. ;
float z(lat, lon) ;
z:long_name = “z” ;
z:cpt = “@earth_faa.cpt” ;
z:_FillValue = NaNf ;
z:actual_range = -104.800018310547, 455.225006103516 ;

// global attributes:
:Conventions = “CF-1.7” ;
:title = “” ;
:history = “gmt grdcut @earth_faa_01m_p/ -Gfaa_test_1m.grd -R-10/10/-10/10 -V” ;
:description = “” ;
:GMT_version = “6.4.0 [64-bit] [MP]” ;
:node_offset = 1 ;
}

Creating the equivalent in HDF5 of the following Octave code is key:

nccreate(‘Octave_test1b.grd’, ‘lon’, ‘Dimensions’, {‘lon’, nx}, ‘Format’, ‘netcdf4’);
nccreate(‘Octave_test1b.grd’, ‘lat’, ‘Dimensions’, {‘lat’, ny}, ‘Format’, ‘netcdf4’);
nccreate(‘Octave_test1b.grd’, ‘z’, ‘Dimensions’, {‘lon’, nx, ‘lat’, ny}, ‘Datatype’, ‘single’,‘Format’, ‘netcdf4’);

This Octave code - ‘Dimensions’, {‘lon’, nx, ‘lat’, ny} can be replicated in Scilab as:
dimensions = struct(‘lon’,length(longitude),‘lat’,length(latitude));

Netcdf4 is a simpler structure compared to HDF5, but not straightforward to build dimension referencing and the compound structure for lon, lat and z.