making a 2D plots for aeronet data

5 posts / 0 new
Last post
Masaru Yoshioka
making a 2D plots for aeronet data

Hi,

I want to make a global 2D plot for monthly averaged AERONET data concatenated for all available (178) stations for July 2008. This should look like coloured dots on the global map. This was successful for the old data. I did;

cis plot AOT_440:AOT_440_allstations_2008-07_monave_nco_obs.nc &

And here is the resulting plot: https://drive.google.com/file/d/1N4apvxKdhMQhk0RL_pZoC30L8-GMBEZ_/view?u...
Because cis aggregate did not give me a correct results, I used NCO to calculate monthly averages (see http://www.cistools.net/node/5286) and create this data.

Now that James O'Neill figured out how to correctly calculate monthly averages using cis aggregate (follow the link above again), I created the equivalent data using cis aggregate. This data is AOT_440_allstations_2008-07_mav_obs.nc . Although this turned out to be far more complicated than the method using NCO (mainly because cis aggregate completely changes the dimension structure of the data... Is there a way to prevent this?), the benefit of using cis aggregate is that it gives not only the averages but also standard deviations and number of data values. Also our aim is to establish a standard method of data processing using CIS. Anyway, I did the same thing as above for this data but cis plot created a line plots with stations in x-axis and values in y axis.

(I tried "--type=contour" option but this failed with a message saying that Z must be a 2D array. I can understand that but then what type of plot should I specify to make the same plot as the one above? With "--type=scatter" it created a scatter plot similar to the line plot which is not what I want.)

I compared both data and tried to make the second data similar to the first one by;

- removing all spatial and temporal dimensions
- adding a record dimension "obs" instead
- removing bounds variables like longitude_bnds
- removing variables attributes related to bounds like: longitude:bounds = "longitude_bnds"
- removing other suspicious attributes like cell_methods and -FillValue from the main variables
- removing the very long history in global attributes

but the result is the same. There still are some differences in the headers in these two files. Below I copied both of them. I copied both data to /home/users/myoshioka/temp/ as well. Can you see or guess what is preventing cis from making a 2D plot for the second data? If possible please could you also answer two questions within the text above (marked by "?")?

Thank you.
Masaru

netcdf AOT_440_allstations_2008-07_monave_nco_obs {
dimensions:
obs = UNLIMITED ; // (178 currently)
variables:
double longitude(obs) ;
longitude:standard_name = "longitude" ;
longitude:units = "degrees_east" ;
longitude:cell_methods = "record: mean obs: mean" ;
double latitude(obs) ;
latitude:standard_name = "latitude" ;
latitude:units = "degrees_north" ;
latitude:cell_methods = "record: mean obs: mean" ;
double altitude(obs) ;
altitude:standard_name = "altitude" ;
altitude:units = "meters" ;
altitude:cell_methods = "record: mean obs: mean" ;
double time(obs) ;
time:standard_name = "time" ;
time:units = "days since 1600-01-01 00:00:00" ;
time:calendar = "gregorian" ;
time:cell_methods = "record: mean obs: mean" ;
double AOT_440(obs) ;
AOT_440:_FillValue = -999. ;
AOT_440:units = "1" ;
AOT_440:long_name = "AOT_440" ;
AOT_440:missing_value = -999. ;
AOT_440:history = "2017-12-22T17:12:48Z Subsetted using limits: time: [149201.0, 149231.999988]" ;
AOT_440:cell_methods = "record: mean obs: mean" ;
int obs(obs) ;
obs:standard_name = "obs" ;
obs:units = "index" ;

// global attributes:
:source = "CIS1.5.4" ;
:nco_openmp_thread_number = 1 ;
:history = " (very long. omitted here) " ;
:NCO = "\"4.5.5\"" ;
:history_of_appended_files = "Tue Feb 20 17:11:23 2018: Appended file obs_Agoufou.nc had no \"history\" attribute\n" ;

#####################################################
netcdf AOT_440_allstations_2008-07_mav_obs {
dimensions:
obs = UNLIMITED ; // (178 currently)
variables:
double AOT_440(obs) ;
AOT_440:long_name = "AOT_440" ;
AOT_440:units = "1" ;
double AOT_440_num_points(obs) ;
AOT_440_num_points:long_name = "Number of points used to calculate the mean of AOT_440" ;
double AOT_440_std_dev(obs) ;
AOT_440_std_dev:long_name = "Corrected sample standard deviation of AOT_440" ;
AOT_440_std_dev:units = "1" ;
double altitude(obs) ;
altitude:units = "meters" ;
altitude:standard_name = "altitude" ;
altitude:cell_methods = "altitude: mean" ;
double latitude(obs) ;
latitude:axis = "Y" ;
latitude:units = "degrees_north" ;
latitude:standard_name = "latitude" ;
latitude:cell_methods = "latitude: mean" ;
double longitude(obs) ;
longitude:axis = "X" ;
longitude:units = "degrees_east" ;
longitude:standard_name = "longitude" ;
longitude:cell_methods = "longitude: mean" ;
int obs(obs) ;
obs:standard_name = "obs" ;
obs:units = " " ;
obs:description = "index for stations" ;
obs:caution = "dont set units = index or you will get a warning" ;
double time(obs) ;
time:axis = "T" ;
time:units = "days since 1600-01-01 00:00:00" ;
time:standard_name = "time" ;
time:calendar = "gregorian" ;
time:cell_methods = "time: mean" ;

// global attributes:
:Conventions = "CF-1.5" ;
:nco_openmp_thread_number = 1 ;
:history_of_appended_files = "Thu Feb 22 10:51:51 2018: Appended file obs_Agoufou.nc had no \"history\" attribute\n" ;
:NCO = "\"4.5.5\"" ;
:history = "Thu Feb 22 15:06:43 2018: ncatted -a history,global,d,, AOT_440_allstations_2008-07_mav_obs.nc" ;

james_oneill
Workaround: plotLatLon plugin

Hi Masaru,

As you know, I have emailed you a workaround for this issue, but am also adding details here in case anyone else finds it useful.

Running "cis info <filename>" on your first (nco-created) file shows that CIS recognises the following variables in the file:
AOT_440
altitude
longitude
time
latitude
obs
whereas running the same command on your second (cis-created) file gives no output. Oddly, if you include the variable name, e.g. "cis info AOT_440:<filename>", you do still get information about that variable for the second file. I'm not sure why this is yet, but my suspicion is that the fact the variables don't appear when you just type "cis info <filename>" is what is causing CIS to not be able to plot the 2d lat/lon scatter plot you want.

Until we figure out why this has happened, you can use the following workaround, which uses a plugin to read in the latitudes and longitudes from the ungridded netCDF file explicitly:
- Save the python code below to a file and put it somewhere on the machine you want to run CIS on (you might have to add code indentation back in; the preview of this message is showing that all leading space is removed)
- Tell CIS where the plugin is by adding the following environment variable (type this at the command line):
$ export CIS_PLUGIN_HOME='full/path/to/plugin/directory'
- Run the CIS plot command as before, but specify the product as plotLatLon, i.e.:
$ cis plot AOT_440:AOT_440_allstations_2008-07_mav_obs.nc:product=plotLatLon

Thanks,
James

====================

from cis.data_io.products.AProduct import AProduct
from cis.data_io.Coord import Coord, CoordList
from cis.data_io.ungridded_data import UngriddedData
from cis.data_io.ungridded_data import Metadata
import netCDF4
import numpy as np

class plotLatLon(AProduct):

def get_file_signature(self):
return [r'.*.pll']

def _create_coord_list(self, filenames, data=None):
if data is None:
data = {} #initialise data dictionary
inData=netCDF4.Dataset(filenames[0]) #open netCDF file
data['longitude']=np.array(inData.variables['longitude']) #extract longitudes
data['latitude']=np.array(inData.variables['latitude']) #extract latitudes
inData.close() #close netCDF file
coords = CoordList() #initialise coordinate list
#Append latitudes and longitudes to coordinate list:
coords.append(Coord(data['longitude'],Metadata(name="longitude",long_name='longitude',standard_name='longitude',shape=(len(data),),missing_value=-999.0,units="degrees_east",range=(-180, 180)),"x"))
coords.append(Coord(data['latitude'],Metadata(name="latitude",long_name='latitude',standard_name='latitude',shape=(len(data),),missing_value=-999.0,units="degrees_north",range=(-90, 90)),"y"))
return coords

def create_coords(self, filenames, variable=None):
return UngriddedCoordinates(self._create_coord_list(filenames))

def create_data_object(self, filenames, variable):
data_dict = {} #initialise data dictionary
inData=netCDF4.Dataset(filenames[0]) #open netCDF file
data_dict['longitude']=np.array(inData.variables['longitude']) #extract longitudes
data_dict['latitude']=np.array(inData.variables['latitude']) #extract latitudes
data_dict[variable]=np.array(inData.variables[variable]) #extract requested variable
inData.close() #close netCDF file
coords = self._create_coord_list(filenames,data_dict)
return UngriddedData(data_dict[variable],Metadata(name=variable,shape=(len(data_dict),),missing_value=-999.0),coords)

def get_variable_names(self, filenames, data_type=None):
return ['Info unavailable with this plugin']

james_oneill
No need for new plugin

Hi Masaru,

After a bit more playing, I've discovered that you don't need my plugin at all. Just use the in-built product 'cis':

$ cis plot AOT_440:AOT_440_allstations_2008-07_mav_obs.nc:product=cis

Masaru Yoshioka
Now I can make 2D plots but not a scatter plot!

Thank you James. Now I can plot AERONET data and collocated UM output data in 2D maps. It failed a couple of times with an error message " 'NoneType' object has no attribute 'name' " but at the moment plots are made without an error. I think probably adding ":procuct=cis" solved this issue.

Now I'm trying to create a scatter plot to compare monthly average AERONET data at 178 stations and monthly average of UM outputs collocated with AERONET measurements;

cis plot AOT_440:/group_workspaces/jasmin2/crescendo/Data/AERONET/AOT/LEV20/monave/AOT_440_allstations_2008-07_mav_stn.nc aod440_total:/group_workspaces/jasmin2/ukca/vol1/myoshioka/um/Dumps/Links2NC/teayh/AERONET_collocated/aod440_total_allstations_2008-07_mav_stn.nc --type comparativescatter

but this fails with the same error ( 'NoneType' object has no attribute 'name'). So I added ":procuct=cis"

cis plot AOT_440:/group_workspaces/jasmin2/crescendo/Data/AERONET/AOT/LEV20/monave/AOT_440_allstations_2008-07_mav_stn.nc:procuct=cis aod440_total:/group_workspaces/jasmin2/ukca/vol1/myoshioka/um/Dumps/Links2NC/teayh/AERONET_collocated/aod440_total_allstations_2008-07_mav_stn.nc:procuct=cis --type comparativescatter

but this also fails with an error "The following optional arguments could not be parsed: ['procuct=cis']"

Any idea how to create a scatter plot?

Masaru

Masaru Yoshioka
oops!

Actually there was a spelling error! Now I corrected the spelling error;

cis plot AOT_440:/group_workspaces/jasmin2/crescendo/Data/AERONET/AOT/LEV20/monave/AOT_440_allstations_2008-07_mav_stn.nc:product=cis aod440_total:/group_workspaces/jasmin2/ukca/vol1/myoshioka/um/Dumps/Links2NC/teayh/AERONET_collocated/aod440_total_allstations_2008-07_mav_stn.nc:product=cis --type comparativescatter

and a scatter plot was created without an error.

Thank you.
Masaru

Add comments

Log in Register
Website designed & built by OCC