reading an UM output 2

10 posts / 0 new
Last post
Masaru Yoshioka
reading an UM output 2

I have a problem in reading UM output data in .pp files with CIS.

The 3 hourly output file teafwa.pb20080101.pp contains 58 variables when opened with xconv, but "cis info teafwa.pb20080101.pp" returns only about 30 of them. same thing if I do

cis.get_variables('teafwa.pb20080101.pp', product=None, type=None)

in ipython. The returned list does not include any of aerosol optical depths (AODs) I want to compare with AERONET AOTs. The same thing happens for monthly data. So I don't know the proper names for AODs.

Is it possible to get variable names for AODs? Or is there any other way to specify them in cis operations?

Thanks,
Masaru

duncanwp
reading an UM output 2

Hi Masaru,

Could you try reading the data in Iris and telling me if it gives you the right Cubes?

E.g. what is the output of:

import iris
iris.load('teafwa.pb20080101.pp')

Masaru Yoshioka
I got these below. What I

I got these below. What I need are m01s02i50*. So I did this

cis info m01s02i500:teafwa.pb20080101.pp

and got an error

ERROR - An error occurred retrieving data using the product HadGEM_PP. Check that this is the correct product plugin for your chosen data. Exception was InvalidVariableError: Variable not found: m01s02i500
To see a list of variables run: cis info teafwa.pb20080101.pp. - check cis.log for details

[<iris 'Cube' of m01s02i440 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i441 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i442 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i443 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i444 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i445 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i500 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i501 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i502 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i503 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i504 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i505 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i362 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i363 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_nucleation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_nucleation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_Aitken_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_accumulation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_seasalt_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_coarse_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_seasalt_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_Aitken_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_insoluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_insoluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_accumulation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_insoluble_accumulation_mode_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_coarse_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_insoluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_nucleation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i401 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i402 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i403 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i404 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i405 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i406 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i407 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i408 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i409 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i410 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i411 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i412 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i413 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i414 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i415 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i479 (time: 8; model_level_number: 52; latitude: 145; longitude: 192)>,
<iris 'Cube' of air_potential_temperature / (K) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of air_pressure / (Pa) (time: 8; latitude: 145; longitude: 192)>]

duncanwp
Hmm, that's strange.

Hmm, that's strange.

Are you able to put the file somewhere on JASMIN that I can test it? Perhaps in your home directory if it isn't too large?

Masaru Yoshioka
I copied it here: /home/users

I copied it here: /home/users/myoshioka/temp/

thanks

duncanwp
Great, thanks.

Great, thanks.

I've had a look and it seems this is a bug in CIS, and Iris (https://github.com/SciTools/iris/issues/2927), where all the variables end up being bunched up into 'unknown' because they don't have any metadata other than a STASH code. You're not having much luck!

I've written a plugin which will work around this issue for now: https://github.com/duncanwp/cis_plugins/blob/master/hadgem_unknown_vars.py

Hopefully this will help and you can get on with your collocations!

Masaru Yoshioka
Reply to #6

Hi Duncan,
Many thanks for writing the plugin! Following the documentation, I put it in ~/cis/ , set "export CIS_PLUGIN_HOME=$HOME/cis" in my .bashrc, did source .bashrc and then did

cis info teafwa.pb20080101.pp

It returned a couple of lines of warning messages like this;

2017-12-20 11:41:42,634 - WARNING - /usr/lib/python2.7/site-packages/iris/fileformats/_pyke_rules/compiled_krb/fc_rules_cf_fc.py:1811: UserWarning: Ignoring netCDF variable u'hybrid_ht' invalid units u'level'
warnings.warn(msg.encode('ascii', errors='backslashreplace'))

and still doing something 35 minutes later... I suspect this may not be working well?

Masaru

Masaru Yoshioka
update to #7

I just got another message;

2017-12-20 12:17:30,465 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure

not sure if this is something to worry about, but this shows cis info is taking time but working. this can be temporary issue of JASMIN. So I tried doing the same thing on sci2 (I'm usually on sci3) but it appears to be taking time as well.

Masaru Yoshioka
another update to #6

cis info finally finished and failed. It took about 50 minutes.

2017-12-20 12:28:30,701 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure
2017-12-20 12:28:31,072 - WARNING - Identified 288 point(s) which were missing values for some or all coordinates - these points have been removed from the data.
Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']

2017-12-20 12:28:31,141 - ERROR - Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']
- check cis.log for details

Masaru Yoshioka
hadgem_unknown_vars.py

If you need to check, I stored hadgem_unknown_vars.py in /home/users/myoshioka/cis/ on JASMIN.

Website designed & built by OCC