I got these below. What I need are m01s02i50*. So I did this
cis info m01s02i500:teafwa.pb20080101.pp
and got an error
ERROR - An error occurred retrieving data using the product HadGEM_PP. Check that this is the correct product plugin for your chosen data. Exception was InvalidVariableError: Variable not found: m01s02i500
To see a list of variables run: cis info teafwa.pb20080101.pp. - check cis.log for details
I've had a look and it seems this is a bug in CIS, and Iris (https://github.com/SciTools/iris/issues/2927), where all the variables end up being bunched up into 'unknown' because they don't have any metadata other than a STASH code. You're not having much luck!
Hi Duncan,
Many thanks for writing the plugin! Following the documentation, I put it in ~/cis/ , set "export CIS_PLUGIN_HOME=$HOME/cis" in my .bashrc, did source .bashrc and then did
cis info teafwa.pb20080101.pp
It returned a couple of lines of warning messages like this;
2017-12-20 12:17:30,465 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure
not sure if this is something to worry about, but this shows cis info is taking time but working. this can be temporary issue of JASMIN. So I tried doing the same thing on sci2 (I'm usually on sci3) but it appears to be taking time as well.
cis info finally finished and failed. It took about 50 minutes.
2017-12-20 12:28:30,701 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure
2017-12-20 12:28:31,072 - WARNING - Identified 288 point(s) which were missing values for some or all coordinates - these points have been removed from the data.
Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']
2017-12-20 12:28:31,141 - ERROR - Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']
- check cis.log for details
Hi Masaru,
Could you try reading the data in Iris and telling me if it gives you the right Cubes?
E.g. what is the output of:
import iris
iris.load('teafwa.pb20080101.pp')
I got these below. What I need are m01s02i50*. So I did this
cis info m01s02i500:teafwa.pb20080101.pp
and got an error
ERROR - An error occurred retrieving data using the product HadGEM_PP. Check that this is the correct product plugin for your chosen data. Exception was InvalidVariableError: Variable not found: m01s02i500
To see a list of variables run: cis info teafwa.pb20080101.pp. - check cis.log for details
[<iris 'Cube' of m01s02i440 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i441 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i442 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i443 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i444 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i445 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i500 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i501 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i502 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i503 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i504 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i505 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i362 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s02i363 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_nucleation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_nucleation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_Aitken_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_accumulation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_seasalt_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_soluble_accumulation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_soluble_coarse_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_sulfuric_acid_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_seasalt_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_soluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_Aitken_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_black_carbon_in_insoluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_insoluble_aitken_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_accumulation_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_insoluble_accumulation_mode_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of number_of_insoluble_coarse_mode_particles_per_molecule_of_air / (1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_dust_in_insoluble_coarse_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of mass_fraction_of_particulate_organic_matter_in_soluble_nucleation_mode_dry_aerosol_in_air / (kg kg-1) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i401 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i402 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i403 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i404 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i405 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i406 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i407 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i408 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i409 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i410 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i411 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i412 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i413 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i414 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i415 (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of m01s38i479 (time: 8; model_level_number: 52; latitude: 145; longitude: 192)>,
<iris 'Cube' of air_potential_temperature / (K) (time: 8; latitude: 145; longitude: 192)>,
<iris 'Cube' of air_pressure / (Pa) (time: 8; latitude: 145; longitude: 192)>]
Hmm, that's strange.
Are you able to put the file somewhere on JASMIN that I can test it? Perhaps in your home directory if it isn't too large?
I copied it here: /home/users/myoshioka/temp/
thanks
Great, thanks.
I've had a look and it seems this is a bug in CIS, and Iris (https://github.com/SciTools/iris/issues/2927), where all the variables end up being bunched up into 'unknown' because they don't have any metadata other than a STASH code. You're not having much luck!
I've written a plugin which will work around this issue for now: https://github.com/duncanwp/cis_plugins/blob/master/hadgem_unknown_vars.py
Hopefully this will help and you can get on with your collocations!
Hi Duncan,
Many thanks for writing the plugin! Following the documentation, I put it in ~/cis/ , set "export CIS_PLUGIN_HOME=$HOME/cis" in my .bashrc, did source .bashrc and then did
cis info teafwa.pb20080101.pp
It returned a couple of lines of warning messages like this;
2017-12-20 11:41:42,634 - WARNING - /usr/lib/python2.7/site-packages/iris/fileformats/_pyke_rules/compiled_krb/fc_rules_cf_fc.py:1811: UserWarning: Ignoring netCDF variable u'hybrid_ht' invalid units u'level'
warnings.warn(msg.encode('ascii', errors='backslashreplace'))
and still doing something 35 minutes later... I suspect this may not be working well?
Masaru
I just got another message;
2017-12-20 12:17:30,465 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure
not sure if this is something to worry about, but this shows cis info is taking time but working. this can be temporary issue of JASMIN. So I tried doing the same thing on sci2 (I'm usually on sci3) but it appears to be taking time as well.
cis info finally finished and failed. It took about 50 minutes.
2017-12-20 12:28:30,701 - ERROR - Unable to concatenate cubes on load:
failed to concatenate into a single cube.
Cube metadata differs for phenomenon: air_pressure
2017-12-20 12:28:31,072 - WARNING - Identified 288 point(s) which were missing values for some or all coordinates - these points have been removed from the data.
Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']
2017-12-20 12:28:31,141 - ERROR - Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20']
Aerosol_CCI: ['.*ESACCI.*AEROSOL.*']
Caliop_L1: ['CAL_LID_L1-ValStage1-V3.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro-Prov-V3.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI: ['.*ESACCI.*CLOUD.*']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
HadGEM_unknown_vars: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']
- check cis.log for details
If you need to check, I stored hadgem_unknown_vars.py in /home/users/myoshioka/cis/ on JASMIN.