Hi,
I noticed something weird in the plugin I'm writing & testing. This is for OMI data. From the info call I get:
Total number of points = 98640
Number of non-masked points = 98640
This is unexpected, because the begin and end of the swath contain fill values (pixels with solar zenith angle > 88 degrees are not processed). I read the data into a masked array and explicitly mask the data with the fill value myself. And sure enough, if I count the number of unmasked points in my data plugin, I get 84931 points, not 98640 (counting is done with "np.sum(np.logical_not(data.mask))"). The "Missing value" is set correctly, the full info output is printed below.
Am I missing something?
Maarten
(cis_env)[sneep@bhw465 ~]$ cis info --product OMI ~/nobackup/OMCLDO2/test_v2/2005/01/14/OMI-Aura_L2-OMCLDO2_2005m0114t0129-o02662_v003-2015m1106t181844.he5 -v 'HDFEOS.SWATHS.CloudFractionAndPressure.Data Fields.CloudFraction'
2016-01-18 15:19:45,213 - INFO - Identified input file list: ['/usr/people/sneep/nobackup/OMCLDO2/test_v2/2005/01/14/OMI-Aura_L2-OMCLDO2_2005m0114t0129-o02662_v003-2015m1106t181844.he5']
2016-01-18 15:19:45,220 - INFO - Looking for plugins...
2016-01-18 15:19:45,222 - INFO - importing plugin ['S5P', 'OMI']
2016-01-18 15:19:45,229 - INFO - Retrieving data using product OMI...
2016-01-18 15:19:45,276 - INFO - HDFEOS.SWATHS.CloudFractionAndPressure.Data Fields.CloudFraction: (1644, 60) (valid: 84931)
Ungridded data: CloudFraction / (None)
Shape = (1644, 60)
Total number of points = 98640
Number of non-masked points = 98640
Long name = Effective cloud fraction clipped between 0.0 and 1.0
Standard name = CloudFraction
Units = None
Missing value = -1.26765060023e+30
Range = (0.0, 1.0)
History =
Coordinates:
HDFEOS.SWATHS.CloudFractionAndPressure.Geolocation Fields.Latitude
Long name = Latitude of the center of the groundpixel
Standard name = latitude
Units = degrees_north
Missing value = -1.26765060023e+30
Range = (-89.71006, 80.066536)
History =
HDFEOS.SWATHS.CloudFractionAndPressure.Geolocation Fields.Longitude
Long name = Longitude of the center of the groundpixel
Standard name = longitude
Units = degrees_east
Missing value = -1.26765060023e+30
Range = (-179.99815, 179.99969)
History =
HDFEOS.SWATHS.CloudFractionAndPressure.Geolocation Fields.Time
Long name = Time at Start of Scan (s, TAI93)
Standard name = time
Units = days since 1600-01-01 00:00:00
Calendar = gregorian
Missing value = -1.26765060023e+30
Range = ('2005-01-14 01:51:19.978646', '2005-01-14 02:46:06.603346')
History =
Hi Maarten,
I've had a look at this and it seems there is a small bug in the summary routine which prints the info you see in your output. If you're interested the mistake appears to be in line 595 of ungridded_data.py - the length of the unmasked_points is actually (and confusingly) the length of all of the points. Thanks for catching this, I'll look to fix it in the next release.
This won't affect any of the actual use of your data internally and CIS should still respect your masked points.
Let me know if you hit anything else but it looks like you're going well!
Duncan
Thanks, at least it isn't me. I've got OMI and S5P working, at least for aggregation and plotting.
Now I only have to figure out how to plot observations in a way that the actual ground pixels size is shown, not just a scatterplot. Some sort of nearest neighbour interpolation. In S5P we have approximate pixel corners available (latitude bounds, longitude bounds, stored with CF compliant bounds attributes). It would be nice to use those.