You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bug Description
A few of the details with the python-client avro schemas seem questionable. Need clarity on what's correct and fix either unit tests or schemas.
Change SqsHandlers/S3MessageAdapter's transform to take a single record, since only operates on 1. (snuck this in since was hurting to look at).
Fix relationships['type'], which should it be an enum which allows for:
{"relationships":[{"id":"5b58de08-afef-49fb-99a1-9c5d5c003bde","type":"COLLECTION"}]}
(from onestep-test-data GOES/granules/*.json files)
Fix fileLocationType to an enum
Fix fileLocations in ParsedRecord to be a Map/Dict[str, FileLocation}
"fileLocations":{
"s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc":{
"serviceType":"Amazon:AWS:S3",
"deleted":False,
"restricted":False,
"asynchronous":False,
"locality":"us-east-1",
"lastModified":1572430074000,
"type":"ACCESS",
"optionalAttributes":{
},
"uri":"s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc"
}
}
Severity
Severity: MEDIUM (High/Medium/Low)
Priority: MEDIUM (High/Medium/Low)
To Reproduce
Steps to reproduce the behavior:
Run test_ParsedRecord.py unit test on this branch: 1508-avro-schemas
Expected Result
All tests should pass
Actual Result
ERROR: test_fileLocations (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Superfluous parameters in call: {'s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc'}
ERROR: test_relationships (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Key type has incorrect type: str instead of Optional[RelationshipType].
Additional context
Found bug by running sme/sme.py script with data loaded in local OneStop from OneStop-test-data repo (GOES triggered the errors mostly)
The text was updated successfully, but these errors were encountered:
Note, manually did this but added to next fixing story:
needed try/except around get_uuid_metadata with a return in except since some records were causing "(404) when calling the HeadObject operation: Not Found" 'cuz I think they didn't exist, were deleted records.
Also changed SqsHandlers and S3MessageAdapter to operate/transform on a single record, and only take 1 in as a parameter. Since it already only operated on 1 record.
Bug Description
A few of the details with the python-client avro schemas seem questionable. Need clarity on what's correct and fix either unit tests or schemas.
{"relationships":[{"id":"5b58de08-afef-49fb-99a1-9c5d5c003bde","type":"COLLECTION"}]}
(from onestep-test-data GOES/granules/*.json files)
"fileLocations":{
"s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc":{
"serviceType":"Amazon:AWS:S3",
"deleted":False,
"restricted":False,
"asynchronous":False,
"locality":"us-east-1",
"lastModified":1572430074000,
"type":"ACCESS",
"optionalAttributes":{
},
"uri":"s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc"
}
}
Severity
Severity: MEDIUM (High/Medium/Low)
Priority: MEDIUM (High/Medium/Low)
To Reproduce
Steps to reproduce the behavior:
Expected Result
All tests should pass
Actual Result
ERROR: test_fileLocations (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Superfluous parameters in call: {'s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc'}
ERROR: test_relationships (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Key type has incorrect type: str instead of Optional[RelationshipType].
FAIL: test_discovery (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
assert len(args) == 2
AssertionError
Additional context
Found bug by running sme/sme.py script with data loaded in local OneStop from OneStop-test-data repo (GOES triggered the errors mostly)
The text was updated successfully, but these errors were encountered: