Why aren’t my records processing?
Records being processed into NSSP’s BioSense Platform are checked for core data elements before being passed to the next stage of data processing. Like in baseball, it begins at home base. For NSSP, “home base” is when data arrive on the BioSense Platform. Arrival—or, stepping up to the plate—isn’t enough. These data must be ready to perform to be passed to first base—the Raw Table.
Aim for First Base
Each record is checked to make sure its contents meet message standards. Messages with all the essential information get a status of “Read” and are ready for processing (rounding the bases). If minimal standards are not met, the record is filtered and stored in the Raw Table with a status of “Filtered” (message_status=‘Filtered’). Filtered records are not processed.
Records are filtered for three main reasons:
1. Message lacks date/time from MSH_7.1 segment
2. MSH_9.1 Not Equal to ADT
3. MSH-4 lacks sending facility information
So, how can you keep your records out of the filtered zone? You do this by making sure each facility and vendor send essential data elements. Check your monthly Data Quality Reports to see how many records are being filtered. Then, as needed, contact the associated facilities or vendors for further action. If you need additional support, the Analytic Data Management (ADM) team is at your disposal to go over monthly data quality reports with you and answer questions.
Rounding the Bases
Once the record passes the initial filtering test and makes it to the Raw Table (first base), another stringent test must be passed to reach the Processed Table—or, second base. This check makes sure core data elements are present. And if not, records are removed and placed in a separate, site-specific Exceptions Table. Once the record passes the Exceptions check, the record is prepped per business processing rules and collapsed into holistic, visit-level data. Then the record is ready to round second base (Processing) toward third base—ESSENCE. After ESSENCE processing, site personnel will be able to visualize these data and conduct additional analyses. Data from the Raw, Exceptions, and Processed Tables then head to Home Base and into the DataMart, where site personnel can use tools such as Adminer, R, and eventually SAS Studio to further access these data. To re-“baseball” cap. Records can be Filtered to the Raw Table, sent to the Exceptions Table, or successfully passed to the Processed Table and into ESSENCE. The Processed Table and ESSENCE are the end goals to ensure you have complete, usable, and actionable data. Filtered and exceptions Tables are the places where records can be stopped before getting there.
With your help in maintaining and improving data quality, more and more data are “rounding” the bases and hitting home runs! We hope you find this explanation helpful. As always, please contact the Analytic Data Management (ADM) team site inspectors if you have questions or concerns.