Introduction to hm.Survey
hm.Survey: Generic Survey Data Entry/Reporting System
hm.Survey provides a facility for automatically generating template-based
- Survey data entry programs for both
- Online (browser based) desktops, and
- Android devices
- Tabulations, both online (via browser) and offline (in HTML and CSV formats).
The survey data entry/reporting program condenses over fifteen years of writing survey data entry programs in various countries (Maldives, Ghana, Lebanon, Tanzania, Iraq, Oman, Laos). The idea is to capture the periodic data capture of a central list of entities (establishments, households, farmers) which are queried repeatedly (yearly, quarterly, monthly) and whose data is tabulated according to guidelines preferred by aid agencies (UNIDO, UNDP, IMF, World Bank, DFID, Asian Development Bank, ECOWAS) involved in survey work.
Figure 1. Conceptual data hierarchy
The system is platform and database neutral since it employs cross-platform technologies. The current system has been tested extensively on the Ubuntu platform as the server (clients are browser based, so they are cross-platform by definition) using both MySQL, Oracle and SQL Server databases.
Fig 2: Architecture
Also, the system has been designed from the ground up to be multi-language as shown in figures 3a and 3b.
Fig 3a. Multi language (English)
Figure 3b. Multi Language (Arabic)
- The system utilizes a register which can be companies, households, farmers, etc. The only requirement is that the register has a unique ‘search_id’ text field and a text field named ‘name’.
- Data entry for each period (single for a one-off survey or more than one for a multi-period survey) is broken down into a number of sections. The data for each section is saved in a database table (MySQL, Oracle or SQL Server) corresponding to the section. Each entry in a section corresponds to an entry in the register.
(There are other slightly more technical assumptions given in the design document.)
Data entry programs for browser based desktops and Android devices
The template specifies the data entry programs completely. There are two programs that use the underlying ‘schema.json’ file to present to the user a data entry interface corresponding to the structure specified in the schema file. One program runs on Android devices, and another runs on a web-browser. Both programs communicate with an online database (MySQL, Oracle, SQL Server) through a script that links the the database to each. This database too is created using a script generated by hm.Survey using the schema file.
All data is saved online. (It is possible to host a local web-server that communicates with the browser and Android device, but that would restrict the data transfer from the handheld devices and browsers to the local area network).
Figure 4 shows the Android application with the list of establishment (in Arabic).
Figure 4. List of establishments (in Arabic) on Android device
Tabulation generation programs
There are two types of tabulation which are specified in a text (.json) file:
- Simple classification
- Single classification parameter (such as province, legal type of organization, administrative type, employment class, etc.) with one or more numeric parameters which are aggregated.
- Cross classification
- Two classification parameters and a single numeric parameter resulting in a cross-tabulated pivot table.
The tabulations are displayed online (which can be updated/regenerated and saved as HTML files) or generated as offline files (CSV or HTML).
Business Register function
It is possible to have two independent programs (and hence databases) linked via a "Survey-Business Register" relationship whereby the entities in each program would correspond to businesses (companies/factories) and they would be linked together via:
- a list of common fields
As Figure 5 below shows, two operations are possible with the 'Business Register' thus linked (details of linkage covered in the design document):
- Take a frozen frame from the Business Register
- This would bring in those establishments (only the common fields) from the Business Register to the Survey database whose search_ids are specified in the file to be attached (one search _id per line).
- Update the Business Register
- This would update Business Register establishments (only the common fields) using current Survey data.
Figure 5. Business Register update screen
GPS capture on Android
The Android based data entry has been equipped to capture location coordinates (longitude, latitude). GPS enabled devices phones are now commonplace, so extending the data entry program on the Android to capture location enables field based enumerators to update the online database with GPS coordinates of factories, companies, farms etc.
Figure 6 shows the GPS capture screen. Figure 7 shows the coordinates being reflected on the data entry screen.
Figure 6. GPS capture on Android device.
Figure 7. GPS coordinates on data entry screen.
Online data submission by companies
Since the application is online by design, provision has been made for direct data entry by companies. The usual workflow for data entry is as follows:
- Step 1 (entry on paper in field): questionnaires are filled out by enumerators in the field.
- Step 2 (data entry in office): filled paper questionnaires are sent to the office where they are entered in the database by office staff.
By enabling direct online submission by companies, the following workflow becomes possible (Steps in orange are under construction):
- Step 1-online (intimation): company users are intimated via email that they can start submitting data online.
- Step 1-paper (entry on paper in field): This can be done at the same time as initiating the online procedure so that both online and paper data entry can proceed simultaneously.
- Step 2-online (online submission by companies): Upon login by company user (these are specially designated users which are administratively assigned), only the company associated with the user can be viewed and edited. Company users will mark their data entry as ‘finalized’ when they are done.
- Step 2-paper (data entry in the office)
- Step 3-online (review and close in the office): Office (administrative) users can at any given time view which companies have been ‘finalized’ or are in the process of data entry (‘not finalized’). Office users can 'close' companies whose data has been ‘finalized’ thereby preventing further online edits by companies (subsequent editing by office users is possible).
Note that once the process of online intimation and sending out of paper questionnaires to the field has started, the online and paper workflows are independent of each other. Also technically, data entry by both company and office users is ‘online’ (since both use the internet for submission) but in this section, online is used exclusively to mean direct submission by companies.