Integrated Census Microdata (I-CeM)

In the course of the I-CeM project, various documents, spreadsheets, and datasets, have been created, partly as spinoff’s from the work done, or in order to explain various facets of the data. These are made available here to help users of the I-CeM dataset, but also as stand-alone academic products in their own right.

Browse the following sections for supporting documentation.

This will be the general user manual for the I-CeM data collection covering:

• the history of British census-taking, 1851-1911;
• the history of the individual censuses, with lists of the official publications created, and images of the schedules and enumerators books used in the census;
• the provenance of the I-CeM data;
• the enrichment programme for the I-CeM data, explaining the processes of reconciliation, reformatting, standardisations, and the creation of inference and enriched variables from the original data;
• descriptions of all the variables in the I-CeM dataset, including the variable labels, form and length, which census years they cover, their access status, and their possible values;
• and, means of access to the dataset.

The Guide will have a navigation pane for ease of use. To view the .pdf navigation pane, click the Adobe Reader Toolbar icon at the end of the Tools pop up, and then click the 'Bookmarks' icon (second down).

• The most recent I-CeM Guide will be provided in the coming weeks.

The latest version of the Guide will be dated August 2020. It is recommended that users of I-CeM have access to the latest version of the Guide.  

It is important for users to realise that not all census records have survived. This especially true for 1851 and 1861 wherein some cases data for whole parishes or even Registration Sub-Districts is ‘wanting’. Further details on missing data is available from both The National Archives and FindMyPast websites, as follows:

• see section 10 of TNA’s guide to census data
• list of known missing ‘pieces’ from the FindMyPast

In addition, users are referred to a paper by Bennett, van Lieshout and Schürer which estimates the levels of missing data in the I-CeM data collection, together with ‘weights’ that have been calculated to ‘correct’ for missing data when undertaking analyses at a national level.

A further problem exists in the identification of missing census records at the level of the parish. As part of the I-CeM project, an attempt was made to reallocate each individual census record to the census parish in which it was enumerated.

This process is known as reconciliation. Whilst this sounds like a straight-forward exercise, it proved to be laborious and time-consuming and can never be exact due to the census information available that underpins this process.

In short, in order to allocate each page of the census data to an administrative parish, the I-CeM project team had to rely on the transcribed information at the head of each census page, which is non-standard, imprecise and sometimes incorrect. Despite this, each census record was allocated a parish identifier (PARID, see Data Dictionaries section below). The table below shows the difference between the population of parishes found in the raw I-CeM data following reconciliation, and what one would expect from the published Census Reports.

• Differences in Population Counts by Parish Table (.xlsx)

Subsequent to the release on the I-CeM data via the UK Data Service, a number of corrections to the PARID allocation have been made for parishes in England and Wales. These corrections can be implemented by downloading the following look-up tables.

• 1851
• 1861
• 1891
• 1911

Both the original census data and the FindMyPast transcriptions contain a relatively small number of duplications as well as empty (null). Whilst every effort was made to identify these during the processing stages of the I-CeM some additional duplicate and null/empty records have subsequently been identified. These can be identified via the following look-up table.

• Duplicate Null records

Given that many users will wish to analyse census data over time for comparative geographical units, the I-CeM team endeavoured to identify the parishes of enumeration consistently across all censuses, thus creating the CONPARID variable in the I-CeM dataset - see the I-CeM Guide for a definition. This is based on the work of Professor Sir E. A. Wrigley for England and Wales1, and by Professor Michael Anderson for Scotland.

In creating consistent geographies the basic logic is to amalgamate parishes where necessary so that the geographical territory under consideration remains constant over time. So, for example, assume that part of parish A was transferred to parish B between census years. In order to create a consistent geographical unit over time one would need to treat them not as separate parishes, but as a single entity. The reasoning for producing a consistent geography variable is that it facilitates comparisons over time, where, as far as possible, like is being compared to like.

Due to the multiple changes in census enumeration geography over time, and the fact that parish boundaries themselves changes, the project produced two sets of consistent geography: one for the period 1851-1891, the other for 1901-1911. To facilitate mapping of the I-CeM data, following these consistent geographies, the I-CeM project also created two GIS databases to allow comparable parish-level based maps to be constructed over time. This work was completed in collaboration with Dr Max Satchell and Dr Corinne Roughley, both of the University of Cambridge, whose contribution is duly acknowledged.  For those who wish to undertake their own mapping of the I-CeM data, the GIS databases are available from the UK Data Service.

• Consistent Parish Look Up Table (.xlsx)

As part of the I-CeM project a number of data dictionaries have been created. These essentially provide users with detailed information on how key variables have been coded and standardised. These are made available here to help users of the I-CeM data collection but also as stand-alone academic products in their own right which can be used in conjunction with related research projects.

This dictionary gives the text strings relating to marital condition in the COND variable of the I-CeM dataset, and their relationship to the MAR coded variable – see I-CeM Guide for definitions.

• Marital Condition Dictionary (.xlsx)

This dictionary gives the text strings relating to relationship to head in the RELAT variable in the I-CeM dataset, and how these relate to the codes in the RELA variable – see the I-CeM Guide for definitions.

• Relationship to Head Dictionary (.xlsx)

The I-CeM Occupational Matrix indicates the position of occupational headings in the occupational tables in the published Census Reports for England and Wales and Scotland for the years 1851 to 1911. Each row of this Matrix gives the position of one occupational heading in the occupational classifications used in the Census Reports for this period. The HISCO coding of each of the occupational headings is also given. Each row is given a number which then becomes the value found in the OCCODE variable in the I-CeM dataset. A more detailed introduction to the Matrix, its construction and uses is available.

• I-CeM Occupational Matrix introduction (.docx)
• I-CeM Occupational Matrix (.xlsx)

The I-CeM Employment Status Dictionary gives the text strings relating to employment status found in the EMPLOY variable in the I-CeM dataset, and the equivalent employment status code EMPLOYCODE – see I-CeM Guide for definitions.

• I-CeM Employment Status Dictionary (.xlsx)

This identifies the ‘parish’ of enumeration listed in the various tables published year by year in the GRO and GRO(S) Census Reports. It is, therefore, not consistent over time. Equally, the same named parish in different years may not cover the same geographical territory, due to boundary changes over time. This is the basis of the PARID variable in the I-CeM dataset – see the I-CeM Guide for a definition. A full list of these parish units in available for each year.

• Parish ID for All Years Table (.xlsx)

This gives a list of all the places, hamlets, townships, etc., given as birthplaces in the I-CeM dataset, with the STD_PAR parish, CNTI county, and ALT_CNTI county, in which they fall. See the I-CeM guide for definitions of these variables.

• I-CeM Placelist (.xlsx)

The I-CeM Disability Dictionary gives a listing of the text strings found in the disabilities column (DISAB), and the DISCODE1 and DISCODE2 values they are given in the I-CeM dataset – see the I-CeM Guide for definitions. The DISCODE1 variable is a 7 digit code but the Dictionary only gives the code from the first positive value. Thus, ‘Deaf Dumb Born Like It’ has a value of 0100010 in the dataset, which indicates that information on dumbness and hearing impairment is present, as is information about duration of disability, but there is no information regarding visual impairment, idiocy and imbecility, lunacy, other disabilities, or information relating to severity of disability. But in the Dictionary the string has a value of 100010, the first ‘0’ indicating the absence on information on visual impairment being omitted.

• I-CeM Disability Dictionary (.xlsx)

This dictionary gives the text strings relating to language spoken in the LANG variable of the I-CeM dataset, and its relationship to the LANGCODE variable – see I-CeM Guide for definitions.

• I-CeM Language Dictionary (.xlsx)

This dictionary gives the text strings relating to the number of rooms in the NOOFROOMS variable in the I-CeM dataset, and their relationship to the NOOFROOMSCODE variable – see the I-CeM Guide for definitions.

• I-CeM Number of Rooms Dictionary (.xlsx)

This gives the text strings relating to building types in the BUILDTYPE variable in the I-CeM dataset, and their relationship to the BTCODE variable – see the I-CeM Guide for definitions.

• I-CeM BUILDTYPE Dictionary (.xlxs)