CSV Formatting for OneRoster Providers

The file format for each of the data files is a Comma Separated Values (CSV) format as specified in RFC 4180 with the extra restriction that carriage returns are not permitted within a field. Fields containing commas and double-quotes must be enclosed in double-quotes. If double-quotes are used to enclose a field, then a double-quote appearing inside the field must be escaped by preceding it with another double-quote. The CSV files must be UTF-8 encoded RFC3629. Importing processors must tolerate BOM (Byte Order Mark) prefixes and ignore them. In a UTF-8 encoded file with a BOM, the BOM will appear as the 3-byte sequence 'EF BB BF' at the beginning of the file. If present, the CSV header will begin at the 4th byte of the file; if not present, the CSV header will begin at the 1st byte of the file.

Other key points are:

• All header names and content values within the CSV file are case-sensitive. Incorrect case will result in conformance failure and should be logged as a detected error;
• The Header row for each file is required;
• The Header fields for a file must be supplied in the same order as defined in the tables below. If metadata extension fields are used, then they must be added to the right of the defined data fields in the specification, as the last set of columns. This ensures that the sequence of the defined data fields remains fixed irrespective of the presence or not of metadata fields;
• Header field names must be uniquely named within a file;
• Files that contain no data rows are NOT permitted;
• For ALL fields that are optional and which have a defined enumeration token set, then the value of NULL/blank (i.e. a blank value "") is permitted to denote that NO data is supplied;
• The support of ALL string-based data-types requires that a maximum length of at least 255 must be supported by implementations. If string lengths of greater than 255 are used then system may lose some part of the string without failing conformance.

When mapping from the data model OneRoster, 17a to the equivalent column headers the following special rules have been applied:

• When referencing an object in the data model (using the data-type GUIDRef) the name has been extended using SourcedId e.g. 'org' to 'orgSourcedId';
• When referring to a set of objects in the data model maps to a comma separated list, enclosed in quotation marks, under the corresponding column header (as per RFC 4180);
• The references to resources in the Class and Course data models have NOT been mapped.

CSV import/export is envisioned to work in one of two ways, bulk and delta:

BULK Processing

The CSV files MUST NOT contain values in the "dateLastModified" and "status" fields. That is to say, for each row, there will be no data for either of these fields. Importing Systems MUST view this incoming data as the reference version of all data. That is to say, if records that were previously imported do not appear in this bulk csv file, then the importing system should internally mark those records as 'tobedeleted' with the 'dateLastModified' value set to the time of the bulk files import processing. If said records subsequently appear in a future bulk file, then those records should be updated and made active. When a set of Bulk files are created they must be semantically complete i.e. every object referenced by another object MUST also be defined in the corresponding source CSV file and included with the manifest.csv file.

DELTA Processing

The incoming CSV file MUST contain values in the "dateLastModified" and "status" fields. That is to say, each row in the CSV file should include data describing that (the row) data was last modified and its status. Importing systems must view this incoming delta file as changes to the reference data, and should make the appropriate changes, deletions and insertions of the new data.

Field Format Type	Description
GUID	Denotes some form of globally unique identifier (must be less than 256 characters). Not restricted to the 128 bit UUID format
GUID Reference	Denotes the GUID of an object defined in some other CSV data file
List of GUID Reference	Denotes a comma-delimited list of GUID References of objects defined in another CSV data file
ID	Denotes an identifier defined outside of the OneRoster specification
String	Denotes a sequence of characters that should follow the description
List of Strings	Denotes a sequence of Strings without commas, comma separated
Enumeration	Denotes a fixed set of values
Enumeration List	Denotes a list of Enumerations
Float	Denotes a floating point number
DateTime	Must be in W3C profile of ISO 8601 with UTC timezone and millisecond resolution
Date	Must use ISO 8601 format ("YYYY-MM-DD")
Year	Must use ISO 8601 format ("YYYY")

A number of important changes have been made from v1.0:

• The 'dateLastModified' in v1.1 had been changed from the v1.0 the resolution 'YYYY-MM-DD' to 'YYYY-MM-DDTHH:MM:SS.sssZ'. For backwards compatibility, a v1.0 format of 'YYYY-MM-DD' should be transformed to 'YYYY-MM-DDT23:59:59.999Z' in a v1.1 context;
• In the 'status' field the enumeration value of 'inactive' has been removed. For compatibility with v1.0, a value of 'inactive' should be interpreted as 'tobedeleted';
• When delta-processing information is being exchanged and records are marked 'tobedeleted' then all fields that have been identified as mandatory (with the exception of the object 'sourcedId') should be considered optional.

For CSV exchange, the data model can only be extended by adding new columns to the end of the defined set of columns. A suitable naming convention should be used for the new header names. An example is: metadata.orglabel.namepart Where: metadata is the initial set of characters for all extensions; orglabel is the label used to identify the organization that has created the extension. In the case of an extension from IMS Global this field will be empty and one of the delimiters removed; namepart the unique header name part allocated by the organization for the data model. Examples are: 'metadata.hmh.homeemail', 'metadata.pearson.namesuffix', etc.