Types

Introduction

Data layout and API can be defined by a composition of: * Scalars: Int8, Int16, Int32, Int64, Float32, Float64 * Compounds: Struct, Array, Ref

Types can have a fixed size (static types) or a variable size (dynamic size). The size of one instance cannot change after creation, but their content can.

Data Layout

Data structure is organized in 64bit slots.

Table indicates slot size and the description.

Scalar

Slots	Description
8	Data

String of n-bytes

Slots	Description
8	Size in bytes
ceil((n+1)/8)	Data in UTF-8 null terminated

Struct with static and n dynamical fields

Slots	Description
8	Optional: size if n>0
…	Fixed-size field data
8*(n-1)	Optional: Offsets of the 2nd to n-th last variable sized
…	Optional: Variable sized data

Array with n dimensions, whose l are dynamic of m items.

Slots	Description
8	Optional: size
8*l	Optional: size of l-variable dimensions
8*n	Optional: strides for n-dimensions larger than 2
m*s	Data or offsets for m-items (s is the item-size or 8)
…	Optional: Items data if the item type is dynamic

Union

Slots	Description
8	Type id
…	Item data

Ref

Slots	Description
8	Offset from start of the buffer
8	Optional: typeid

Python type interface

The following describe the generic interface

Class variables:

name	Description
_size	Size in bytes, None if not static size

Class methods:

name	args	return	Description
_inspect_args	*args, *kwargs	Info	Return at least info.size and optionally other metadata to build objects in memory from python objects. Args are propocessed by _pre_init
_pre_init	*args, *kwargs	args, kwargs	Preprocess arguments and generate standard values
_to_buffer	buffer offset value	None	Serialize python in buffer from offset. Assume sufficient size is available
_from_buffer	buffer offset	instance	Return Python instance from buffer and and offset
_to_schema		string	TODO String representation of the type to deserialize objects

Instance variables:

name	Description
_size	Optional: size in bytes for dynamic-size

Class methods:

name	args	return	Description
_post_init			Run after object creation in __init__
copy	buffer or context	None	TODO Return a copy into a buffer Assume sufficient size is available
_update	value	instance	Update values of an exsiting objec
_to_json		json	TODO json

__init__

init is used to create a new xobject by allocating memory on the buffer. Ref and Scalars cannot be initialized alone (maybe temporarily)

Init can take by default:

• a json object:

  • dict for Struct

  • list for Array,

  • number of Scalars

  • String for String

  • (Typename, {}) for Ref of multiple types

• dimensions (for string or arrays)

• an xobject of the correct type

• None for Ref

The data is dispatched to _pre_init using the following convetion: * a dict is passed with ** * a tuple is passed with * * any other object is passed verbatim

Init uses the following steps:

• pre_init to pre_process input

• inspect_args to calculate sizes and collect metadata

• get_a_buffer to allocate space

• _to_buffer to write to memory

• _from_buffer to build a python object

• post_init

_from_buffer

When accessing data inside compound object the _from_buffer method is used. For scalar and string, python data is returned. For coumpund object, XObjects are returned

setters Setters uses to_buffer or _update to update existing data.