Wiki Tools:

Extended Array

From CorePy

The ExtArray class is based largely on the Python array module interface, but extends the functionality by providing guaranteed page-aligned memory allocation and other low-level features useful for programming with CorePy. ExtArray may be found in corepy/lib/extarray. Since ExtArray is similar to Python's array module, this page only documents the differences between the two. Documentation for the Python array module is available here:

http://docs.python.org/library/array.html

Memory is allocated on a page basis, and always aligned to a page. Hugepages are supported on linux platforms (only tested on ppc64 so far), and are enabled by setting an optional boolean keyword argument to the array constructor. From that point, use of regular pages vs huge pages is transparent.

1 ExtArray Interface
2 Unsupported array Functionality
3 Extended Buffer
- 3.1 ExtArray Interface
  - 3.1.1 Constructor
  - 3.1.2 Buffer Interface

[edit] ExtArray Interface

[edit] Constructor

__init__(self, typecode, init = None, huge = False)

The constructor now optionally takes an integer instead of a list as an initializer ('init' keyword). In this case, memory for this many elements of the requested type is allocated, but not initialized.

Use of huge pages is enabled using the optional keyword argument, 'huge'. Set this to True to have huge pages allocated instead of regular sized pages. Huge page support only works on Linux, and has only been tested on the PPC64 architecture. x86 is expected to work as well. Note that huge page support has to be enabled in the kernel, and is not something most distributions do by default. See the 'Enabling Large Page Support' section in http://linuxgazette.net/155/krishnakumar.html for more information.

Below is an example of how to create an extended array of 16384 uninitialized integers, using huge pages.

import corepy.lib.extarray as extarray
 
data = extarray.extarray('i', 16384, huge = True)

All architectures supported by CorePy currently use 4k pages by default. On PPC64 (PS3), 16mb huge pages are used. x86/x86_64 will use either 2mb or 4mb pages.

[edit] Array Slicing

ExtArray supports referenced-based slicing operations. What this means is that an ExtArray can be sliced just like a normal Python array, but the memory backing the array slice is the same memory as the array that was sliced from. The ExtArray objectc created by the slice has its memory locked; it is an error to try to re-allocate or re-size the array. This behavior is different from lists and the array module, and instead behaves similarly to NumPy arrays.

[edit] Additional Methods

alloc(length)

Allocate memory for at least length number of items, without initializing that memory. If the new size is longer than the size required by the items stored in the array, every item in the array is preserved. If the new size is shorter, the first length items are preserved.

change_type(typecode)

Change the type of elements in the array without changing the array data. This has an effect similar to typecasting in C. The array length (in bytes) must be a multiple of the new type. Otherwise, a TypeError exception is raised.

clear()

Clear the memory representing the array to zero. This is provided as a faster alternative to iterating over the array and setting each element to zero.

copy_direct(str)

Copy data in str directly into the array, regardless of item size. Overwrites any data that may already be in the array, reallocating more memory only if needed. Useful for quickly initializing an array with raw data.

memory_lock(enable)

Enable/disable memory allocation operations. If memory is locked, an exception will be thrown whenever an operation would cause (re)allocation of memory to occur.

set_memory(addr, length)

Set the address and length (in bytes, must be a multiple of itemsize) of the memory used to store the array items, and lock the array's memory. Locking only prevents (re)allocation of memory, and can be enabled/disabled using memory_lock() if desired.

If the array has already allocated some memory and the memory lock is False, that memory is freed.

Useful for fitting an ExtArray over existing memory allocated elsewhere, for example a memory-mapped file or the memory-mapped SPU local store. See the CorePy_Cell/SPU#Working_With_The_Local_Store documentation for an example of how to obtain an ExtArray object mapping to an SPU local store.

[edit] Unsupported array Functionality

The following methods have not (yet) been implemented in ExtArray, due to lack of demand from developers/users. If any particular functionality is desired, contact the developers via the Mailing_Lists.

count()

fromfile()

fromunicode()

index()

insert()

pop()

remove()

reverse()

tofile()

tolist()

tostring()

tounicode()

The unicode 'u' typecode is not supported.

[edit] Extended Buffer

In addition to the ExtArray object, CorePy provides an implementation of the new (Python 2.6) buffer interface called ExtBuffer. The purpose of this is to support integration of page-aligned, huge-page, or other custom memory regions with NumPy and other libraries supporting the buffer interface.

[edit] ExtArray Interface

[edit] Constructor

__init__(self, data_len, memory = 0, huge = False)

The constructor requires that the length of the buffer to be created be specified, in bytes. Two optional keyword arguments are supported. The 'memory' keyword argument allows the user to specify a pre-allocated memory region for use by the buffer. This allows for an existing memory region to be used with anything supporting buffer objects. See the CorePy_Cell/SPU#Working_With_The_Local_Store documentation for an example of how to obtain an ExtBuffer object mapping to an SPU local store, then use that buffer to create a NumPy array object. If no memory pointer is specified, the ExtBuffer object will allocate data_len bytes of page-aligned memory. The 'huge' boolean parameter enables/disables the use of huge pages just like ExtArray.

Below is an example demonstrating how to create a NumPy array of 16384 32-bit integers using an ExtBuffer and huge pages:

import corepy.lib.extarray as extarray
import numpy
 
buf = extarray.extbuffer(16384 * 4, huge = True)
array = numpy.frombuffer(buf, dtype=numpy.int32)