• Project overview
• Goals
• Entity-component-system (ECS)
  • An example ECS
  • Benefits of ECS
  • ECS contrasted with Object Oriented Design
• Polymorph: a generative approach to a queryless, system oriented ECS.
  • Compile time focus
  • Features
• Polymers companion library
• Why Nim?
• Overview of building an ECS
• After makeEcs/makeEcsCommit
• Defining components
• registerComponents generated types
• Components and instance types
• Instances in components
  • Instance volatility
• Component utilities
  • Matching run time ComponentTypeId
• Defining systems
• System anatomy
  • System scope blocks
  • Work item blocks
  • Accessing items
  • Access templates and aliasing
• System component negation
• Systems without components
• Committing systems
• ecsImport
• System execution order
• Running systems at time intervals
• Adding custom fields to systems
• Grouping systems
  • Execution control
  • Removing components during all or stream blocks
• System utilities
  • clear
  • remove/removeComponents
  • systemsUsed
  • caseSystem
• Working with entities
• Creating entities
• Adding components to existing entities
• Fetching and checking for components
• Removing components
• Deleting entities
• Checking entities satisfy specific systems
• Constructing entities at run time
• Constructing multiple entities
• Cloning entities
• Switching components with component lists
• Updating entities with component lists
• Events
• Inline events
• System events
• Mutating entities within events
• Change events
• onEcsBuilt
• Construction and clone events
  • Construction events
• Writing component libraries
• Code generation options
• EcsCompOptions
• EcsSysOptions
• EcsEntityOptions
• Compile switches
• Performance considerations
• Memory access patterns
• Fragmentation analysis
• Owned components and removing indirection
• Satisfying owner systems
• Deleting from owner systems
• ECS identities
• Multiple ECS outputs with a single identity
• Private ECS
• Multiple identities
• Future work
• Performance

Project overview

A lean, generative abstraction for writing programs with the entity-component-system pattern.

Goals

• Manage complexity with declarative dispatch and run time composition.
• Scalable, low boilerplate platform for composing data oriented designs.
• No runtime, zero system iteration overhead.
• Leverage static typing and metaprogramming to elide run time work.
• Support low resource embedded devices.
• No external dependencies.

Entity-component-system (ECS)

This pattern lets you build composite types at run time and dispatch program logic for specific sets of types.

The three elements of ECS are:

• Entity: a handle that lets you add or remove data types.
• Component: a data type that can be added to an entity.
• System: logic with component parameters that runs for matching entities.

Entity-component-systems offer a way to structure programs 'bottom up' by combining data to compose system behaviour at run time.

An example ECS

import polymorph

# Create some component data types.
register defaultCompOpts:
  type
    Pos = object
      x, y: int
    Vel = object
      x, y: int

# Define some logic for when Pos and Vel are together.
makeSystem "move", [Pos, Vel]:
  all:
    pos.x += vel.x
    pos.y += vel.y

# Generate the ECS for use.
makeEcsCommit "runSystems"

# Compose Pos and Vel to trigger "move".
let
  moving = newEntityWith(
    Pos(x: 0, y: 0),
    Vel(x: 1, y: 1)
  )

# Run the "move" system a number of times.
for i in 0 ..< 4:
  runSystems()

# Confirm the updated Pos value.
let pos = moving.fetch Pos
assert pos.x == 4 and pos.y == 4

Benefits of ECS

• The principles of SOLID
• Inversion of control
• Naturally data oriented and data driven
• Remove ambiguities, coupling, dependencies and slow virtual calls of inheritance trees
• Maintain design agility and rapid prototyping by composing behaviour at run time
• Avoid data coupling and the 'conceptual crystallisation' of top down design
• Encourage decoupled and isolated logic that's easy to reuse, extend, and maintain
• High performance through machine friendly batch processing with uniform lists
• Natively asynchronous

ECS contrasted with Object Oriented Design

Objects in OOD describe is a relationships with hierarchies of types. Subtyping and encapsulation naturally fit a top down design process, where a general overview of a task is split into sub-tasks. The mechanics of execution is often bespoke to each solution.

By contrast, entities in ECS describe has a relationships with sets of types. Mutable composition lends itself more to a bottom up design process, where behaviour is built by composing tasks. Systems dispatch over matching entities automatically in a fixed order.

For a view of ECS from an OOP perspective, see here.

Polymorph: a generative approach to a queryless, system oriented ECS.

Traditionally, ECS implementations orient their data model from the perspective of components (aggregate data) or entities (data compositions).

In order to perform work in these models, a system must query a management layer for live data to process. The efficiency of this process can become a key factor in complex designs.

Polymorph instead orients its data model from the perspective of systems (execution); systems hold component state for matching entities, and are updated when entities add or remove components.

By avoiding the disconnect between execution and state, systems are always available to run without any iteration overhead.

Entity operations are transactional system state changes, and the component types involved infer the systems affected.

This allows entity operations to be entirely generated at compile time, guided by system/component relationships to emit pared down system updates that perform the minimum run time work possible.

The output is statically dispatched with a linear execution flow (no need for callbacks), consisting of simple loops over incrementally updated component data.

Compile time focus

Polymorph takes extensive advantage of Nim's metaprogramming features to shift as much work to compile time as possible.

Functionality such as building entities from blueprints, cloning entities, and debugging utilities are also fully generated from your types and system design at compile time.

The more that state can be determined at compile time, the less run time work is needed. Creating a new entity with a set of components fully specifies the entity state, and systems are unconditionally matched at compile time. Any system updates required are then generated as direct, static updates, without any run time speculative work.

Adding and removing multiple components at once is also minimised at compile time, outputting single pass inline operations.

Features

• Design driven: code generation directly from component/system interactions.
• Zero system overhead: no queries or iteration overhead for systems.
• Cheap to change components: freely evolve entities with wildly disparate components without moving memory.
• Sequential code flow: flatten run time composition, declarative code execution, and event hooks into a linear execution flow without needing callbacks or virtual calls.
• Architecturally simple output: outputs simple loops over lists in a set order. Aims to scale from stack only, low resource environments to cache efficient, high performance data processing.
• Granular code generation options: select different data structures for each component and system, choose error handling mechanisms, system interval execution, indexing and removal strategies, and more - without changing any code.
• Compile time checked:
• Event expansions are parsed for conflicting changes and potential cycles.
• Iterating systems detect component removals that affect themselves:
  • Automatically adjust loop generation.
  • Subsequent access of the iteration entity is a compile time error.
• Compile time optimised:
• Only

Polymers companion library

The Polymers library provides ready-made components and systems as both an effort at a data oriented 'stdlib' and as practical examples:

• OpenGl: render instanced models using the glBits shader wrapper.
• Physics: components for interacting with the Chipmunk2D physics engine.
• Console: reading keyboard and mouse events, writing text with normalised (-1, 1) coordinates.
• Database: performing queries with ODBC.
• Networking: components for:
  • socket TCP/IP (Windows IOCP) and UDP,
  • HTTP processing,
  • Serving webpages,
  • JSON RPC over HTTP.

Why Nim?

Nim is an adaptable language with low development friction, very high performance, and fast compile times. It's built to be readable with a flexible syntax.

The language is extremely portable, compiling to C, C++, ObjC, and JavaScript, along with good Python interop. Nim's static typing and high level abstractions can be shared across domain boundaries and interface with a huge variety of ecosystems.

Extensibility is a core philosophy, with hygienic macros using the language in a VM to process abstract syntax trees directly. Nim's compile time evaluation and well supported metaprogramming make this library possible.

Overview of building an ECS

1. Create component types with register/registerComponents
2. Create system code to process components with defineSystems/makeSystem.
3. Generate the code with makeEcs/makeEcsCommit.

For example:

import polymorph

# Design stage.
register defaultCompOpts:
  type MyComponent = object
makeSystem "mySystem", [MyComponent]: discard

# Seal the design and generate the ECS and system code.
makeEcsCommit "runSystems"

# ECS can now be used.
let e = newEntityWith(MyComponent())
runSystems()

After makeEcs/makeEcsCommit

Once all the components and systems have been defined, the makeEcs macro generates the ECS:

• Macros:
  • newEntityWith: create an entity with a set of components.
  • add/addComponents: add components to an entity.
  • remove/removeComponents: remove components from an entity.
• Procs:
  • newEntity: create an entity.
  • fetch/fetchComponent: returns a component instance from an entity.
  • delete: delete the entity.
  • construct: build an entity from a list of components.
  • clone: copy an entity.
• Systems:
  • By default, system types are created and instantiated.
• Templates:
  • caseComponent: a case statement for run time component type ids.
  • caseSystem: a case statement for run time system ids.
• Debugging:
  • $ operators for entities, component instances, and other supporting types.

Defining components

Components are the data and attributes that make up the program state. They act as the parameters to dispatch systems, and therefore define the granularity of program behaviour. Whilst system program code is fixed at compile time, entities can freely change components to drive system execution.

Component design is conceptually flexible. They may contain all or part of the raw data for systems to process, act as program events, orchestrate collections of entities, modify behaviour as dataless tags, serve as descriptive annotations, and so on as required.

Creating components is as simple as wrapping type definitions with registerComponents. This will process any typedef and pass the block through unaltered.

registerComponents takes two parameters:

• An ECSCompOptions object. This controls code generation for these components.
• A block of code with type definitions.

For example, to create three components, A, B, and C.

import polymorph

# Create components with the default options.
registerComponents defaultCompOpts:
  type
    A = object

    B = object
      text: string

    C = object
      value: int

You can use registerComponents multiple times before actually constructing the ECS, for example to split component definitions over separate modules. Each registerComponents may have separate options.

As a special case, you can include type definitions in registerComponents that should not be made into components with the {.notComponent.} pragma. This can be useful when you want to refer to types within component definitions, but can't (or don't want to) define them externally to the type block.

registerComponents defaultCompOpts:
  type
    SubData {.notComponent.} = object
      contents: seq[int]

    Data = object
      value: SubData

In general, it's recommended to use object types for components for convenience. For non-object types, for example type MyComponent = float, use access to read and update to write the value.

You can also pass existing types to register them as components:

type
  ExternalType = object

register defaultCompOpts:
  ExternalType

registerComponents generated types

Each type passed to registerComponents creates two other types to support its use as a component:

1) An instance type, generated with the Instance postfix. This is used to point to a specific component's data in storage.

2) A container type, generated with the Ref postfix. This is inherited from the Component supertype, and allows seq[Component] with different component types for construct to build arbitrary entities at run time.

Components and instance types

Instance types are distinct integers that reference values for that type, and are how components are generally represented within the ECS.

For example systems use instances by default, and fetch returns them. Any component attached to an entity can be represented by an instance.

It can be useful to understand how this mechanism works for usability and performance reasons.

Polymorph implements the dot accessors . and .= for instance types so that fields can be accessed like the original data type. This works by using the integer value of the instance to index into the type's associated component storage.

import polymorph

registerComponents defaultCompOpts:
  type
    Foo = object
      text: string
      value: int
    Bar = object
      value: float

makeEcs()

# Create an entity with both components.
let
  entity = newEntityWith(
    Foo(text: "Hello!", value: 123),
    Bar(value: 12.34))

# Get instances for the components.
let
  foo = entity.fetch Foo
  bar = entity.fetch Bar

# Act on component fields through instances.
foo.text = "Hello"
foo.value = 17
assert foo.value == 17
bar.value = 5.5

# This is transformed into something like:
storageFoo[foo.int].text = "Hello"
storageFoo[foo.int].value = 17
assert storageFoo[foo.int].value == 17
storageBar[bar.int].value = 5.5

Instance types have reference semantics, and as they are simply integers they're cheap to pass about and validate without invoking the GC. Looking up the index to storage is one indirection with list storages, and that indirection should have a good chance of being in the cache - assuming a linear component allocation pattern.

Sometimes, however, you want access to the underlying type directly. For this, there is the access template, which transposes the storage lookup for the whole component type.

  let copyFoo = foo.access
  # Translates to something like this:
  let copyFoo = storageFoo[foo.int]

You can also update whole components using instances:

foo.update Foo(text: "Hey there", value: 18)

To check if instances returned from fetch were found, use the valid template.

let foo = entity.fetch Foo
if not foo.valid:
  echo "Cannot find Foo on this entity"

Instances in components

registerComponents generates and inserts instance types before the component definitions themselves. This lets you use instance types within component definitions and have type safe links to other components.

Instance types are always defined as the component type name postfixed with "Instance".

# Parse and generate instance types.
registerComponents defaultCompOpts:
  type
    A = object
      b: BInstance
    B = object
      a: AInstance

# Outputs something like:
type
  AInstance = distinct int
  BInstance = distinct int

  A = object
    b: BInstance
  B = object
    a: AInstance

Instance volatility

Component instances don't store generation information, and should be considered: - current within system storages and when directly fetched from entities. - volatile when stored in components or other data structures.

Whilst you can check if a stored instance is valid or alive, potentially the storage it refers to could have been deleted and recreated for another entity.

When storing instances it's up to the user to make sure things make sense, but this is usually easy thanks to the general semantics of how components relate to entities in an ECS and the event system.

For more longer term references, passing a component instance to toRef returns a ComponentRef and captures the generation for the instance. This can be used to detect if a reference to a particular value no longer exists.

Component utilities

• componentCount: retrieves the number of instances in use for a particular component type.

• typeName(typeId: componentTypeId) converts a run time type id to the string of the type it represents.

Matching run time ComponentTypeId

The caseComponent template creates a case statement to handle all components from a run time ComponentTypeId. Within a caseComponent block the details of a component's type can be accessed with various templates. This allows code to perform type specific actions with dynamic component types such as in ComponentList.

Note:

• Has no concept of entity, this is a static case statement with injected actions
• The same action block is compiled for every choice.

The following templates are available with the caseComponent block:

• componentId: the ComponentTypeId being matched.
• componentName: the string of the component name.
• componentType: the type of the component.
• componentRefType: the ref container type of the component.
• componentDel: the delete procedure for manually deleting a component slot.
• componentAlive: the alive proc for this component.
• componentGenerations: the storage list for this component's generation info.
• componentInstanceType: the instance type for this component.
• componentData: the storage list for this component.
• isOwned: returns true when the component is owned by a system, or false otherwise.
• owningSystemIndex: the SystemIndex of the owner system, or InvalidSystemIndex if the component is not owned.
• owningSystem: this is only included for owned components, and references the owner system variable.

Example of use:

import polymorph

registerComponents defaultCompOpts:
  type Comp1 = object

makeEcs()

let
  compList = cl(Comp1())

caseComponent compList[0].typeId:
  echo "Component type is ", componentName

Defining systems

Systems dispatch code over entities with a specific set of components, and program behaviour is directed by the composition of entities. Systems may be completely stand alone, or work together through components as data pipelines.

Systems are efficient: their state is incrementally updated by entity changes so they don't perform work to iterate, and they execute as batch operations without call overhead. Even single line systems can be powerful processing tools.

For systems to participate in an ECS, they must first be defined with defineSystem by passing the name of the system along with the component types it uses. This allows makeEcs to create ECS operations based on how systems and components interact.

After a system is defined, it needs program logic assigned to it in order to perform work. Such logic is referred to as a system body.

A system can only have one body, and this contains all the code the system needs to perform.

When defineSystem is used, you can declare the body of a system later with makeSystemBody. This allows you to keep the code for a system separate from its definition.

Alternatively, you can define a system and its body at the same time with makeSystem, or makeSystemOpts if you wish to specify system compile options.

makeSystem is passed the name and component types like defineSystem. If the system is already defined, makeSystem will just add the system body whilst ensuring that the components and options match the previous definition. It's a compile time error for the types or options to be mismatched between defineSystem and makeSystem for the same system.

Once a system is defined (whether using defineSystem or makeSystem), the system type is created and its variable is instantiated. These variables are created with the system's name, prefixed with sys and are immediately accessible.

import polymorph

registerComponents defaultCompOpts:
  type
    Comp1 = object
    Comp2 = object

# Define a system.
defineSystem "mySystem1", [Comp1]

# Define a system and pass compile options to it.
defineSystem "mySystem2", [Comp1, Comp2], defaultSysOpts

# Define a system and code body at the same time.
makeSystem "mySystem3", [Comp1, Comp2]:
  echo "Hello from system ", sys.name

# Once the ECS is sealed, new systems can't be defined using these components.
# By default, system types are created and instantiated here.
makeEcs()

# Access the instantiated system.
assert sysMySystem1.count == 0

# Define a code body for a previously defined system.
makeSystemBody "mySystem1":
  echo "Hello from system ", sys.name

# Define a code body for a previously defined system with makeSystem.
# Options are retrieved from the system's defineSystem.
makeSystem "mySystem2", [Comp1, Comp2]:
  echo "Hello from system ", sys.name

# Output systems with bodies defined so far.
commitSystems "runSystems"

runSystems()

Outputs:

Hello from system mySystem1
Hello from system mySystem2
Hello from system mySystem3

System anatomy

Systems offer several labelled blocks that allow executing code in different contexts. These are classified into two categories: system scope blocks and work item blocks.

System scope blocks (init, start, and finish) are run at specific points in a system's execution. They may be written at any point in the root of the system's body, but are extracted from the body and pieced together in the order they're written, to be executed within the appropriate scope.

Work item blocks (all and stream) are run for items in the system's work list. These blocks are expanded in place within the system body, allowing multiple processing passes within a system.

Code in the root of a system body is executed when the system is run and paused and disabled are false.

Within a system the sys template refers to the variable for the current system.

System scope blocks

• init: run when the system's initialised field is false. After execution, initialised is set to true.
• start: executed every time a non-disabled system begins running, before the check for sys.paused.

The start block is in the same scope as the system body and the all, stream, and finish blocks, so variables and data defined in start can be used within these blocks.

• finish: run after a system body has finished executing, regardless of sys.paused state.

Work item blocks

These block include an item template to access the current row entity's components. This template provides access to the entity being processed as item.entity, and components the system uses defined as the type name in lower case (eg; item.myComponent).

• all: run for every entity in the system.

• stream: run for an arbitrary number of entities in the system:

• stream: run for up to sys.streamRate entities in order. If sys.streamRate is zero, all entities are processed as if this were an all block.

• stream N: run for up to N entities in order.
• stream multipass: run for sys.streamRate entities, reprocessing entities in order if necessary.
• stream multipass N: run for N entities, reprocessing entities in order if necessary.
• stream stochastic: selects sys.streamRate entities, may select entities multiple times.
• stream stochastic N: selects N entities, may select entities multiple times.

import polymorph, random

registerComponents defaultCompOpts:
  type DemoBlocks = object

makeSystem "allTheBlocks", [DemoBlocks]:
  init:
    echo "Init: first run for ", sys.name
    randomize()

  start:
    echo "Start: system entities: ", sys.count

  echo "System running..."

  template entId: string = "entity " & $item.entity.entityId.int
  let srStr = "(rate: " & $sys.streamRate & ")"

  all:
    echo "  All: ", entId

  stream:
    echo "  Stream: ", srStr," ", entId

  let itemCount = 3
  stream itemCount:
    echo "  Stream up to ", itemCount, ": ", entId

  stream multipass 4:
    echo "  Stream multipass 4: ", entId

  stream stochastic 2:
    echo "  Stream stochastic 2: ", entId

  stream stochastic:
    echo "  Stream stochastic ", srStr, ": ", entId

  echo "System completed."

  start:
    sys.streamRate = 1
    echo "Start: set stream rate to ", sys.streamRate

  finish:
    echo "Finish: finished execution for ", sys.name

makeEcs()
commitSystems "run"
for i in 0 ..< 3:
  discard newEntityWith(DemoBlocks())
run()

The above outputs (stochastic blocks may vary):

Init: first run for allTheBlocks
Start: system entities: 3
Start: set stream rate to 1
System running...
  All: entity 1
  All: entity 2
  All: entity 3
  Stream: (rate: 1) entity 1
  Stream up to 3: entity 2
  Stream up to 3: entity 3
  Stream multipass 4: entity 1
  Stream multipass 4: entity 2
  Stream multipass 4: entity 3
  Stream multipass 4: entity 1
  Stream stochastic 2: entity 2
  Stream stochastic 2: entity 1
  Stream stochastic (rate: 1): entity 3
System completed.
Finish: finished execution for allTheBlocks

Accessing items

When iterating through a system, such as with an all or stream block, the current row is accessed with the item template.

This template contains the current entity, as well as fields for all the components the system uses.

Other components need to be fetched from the entity.

Access to item is checked at compile time for 'use after free' - for example when a system removes rows while iterating.

The injected variable entity contains the entity that the row started with. This is the same as item.entity, but is still valid if the system removes its own rows during iteration.

makeSystem "displayItem", [Comp1]:
  all:
    echo "Current entity: ", item.entity
    echo "Comp1: ", item.comp1
    assert entity == item.entity
    echo entity.fetch(SomeOtherComponent)

Access templates and aliasing

Lots of item ends up becoming boilerplate.

For easier reading, a set of templates is created for each component that does the equivalent to item.component:

makeSystem "displayItem2", [Comp1]:
  all:
    echo "Comp1: ", comp1 # Equivalent to item.comp1.

These can be aliased using colons in the system's component requirements.

Note that this only renames the access template, not the component itself, nor it's field in the current item.

makeSystem "displayItem2", [c1: Comp1]:
  all:
    echo "Comp1: ", c1 # Equivalent to item.comp1.

Aliasing is also useful for component libraries that combine with externally created components:

template systemUsingComp(userComponent: typedesc) {.dirty.} =
  makeSystem "use" & $userComponent, [uc: userComponent]:
    echo sys.name
    all:
      # 'userComponent' is a type.
      # To access it in the current row we have to know the ident in
      # lower case.

      # We could do this:
      echo userComponent.access

      # Aliasing makes things more clear:
      echo uc

register defaultCompOpts:
  type
    MyComp1 = object
      data: int
    MyComp2 = object
      data: int

# Create systems for MyComp1 and MyComp2.
systemUsingComp(MyComp1)
systemUsingComp(MyComp2)

System component negation

Systems can also be defined to only match when components are not present:

makeSystem "notB", [A, not B]:
  discard

Systems without components

Systems can be defined without components in order to run code within the system workflow:

makeSystem "noComponents":
  echo "Runs within the expect system order"

These systems cannot use all or stream blocks, since they don't store components to process.

Committing systems

Once committed, each system is output as a procedure named after the system, prefixed with "do".

Separating makeEcs and commitSystems allows for more design flexibility, for example: - to write code that uses the sealed ECS (after makeEcs) and is also used within system code, - to allow the sealed ECS and system code as separate imports, - to split systems into separate wrapper procs.

Systems bind at the site they're output, not where the body is defined with makeSystem.

By default, systems types and instances are emitted by makeEcs, and commitSystems emits the system procedure.

As such, external routines or variables used by systems need to be accessible at the commit site.

# CommitSystems uses the string parameter to create a wrapper proc that
# runs the systems it outputs in order.
commitSystems "runMySystems"

# Run systems output by `commitSystems` above.
runMySystems()

# You can run systems individually using their `do` proc.
doMySystem()

ecsImport

In this example, foo is only visible in moduleA, but the "useImported" system is actually output in moduleB:

# Module A
from localModule import foo
makeSystem "useImported": echo foo  # Can't find 'foo'!

# Module B
import moduleA
makeEcs()
commitSystems()  # <- "useImported" is emitted here, where 'foo' is not imported.

To help with this kind of deferred output, ecsImport and ecsImportFrom allow deferring imports to makeEcs. Similarly, ecsImportCommit/ecsImportCommitFrom allows deferring imports to just before the system procedure is emitted.

This offers two advantages.

Firstly it lets you write ecsImport alongside your system definitions, and have them emitted where the code is placed.

Secondly, imports are tried relative to the call site path first, and passed through unchanged if that doesn't compile. This lets you spread system declarations and support modules into different directory structures and write the imports as if they were run at the call site.

# Module A
ecsImportFrom localModule, foo
makeSystem "useImported": echo foo  # 'foo' is found!

# Module B
import moduleA
makeEcs()       # <- 'from localModule import foo' emitted here.
commitSystems() # <- "useImported" can now see 'foo'.

System execution order

The order systems are run defines how your ECS operates.

By default, systems are emitted by commitSystems in the order they're defined by defineSystem.

When no matching defineSystem is present, makeSystem will invoke defineSystem for you.

import polymorph

var executionOrder: seq[string]

registerComponents defaultCompOpts:
  type Foo = object

# The order systems are defined sets the order they're run when output
# by `commitSystems`.
defineSystem "a", [Foo], EcsSysOptions(maxEntities: 1)
defineSystem "b", [Foo]

# Define the system "c" and also add a code body.
makeSystem "c", [Foo]:
  executionOrder.add sys.name

# Seal and generate the ECS.
makeEcs()

# Define the code body for "a".
makeSystemBody "a":
  executionOrder.add sys.name

# Both "c" and "a" have bodies waiting to be committed, but "b" doesn't
# have a code body yet and so is not included in the output of `commitSystems`.
#
# As systems are run in the order they're defined, the output proc will
# run "a" then "c".
commitSystems "runAC"

# Define the body for "b". This will be included in the next `commitSystems`.
makeSystemBody "b":
  executionOrder.add sys.name

# Only "b" has an uncommitted code body so the output will just run "b".
commitSystems "runB"

# Execute the two run procs.
runAC()
runB()

# Check the order of execution is as expected.
assert executionOrder == @["a", "c", "b"]

Running systems at time intervals

The timings field in EcsSysOptions lets you control the type of timing code the system uses:

• stNone: the default is to not insert timing code.

• stRunEvery: insert code to allow the system to run at intervals.

Inserts the following fields into the system: - lastTick: keeps track of the last time a system was executed. - runEvery: when this field is non-zero, the system will only trigger after this many seconds. A value of zero runs the system without delay as if it were defined with stNone.

• stProfiling: implies stRunEvery and adds fields for measuring system performance.

The following procedures access per run timings:

• timePerRun: time taken for the last system run.
• timePerItem: the timePerRun divided by the number of items in the system.

Min/max timing procedures return accumulated times over multiple system runs:

• minTimePerItem: the minimum time recorded for a single item so far.
• maxTimePerItem: the maximum time recorded for a single item so far.
• minTimePerRun: the minimum time recorded for a system run so far.
• maxTimePerRun: the maximum time recorded for a system run so far.

These timings can be manually reset with the resetMinMax procedure.

import polymorph, os, times

registerComponents defaultCompOpts:
  type Foo = object
    seen: int

makeSystemOpts "runEvery", [Foo], EcsSysOptions(timings: stRunEvery):
  all: foo.seen += 1

makeSystemOpts "sleepy", [Foo], EcsSysOptions(timings: stProfiling):
  all: sleep(10)

makeEcs()
commitSystems "run"

sysRunEvery.runEvery = 0.1

let entity = newEntityWith(Foo())

for i in 0 ..< 200:
  run()

echo "Last: ", sysSleepy.timePerRun
echo " Min: ", sysSleepy.minTimePerRun
echo " Max: ", sysSleepy.maxTimePerRun

let
  start = cpuTime()

while cpuTime() - start < 1.0:
  run()

echo "Seen: ", entity.fetch Foo

Output (timings will vary as sleep works with OS time slices):

Last: 0.016
 Min: 0.015
 Max: 0.025
Seen: (seen: 37)

Adding custom fields to systems

Passing field definitions to defineSystem will add them to the system variable declaration.

const sysOpts = EcsSysOptions()

defineSystem "mySystem", [Comp1], sysOpts:
  myFieldStr: string

sysMySystem.myFieldStr = "Foo"

makeSystemBody "mySystem":
  echo "myFieldStr = ", sys.myFieldStr

These fields can also be initialised at system creation.

defineSystem "mySystem", [Comp1], sysOpts:
  myFieldStr = "Foo"

When the type cannot be inferred but needs to be initialised, you can explicitly define it.

Since name: type = value isn't valid syntax in this context, the type is defined with ->.

defineSystem "mySystem", [Comp1], sysOpts:
  myFieldStr -> string = "Foo"

Fields can also be added when defining a system and its body with makeSystem by using the fields block. If the system has already been defined these fields are checked to match the definition.

makeSystem "mySystem", [Comp1]:
  fields:
    myFieldInt: int
  echo "This system has a custom field: ", sys.myFieldInt

Grouping systems

Systems can be extracted into separate procedures using defineGroup. Grouping can be useful to separate the concerns of multiple systems to particular procedures.

Systems can be part of multiple groups at the same time.

When a system is grouped, it will not be output by commitSystem and must be manually output using commitGroup.

Grouping can be performed explicitly by passing a list of systems to defineGroup, or ad hoc by using defineGroup without specifying systems. In the latter case, any systems that have been defined but not already grouped or committed are added to the group.

It's a compile time error to try to perform commitGroup for systems that don't have bodies.

Grouping can occur before or after makeEcs.

Group names are case insensitive.

import polymorph

registerComponents defaultCompOpts:
  type Comp1 = object

makeSystem "g1a", [Comp1]: discard
makeSystem "g1b", [Comp1]: discard

# Gather previously defined systems into a group.
defineGroup "group1"

makeSystem "g2a", [Comp1]: discard
makeSystem "g2b", [Comp1]: discard

# Add specific systems to a group.
defineGroup "group2", ["g2a", "g2b"]

makeSystem "ungrouped", [Comp1]: discard

makeEcs()

commitGroup "group1", "runGroup1"
commitGroup "group2", "runGroup2"
# Commit the remaining "ungrouped" system.
commitSystems "runUngrouped"

runGroup1()
runGroup2()
runUngrouped()

Execution control

Systems can be paused by setting the system paused field to true.

When paused, the system body doesn't run, but system scope blocks are still executed. This can be useful for things like self pausing "fire once" systems and conditional system execution.

Systems can be disabled by setting the system disabled field to true. When disabled a system performs no work until disabled is false.

import polymorph

registerComponents defaultCompOpts:
  type DemoExecCtrl = object

makeSystem "fireOnce", [DemoExecCtrl]:
  finish: sys.paused = true

  echo "One time."

makeSystem "fireAlt", [DemoExecCtrl]:
  fields:
    ticks: int
  start:
    sys.paused = sys.ticks mod 3 == 0
    sys.ticks += 1

  echo "Alternating: ", sys.ticks

makeEcs()
commitSystems "run"

for i in 0 ..< 5: run()

Outputs:

One time.
Alternating: 2
Alternating: 3
Alternating: 5

Removing components during all or stream blocks

Operations that remove rows from the currently iterating system in a work item block may invalidate the item template until the next row.

For example item.entity.delete or item.entity.removeComponent with components the system uses (both of which will cause the current row to be removed) causes item refer to a different entity and set of components, or even out of bounds memory!

To force checking this condition each time item is accessed at run time, set assertItem = true in the ECSSysOptions passed when defining the system. This inserts an assert inside the item template to ensure it refers to the same row the iteration step started on, and that the row is within the system's item bounds.

Alternatively, to perform compile time checking with destructive iteration pass -d:ecsStrict when compiling (see compile switches). This will halt compilation when item is used after a system removes components that affect the iterating system, or if item is used after any delete operation. Note that this does not perform semantic analysis of the code, and will respond in the same to conditional remove/delete operations even when they are not triggered.

To help with these cases without needing run time or compile time checks, the entity variable accessible in these blocks references the entity that the row originally started with, regardless of the system state. This can be useful when you want to perform destructive operations to the row without worrying about the item being invalidated:

nim makeSystem "processEntities", [SomeComponent]: all: # Removing a component used by the system will remove # the row and invalidates `item`. entity.remove SomeComponent # Accessing `item` here is a compile error as the row no longer exists. # We can, however, still access the original `entity`. entity.add SomeOtherComponent

Polymorph detects when you add or remove components that affect the currently iterating system within all and stream blocks at compile time, and will add extra checks to ensure iterations stay within bounds.

Deleting entities, however, is opaque to compile time analysis as it cannot be known what components exist on a run time entity. This means these checks will always be added when deleting entities inside all or stream blocks.

To see information about which system iteration loops are affected by removes/deletes, compile with -d:ecsPerformanceHints.

Another option for deleting entities is to use the system's deleteList. Any entity added to this seq will be removed after the finish block is executed, and the list is then cleared. These deletes don't affect iteration, but simple appending may cause a heap allocation and potentially memory moving as part of the standard seq operation. For extensive use of deleteList, it may be worth setting the capacity at the start of system execution.

makeSystem "deleteEntities", [SomeComponent]:
  all:
    sys.deleteList.add entity

System utilities

You can perform delete/remove operations on systems as a whole with the following two operations:

clear

Deletes all entities in the given system. Use with caution!

mySystem.clear

remove/removeComponents

Removes one or more components from all entities in the given system.

mySystem.remove Comp1, Comp2, Comp3

Within system blocks you can use the sys template with these utilities to act on the current system:

makeSystem "removeComp1", [Comp1]:
  finish: sys.remove Comp1

systemsUsed

This utility returns a string containing the systems that would be used for entities with a particular set of components. This can be used as a static debugging tool to check a set of components invokes the systems you expect.

echo systemsUsed([MyComponent1, MyComponent2])

caseSystem

Similar to caseComponent, the caseSystem template creates a case statement that matches a SystemIndex with its instantiation.

This generates a runtime case statement that will perform actions for all systems.

This allows you to write generic code that dynamically applies to any system chosen at runtime.

The SystemIndex of a system variable is accessed from the id field at run time.

Within caseSystem, use the sys template to access to the system variable the index represents, and SystemTupleType to reference the tuple type for the system's groups field (in other words, the type of the system's item when iterating).

Working with entities

Creating entities

Once makeEcs has finished, entities can be created with either newEntity or newEntityWith, where the latter allows setting up entities with components.

In particular, newEntityWith can be a very performant way to create entities, as by definition the entity can only contain the components passed to it, and the systems that need to be updated are fully constrained at compile time.

This means that the output code consists of simply updating the entity's internal list and producing static system updates only where parameter components fully satisfy systems. No conditional work is required.

registerComponents defaultCompOpts:
  type
    Comp1 = object
      value: int
    Comp2 = object
      value: string
    Comp3 = object
      value: float

makeSystem "mySystem", [Comp1, Comp2]:
  all: discard

makeEcs()

let
  # An entity without any components.
  entity = newEntity()

  # Create an entity and include it in "mySystem".
  entWithComps = newEntityWith(
    Comp1(value: 123),
    Comp2(value: "Foo"))

Adding components to existing entities

Entities can be updated 'piecemeal' with add or addComponents.

This operation also supports multiple components at a time. Providing multiple components in one operation (as opposed to several singular add operations) can help inform code generation to confirm systems that are definitely being updated, producing non-conditional code.

Note: it's a run time error to add a component type that already exists on the entity.

Adding components returns the component instances that have been added. For multiple components, this is a named tuple of the components being added. For single components, just the instance is returned.

# `comps` is a named tuple with the instances of the components we've added.
let comps = entity.add(Comp1(value: 456), Comp2(value: "Bar")

echo "Comp1: ", comps.comp1, " Comp2: ", comps.comp2

# For single components, just the component being added is returned.
let comp3 = entity.add(Comp3(value: 0.123))

echo "Comp3: ", comp3

Fetching and checking for components

To retrieve a component from an entity, use fetch/fetchComponent. This takes the type of the component and returns an instance.

Fetching a component that doesn't exist on the entity returns InvalidComponent, and can be checked for with valid.

let comp1 = myEntity.fetch Comp1
if comp1.valid:
  echo "Comp1 was found: ", comp1
else:
  echo "Comp1 was not found."

To see if a component exists on an entity without fetching it, use has/hasComponent.

if myEntity.has(Comp1):
  echo "Comp1 was found!"
else:
  echo "Comp1 was not found."

Removing components

Removing components is performed with remove or removeComponents. Much like adding, this operation also allows multiple components to be removed in a single operation.

This takes the type of components.

# Remove components from the entity and systems using Comp1 and/or Comp2.
entity.remove Comp1, Comp2

Deleting entities

Entities are deleting using the delete operation. This operation is fully run time bound, as components cannot be determined at compile time. However, like other operations, it is generated as a static procedure from your design at compile time, so fewer components in a design will generate less code.

In general, the performance of delete depends on how many components the entity has. Deleting also doesn't need to bookkeep the components being removed like remove does.

# Remove the entity and delete associated component and/or system storage.
entity.delete

Checking entities satisfy specific systems

The expectSystems utility will generate a doAssert operation to ensure that an entity is using particular systems. The code is constructed at compile time based on the systems passed to it and is therefore unique to your design and the systems involved.

When the entity doesn't match the parameter systems and the doAssert fails, the following output is produced:

• the expected systems passed in the parameters,
• the current systems the entity satisfies,
• the missing systems, and components the entity lacks to satisfy these systems,
• the entity's current components,
• a summary of all components required to satisfy all missing systems.

entity.expectSystems ["mySystem1", "mySystem2"]

Constructing entities at run time

makeEcs generates a construct procedure that lets you build entities from lists of components.

This is fairly efficient, as the list is parsed for types then the entity is created in a single integrated operation in a similar way to newEntityWith. As such it can elide the speculation and repeated work of multiple separate addComponent operations.

To allow storing different component types in a single list, registerComponents generates a ref container type for each component, descended from the Component object.

These container types are defined as the component type name postfixed with Ref.

The construct procedure then takes a seq[Component], aliased as ComponentList, to build an entity.

All container types have a typeId that must be initialised with the component's ComponentTypeId in order for construct to extract the value from the subtype. Containers with an uninitialised typeId will cause construct to fail at run time.

The makeContainer template will correctly set up containers for individual components.

let mcContainer = MyComponent(data: 1234).makeContainer

The cl macro makes setting up a ComponentList much more convenient.

This macro lets you mix the original component types and ref container types, and ensures containers are correctly set up. Another advantage of cl is that it always outputs ComponentList, and avoids over constraining the list type when single components are used - for example @[MyComponentRef()] is of type seq[MyComponentRef], not the seq[Component] type that construct expects.

import polymorph

registerComponents defaultCompOpts:
  type
    Comp1 = object
      value: int
    Comp2 = object
      value: string

makeEcs()

let
  entityBlueprint = cl(Comp1(value: 1234), Comp2(value: "Foo"))
  entity = entityBlueprint.construct

let
  c1 = entity.fetch Comp1
  c2 = entity.fetch Comp2

assert c1.value == 1234
assert c2.value == "Foo"

Constructing multiple entities

To create multiple entities in one operation, a seq[ComponentList] is used. This is aliased as ConstructionTemplate.

let
  myEntityBlueprints = @[
    cl(Comp1(value: 1234), Comp2(value: "Foo")),
    cl(Comp2(value: "Bar")),
  ]
  myEntities = myEntityBlueprints.construct

echo myEntities[0], myEntities[1]

Cloning entities

Entities may be duplicated at run time with the generated clone procedure.

let myClose = myEntity.clone

This performs less validation work than construct, as the entity state must already be valid for the entity to exist.

Switching components with component lists

The transition template lets you remove one set of components and add or update another. This can be useful for using components for state machine behaviour. Only components within the parameter lists prevState and newState are considered. Other components are unaffected.

This comes in two flavours: - transition(entity: EntityRef, prevState, newState: ComponentList, transitionType: static[EntityTransitionType])

transitionType controls whether to just update components that are in both states, or to always remove components in prevState and add newState.

• A transition type of ettUpdate will remove components that are in prevState but don't exist in newState, and update components that exist in both prevState and newState. Events such as onAdd/onRemove for updated components are not triggered, the data for the component is just updated.

• A transition type of ettRemoveAdd will always trigger events such as onAdd/onRemove, but does more work when many components are shared between prevState and newState and may reorder more system rows. This can be useful for components containing managed resources and other situations where events must be triggered.

• transition(entity: EntityRef, prevState, newState: ComponentList)

This version calls transition with ettUpdate.

Note: be aware when using transition whilst iterating in a system that removing components the system uses can invalidate the current item template.

Note: as components are added/removed individually, designs with systems that own two or more components may not allow such transitions to compile as they are not added in a single state change.

import polymorph

registerComponents defaultCompOpts:
  type
    A = object
      value: int
    B = object
      value: int
    C = object
      value: string
    D = object
      value: string

makeEcs(defaultEntOpts)

let
  compsAB = cl(A(value: 456), B(value: 789))
  compsAD = cl(A(value: 999), D(value: "Bar"))

  entity = newEntityWith(A(value: 123), C(value: "Foo"))

entity.transition(compsAB, compsAD)
# A is overwritten with the A in `compsAC` and D is added.
# There's no B to remove yet and C is unaffected.
# Entity is now: A(value: 999), C(value: "Foo"), D(value: "Bar")

entity.transition(compsAD, compsAB)
# A is overwritten with the A in `compsAB`, D is removed, and B is added.
# C is unaffected.
# Entity is now (A(value: 456), C(value: "Foo"), B(value: 789))

Updating entities with component lists

You can use update/updateComponents to update multiple components on an entity at once using a ComponentList. This only updates the components that exist on the entity, others in the list are ignored. This can be useful for constructed entities that need context sensitive component initialisation after they've been built, or for applying bulk changes to entities at run time without adding or removing components.

entity.update cl(
    A(value: 765),
    B(value: 825)
  )

Events

Polymorph includes a variety of events for different situations. Most of these are 'inline' and are composed ad hoc when required in the output code. All events are called immediately at the point of invocation.

In general, events are invoked after the state has been fully resolved for events that trigger when 'adding' and before the state has been resolved for 'removing' events.

Inline events

These events are directly injected during a state change without any call overhead. This makes them great for component initialisations/deinitialisations, monitoring, and other light work.

Each event appends code, which is run in the order the event code is added. This allows extending events on components even if they already have existing event code.

System events

These events are invoked when system rows are added or removed, and can use the item template to access components within the system row that's affected.

Mutating entities within events

Events are allowed to add or remove components from their host/calling entity, as long as they don't invalidate other events within a state change.

In other words, event execution is immutable: once a state change occurs, such as adding or removing components, all the events associated with that state change must be allowed to execute.

As long as this condition is respected, events are allowed to freely embed further state changes to the entity.

This means, for example, events are free to remove the component being added within an onAdd event, as long as there aren't subsequent add events for this component scheduled to execute.

The compile time expansion of events checks for the following conflicts:

1. Removing components that invalidate a future event from the same state change (as above).
2. Event cycles, even if called indirectly through other events.
3. Deleting the host/caller entity.
4. Recursive events, for example an onAdd that removes a component, coupled with an onRemove event that adds the same component.

However this analysis is currently limited to code expansion, not semantic analysis. This means it cannot determine conditional event triggers, for example.

Events are a powerful feature, and it's worth being aware of how they affect a design, particularly if your systems are being used by other developers who may use their own events which could be triggered along with, or inside, your event code.

You can see the flow of event expansion by compiling with the -d:ecsLogDetails switch.

Change events

This event is triggered whenever an entity state changes. This includes components being added or removed from entities, new entities are constructed from a template of components, or entities are deleted. However it is not triggered for 'empty' entities such as created with newEntity (since no components have been added yet).

This example shows logging all component changes for entities:

onEntityChange:
  echo "Change: ", entity.entityId.int, ": ", state, ": ", types

onEcsBuilt

Code passed to this macro will be emitted after makeEcs has completed.

This can be useful when you want to define logic that uses the ECS and must be available as soon as possible. For example, for use within system bodies, for general set up situations, or for providing utility functions.

In particular, this is invaluable for library components/systems, where some initialisation or utility procedures may use the ECS, but the library doesn't have control over when makeEcs is run.

registerComponents defaultCompOpts:
  type Counter = object
    value: int

onEcsBuilt:
  # This code is inserted after makeEcs has completed.
  echo "makeEcs has finished!"

  # This procedure uses the ECS.
  proc inc(entity: EntityRef) =
    let counter = entity.fetch Counter
    if counter.valid:
      counter.value.inc

makeEcs()
# Outputs "makeEcs has finished!" to the console, and the `inc` proc is
# included.

let ent = newEntityWith(Counter(value: 1))

# Use the previously defined proc.
ent.inc

assert ent.fetch(Counter).value == 2

Construction and clone events

These events allow you to intercept the construction and cloning of entities and edit, replace, ignore, or provide multiple components in response.

Changing the component types being added is not possible with other events.

To register these events makeEcs must have been run, and as such they have full access to ECS operations.

One use for these kind of events is for blueprinting run time only data such as external resources. One component type can be used to provide information for instantiation, and is replaced with a different component that represents initialised data. This can be useful for separating the concerns of systems.

Another example is for meta-components. One component can contain information that adds one or more other components when passed to construct. Components can also create and manage sets of entities on construction as well.

Construction events

• registerConstructor takes a component type and a proc of the form: nim proc (entity: EntityRef, component: Component, context: EntityRef): seq[Component]

• entity: the current entity being constructed.

• component: the Component container supertype passed from the ComponentList being built.

  To get the value stored within, type cast to the container subtype for the component you're hooking and access its value field.

  The component's container type is created by registerComponents as the component name with the Ref postfix, for example a MyComponent type would use MyComponentRef(component).value.

• context: the construct proc allows optionally passing an entity to provide input to construction events.

  If no entity is provided, context will match the entity parameter.

  When construct creates multiple entities from a ConstructionTemplate, the first entity built is always passed as context to these events. This allows you to define the context for multiple entities in the ConstructionTemplate itself.

Components are added by appending to the seq[Component] result. It's possible to add any number of components to the result, as long as there aren't any repeated types. It's also valid to not add to the result and elide the component from the entity.

```nim registerComponents defaultCompOpts: type Original = object data: int Replaced = object data: int

makeEcs()

# This event will replace the Original component type with Replaced # during construction. proc replaceOriginal(entity: EntityRef, component: Component, context: EntityRef): seq[Component] = let original = OriginalRef(component).value result.add Replaced(data: original.data)

registerConstructor Original, replaceOriginal

# Build an entity from a component list using the Original type. let entity = Original(data: 1234).cl.construct

# Confirm the replaced component. assert not entity.has(Original) assert entity.has(Replaced) assert entity.fetch(Replaced).data == 1234 ```

• registerPostConstructor: takes a component type and proc of the form: nim proc (entity: EntityRef, component: ComponentRef, entities: var Entities)
• entity: the entity being constructed.
• component: the component ref that's been assigned to the entity. To convert this to an instance to access it, type cast the index field with the component's instance type, which is the type postfixed with Instance. For example for MyComponent the instance would be accessed with MyComponentInstance(component.index).
• entities: the list of entities that will be returned to the user.

These events are called after multiple entities are built by using construct with a ConstructionTemplate. The event allows work to be performed on the fully constructed sets of entities.

It does not allow changing the type of components as they're added like registerConstructor, but allows full manipulation of the fully constructed entities afterwards.

This can be useful to update components that keep track of other entities, or perform other multi-entity work.

```nim proc countEntities(entity: EntityRef, component: ComponentRef, entities: var Entities) = echo "I counted: ", entities.len

registerPostConstructor MyOtherComponent, countEntities ```

• registerCloneConstructor: takes a component type and proc of the form: nim proc (entity: EntityRef, component: ComponentRef): seq[Component]
• entity: the new cloned entity.
• component: the source component being cloned