Input

A system receives stimuli by declaring stimuli as a read. It gets every stimulus since it last ran, oldest first.

defineSystem({
    reads: ["stimuli"],
    run({ stimuli }: { stimuli: Stimulus[] }) {},
})

`Stimulus`

type Stimulus =
    | { type: "cognet:stimulus:text"; time: Time; data: TextStimulus }
    | { type: "cognet:stimulus:audio"; time: Time; data: AudioStimulus }
    | { type: "cognet:stimulus:visual"; time: Time; data: VisualStimulus }
    | { type: "cognet:stimulus:vector"; time: Time; data: VectorStimulus }

type TextStimulus = { channel: string; content: string; format?: "json" | "markdown" }
type AudioStimulus = { channel: string; ref: Ref; transcript?: string; durationMs?: number; sensed?: number }
type VisualStimulus = { channel: string; ref: Ref; kind: "image" | "video"; caption?: string; sensed?: number }
type VectorStimulus = {
    channel: string
    values: number[]
    unit?: string
    units?: string[]
    labels?: string[]
    profile?: string
    sensed?: number
}

type Ref = { uri: string; mime: string; bytes?: number }
type Time = { ms: number; seq: number }

Fields

Field	Kinds	Is
`channel`	all	the sense it came from, as the body declared it
`sensed`	audio, visual, vector	when it was measured, ms on the body's clock. Absent when the body can't say — then it was sensed when it arrived
`time`	all	when it arrived: `ms`, and `seq`, its place in the session
`content`	text	the text
`format`	text	`json` or `markdown`; absent is plain prose
`ref`	audio, visual	where the bytes are — never the bytes
`transcript`	audio	what was said, when the body already knew
`durationMs`	audio	how long it is
`kind`	visual	`image` or `video`
`caption`	visual	what it shows, when the body already knew
`values`	vector	one reading: every number sensed in the same instant
`unit`	vector	the unit, when every number shares one
`units`	vector	each number's unit — same length as `values`
`labels`	vector	each number's name — same length as `values`
`profile`	vector	the convention it follows, such as `sensor_msgs/Imu` — advisory

Units are SI: metres, seconds, radians.

Example

run({ stimuli }) {
    for (const stimulus of stimuli) {
        if (stimulus.type !== "cognet:stimulus:vector" || stimulus.data.channel !== "/imu") continue
        const [ax, ay, az, gx, gy, gz] = stimulus.data.values
        const sensed = stimulus.data.sensed ?? stimulus.time.ms
        // …
    }
}

`Sense`

What the body declares about each sense, from kernel.body().senses. Find senses here by kind and profile, never by a channel you assumed.

type Sense = VectorSense | VisualSense | AudioSense | TextSense

type SenseBase = { channel: string; frame?: string; rate?: number; profile?: string }
type VectorSense = SenseBase & { kind: "vector"; components: Component[] }
type VisualSense = SenseBase & { kind: "visual"; width: number; height: number; lens: Lens; pixel: Pixel }
type AudioSense = SenseBase & { kind: "audio"; sampleRate: number; channels: number }
type TextSense = SenseBase & { kind: "text" }

type Component = { name: string; unit: string; range?: [number, number]; noise?: number; categories?: string[] }
type Pixel = { kind: "colour" } | { kind: "depth"; range: number } | { kind: "measure"; unit: string; range?: [number, number] }

Field	Is
`channel`	where its stimuli arrive
`frame`	whose axes its numbers or picture are in — a part of the body, or a place. Absent for what has no direction
`rate`	readings a second, nominal. Every reading carries its own `sensed`
`profile`	the convention its readings follow
`components`	vector: each number in `values`, in order — its `name`, `unit` (SI, or `category`), `range`, and `noise`: one standard deviation, what an estimator weighs it by
`width`, `height`	visual: size in pixels
`lens`	visual: how the picture lies on the world — `perspective`, `orthographic` or `polar`
`pixel`	visual: what a pixel measures — `colour`, `depth` (metres, up to `range`), or a `measure`
`sampleRate`, `channels`	audio: samples a second, and how many channels