Skip to content

Relations and maps#

A relation is a set of combinations of labels of dimensions, such as the buses that each generator may connect to. A map gives each row of a dimension or a relation one label of a dimension, such as the bus of each generator. The file declares both, and the rows come with the data. This page lists the keys of relations: and maps:, the data they expect, and how each operator uses them.

relations#

Each key under relations: declares one relation.

dimensions:
  bus: { dtype: str }
  generator: { dtype: str }
  snapshot: { dtype: int }
relations:
  connection: { dims: [generator, bus], missing: absent } # the pairs that are wired
  neighbour: { dims: { from: snapshot, to: snapshot }, missing: absent }

Each row of connection is one pair of a generator and a bus, and a pair with no row is not in the relation.

Key What it sets Default
dims required. The columns of the relation, at least two
missing refused or absent: whether every combination needs a row refused
description free text null

A column is named after its dimension, or after no dimension. Where two columns share a dimension, each column has its own name in the mapping form, as in {from: snapshot, to: snapshot}. A column named after a dimension that it is not over is refused, because mathspec reads the name of a column as its dimension. With zp: { dims: { period: zone, generator: generator } }, mathspec prints:

Relation 'zp' names column 'period' after dimension 'period', but the column is over 'zone'. Rename the column after what it holds.

A relation with one column is refused, because one column holds labels of one dimension, and a map or a condition states that. With r: { dims: [generator] }, mathspec prints:

Relation 'r' has one column under 'dims:', and a relation needs at least two. To give each generator a label, declare a map under maps: generator:. To mark some of them, declare a bool parameter or a condition.

A relation is not a dimension, so its name does not go in dims:, over= or along=. With flow: { dims: [connection] }, mathspec prints:

Variable 'flow' names the relation 'connection', and a relation is not a dimension. Name its dimensions, ['bus', 'generator'], under dims:, and confine the entry with in: [connection].

The order of the columns under dims: changes nothing. The typesetter sorts the columns by name, so connection above prints as \(\mathrm{connection} \subseteq \mathcal{B} \times \mathcal{G}\).

maps#

Each key under maps: names a subject, which is a dimension or a relation. Each key under the subject declares one map, which gives every row of the subject one label of the dimension that refers_to: names.

dimensions:
  bus: { dtype: str }
  generator: { dtype: str }
  line: { dtype: str }
maps:
  generator:
    bus: bus # each generator is on one bus
  line:
    bus0: bus
    bus1: { refers_to: bus, missing: absent } # a line may have an open end

bus: bus is the short form of bus: { refers_to: bus }, and mathspec writes a map back in the short form when refers_to: is its only key. A condition or an operator reads a map as subject[map], so generator[bus] is the bus of a generator.

Key What it sets Default
refers_to required. The dimension whose labels the map gives
missing refused or absent: whether every row of the subject needs a label refused
description free text null

A map is named after the dimension it refers to, or after no dimension. The typesetter prints a map as a function from its subject into that dimension, so generator[bus] prints as \(\mathrm{bus}: \mathcal{G} \to \mathcal{B}\). mathspec refuses a map at load in four cases:

  • The subject is neither a dimension nor a relation.
  • refers_to: names a relation, or names no dimension.
  • The map is named after a dimension that it does not refer to.
  • The map has the name of a column of its subject.

With generator: { period: zone }, mathspec prints:

maps: 'generator': map 'period' is named after dimension 'period', but refers to 'zone'. Rename the map after what it holds.

A relation with two columns over one dimension takes no maps, because a map gives one label per combination of the dimensions of its subject.

Cardinalities#

How many labels a row may have (the cardinality) decides whether a fact is a map or a relation:

What the file states Written Cardinality
each generator has one bus maps: { generator: { bus: bus } } many-to-one
a generator may connect to several buses relations: { connection: { dims: [generator, bus], missing: absent } } many-to-many
a generator has one zone in each period relations: { zone_of: { dims: [generator, period] } } and maps: { zone_of: { zone: zone } } many-to-one, from a pair
a snapshot has a month, a week and a weekday maps: { snapshot: { month: month, week: week, weekday: weekday } } three maps of one subject
a line has two ends, both buses maps: { line: { bus0: bus, bus1: bus } } two maps into one dimension
a snapshot has a representative snapshot maps: { snapshot: { rep: snapshot } } many-to-one, onto itself
a snapshot has neighbours relations: { neighbour: { dims: { from: snapshot, to: snapshot }, missing: absent } } many-to-many, onto itself

missing#

missing: sets whether the data may leave something out. On a relation, it sets whether every combination of the dimensions under dims: needs a row. On a map, it sets whether every row of the subject needs a label.

dimensions:
  generator: { dtype: str }
  bus: { dtype: str }
  line: { dtype: str }
relations:
  connection: { dims: [generator, bus], missing: absent } # most pairs are not wired
maps:
  generator:
    bus: bus # refused: every generator is on a bus
  line:
    bus0: bus
    bus1: { refers_to: bus, missing: absent } # an open end is meant
A map with missing absent is absent at a row with no label The dimension line has three labels: l1, l2 and l3. The map line[bus1], declared with missing absent, gives each line a bus. Its data gives l1 the bus north and l2 the bus south, and gives l3 no label. At l3 the map line[bus1] is absent. line a dimension line l1 l2 l3 line[bus1] gives each line a bus line[bus1] a map line bus1 l1north l2south no label for l3: line[bus1] is absent no label in line[bus1] absent: no row

At l3, the data gives line[bus1] no label, so line[bus1] is absent there.

missing: Result
refused, by default a tool refuses the data when it is attached, and names the combination or the row
absent a combination with no row is not in the relation, and a row with no label maps to nothing, which is absent

No number can replace a missing label, so at load mathspec refuses a value such as missing: 0:

maps.generator.bus.missing: missing: 0 on a map, which takes refused or absent. A row with nothing there is refused, or is not in the data.

Under the default refused, a relation needs a row for every combination, so connection would need every generator wired to every bus. Most relations are sparse, so they declare missing: absent.

The typeset legend lists `connection` is `absent` where the data has no row and `line[bus1]` is `absent` where the data gives no label beside each dimension they are over, and nothing for refused.

The data contract#

With data attached, each relation and each subject with maps has one table, named after it. The table of a relation has one column per column under dims:. The table of a subject also has one column per map, named after the map. So the data for the maps of line is a table line with the columns line, bus0 and bus1. A piece of software that reads a spec (a tool) requires five things of a table:

  • The table has one row per combination of the columns of its subject, so a generator with two rows, on two buses, is refused.
  • Every label in the column of a map is a label of the dimension it refers to, and a label that matches none is refused.
  • The table has a row for every combination of its dimensions, unless the relation declares missing: absent. The table of a dimension has a row for every label of the dimension.
  • A null is refused, except in the column of a map that declares missing: absent.
  • The order of the rows has no meaning, because the order comes from the dimension.

A relation and a map have no where:, so their rows and labels come from the data alone. A tool therefore knows them before it evaluates any condition, such as the where: of a parameter.

The rows of a relation can also be the coordinates of a parameter or a variable, its index set. An entry with in: [connection] has coordinates only at the pairs that connection lists, and a tool refuses a row of the data of a parameter at any other pair (in). A map is not an index set, because it adds no row to its subject.

An assumption can require rows of a relation or labels of a map. For line[bus1] above, {holds: "line[bus1]", where: "closed"} requires an end for every closed line, where closed is a bool parameter over line.

How relations and maps are used#

A relation fixes no direction, so the same table can map generators to buses or buses to generators. A call joins the expression it acts on (its operand) to the table on their shared columns, and then groups the rows it has joined. In a call, s[c] names the subject s and its column c, and s[c, …] names several columns of one table. A column is a map of s, or one of the columns of a relation under dims:. The subject appears once, so one call uses one table.

Kind What it does Written as
join and group by rows of the operand match rows of the table, and each group is added up sum(x, over=d, by=s[c])
join each row of the operand matches one row of the table, and takes its label at(x, by=s[m])
partition the dimensions stay, and the rows are grouped shift, sum_back, position with within=s[m]
test a coordinate stays where the relation has a row, or where the map has a label r or s[m] in a condition

Four rules apply to every call:

  1. A call lists every column it uses.
  2. A map that the call does not list is not used.
  3. The key of a subject cannot change in a call.
  4. A dimension that no column of the table is over passes through to the result.

Under the first rule, mathspec refuses at(x, by=generator), and the message lists the maps of the subject:

Constraint 'c': at(by=generator) names the subject and none of its maps. Write by=generator[<map>], with a map of 'generator': ['bus'].

Sums#

sum(x, over=d, by=s[c]) sums x over the columns in over=, grouped by the columns in by=. As in sum(x, over=d), over= lists what the result loses and by= lists what it gains, and either one may be a list. A sum through a table adds each row of x only to the groups that the table lists for it. A row that the table does not list is left out.

A sum through a map joins, then groups The variable p is a table with three rows: wind, gas and solar. The map generator[bus] has three rows: wind on north, gas on north and solar on south. The sum of p over generator by generator[bus] first joins p to the table of generator on generator, which gives each row of p its bus. It then adds up the rows of each bus. The result is a table with one row per bus: north holds p of wind plus p of gas, and south holds p of solar. maps: generator: { bus: bus } variables: p: { dims: [generator] } sum(p, over=generator, by=generator[bus]) p a variable generator wind gas solar generator[bus] a map generator bus windnorth gasnorth solarsouth join on generator the join generator bus windnorth gasnorth solarsouth sum by bus: adds up the rows of each bus the result an expression bus value northp[wind] + p[gas] southp[solar] the group of bus north the group of bus south

north adds up wind and gas, and south is solar alone.

Each column of the table is either joined on or not, and either grouped by or not, and the call sets both:

Column of the table Joined on Grouped by In the result
a column in over= yes no removed
a column in by= no yes added
a key column the call does not list yes yes kept
a map the call does not list no no not used

With the relation zone_of over [generator, period] and its map zone, the relation connection over [generator, bus], and a variable p over [generator, period], three sums have three results:

Call Joins on Groups by Result
sum(p, over=generator, by=zone_of[zone]) generator, period zone, period [period, zone]
sum(p, over=period, by=zone_of[zone]) period, generator zone, generator [generator, zone]
sum(p, over=generator, by=connection[bus]) generator bus [bus, period]

Five more rules apply to a sum through a table:

  • The operand is over every column that the call joins on, and over none that it groups by. A column that the operand is over is joined on.
  • A name in over= is a column of the table, or else a dimension. With the map rep of snapshot, sum(x, over=snapshot, by=snapshot[rep]) joins on the key column snapshot and groups by rep. Where two columns are over one dimension, the name selects which one, so sum(x, over=from, by=neighbour[to]) joins on from and groups by to.
  • A name in both over= and by= is refused.
  • A dimension in over= that no column is named after is summed away after the call groups the rows. So sum(p, over=[generator, period], by=generator[bus]) adds up each bus over every period.
  • A sum adds up several rows per group, and where the columns grouped by are the whole key, every group is one row. A sum whose groups are one row each is a join with no group-by, which is at.

Under the first rule, with load over [bus], mathspec refuses sum(load * p, over=generator, by=generator[bus]) and prints:

Constraint 'c': sum(by=generator[bus]) groups by ['bus'], which the expression already carries.
Move the factor carrying ['bus'] outside the operator, or group by a map into another dimension.

load * sum(p, over=generator, by=generator[bus]) is accepted, because the sum no longer has load inside it.

Under the last rule, with p over [generator, period, zone], mathspec refuses sum(p, over=zone, by=zone_of[generator, period]) and prints:

Constraint 'cap': sum(by=zone_of[generator, period]): the columns this sum groups by, ['generator', 'period'], hold the whole key ['generator', 'period'], so every group is one row and the sum adds nothing. Write at(..., by=zone_of[zone]), or group by a map.

Lookups#

at(x, by=s[m]) takes the value of x at the label that map m gives each row of s. It joins x on the maps in by= and groups by the key, so every group is one row. A lookup adds up nothing, and each row of the result is the value of x at one coordinate. With a variable build over [bus], at(build, by=line[bus0]) has one row per line:

at reads one bus value once for each line at that bus The variable build has one row per bus, north and south. The map line[bus0] puts lines l1 and l2 on bus north and line l3 on bus south. at of build by line[bus0] has one row per line: l1 and l2 both read build at north, and l3 reads build at south. maps: line: { bus0: bus } build a variable bus north south line[bus0] a map line bus0 l1 north l2 north l3 south at(build, by=line[bus0]) an expression, one row per line line value l1 build[north] l2 build[north] l3 build[south] read at bus one row per line

Lines l1 and l2 both take the value of build at north, and l3 takes it at south.

Three rules apply to a lookup:

  • A lookup reads only maps. A key column in by= is refused, and so is a relation with no map, where one combination can have several rows.
  • The result gains every key column that the operand is not already over. A key column over a dimension that the operand is over is joined on too, unless a map in by= already refers to that dimension. With price over [zone, period], at(price, by=zone_of[zone]) joins on zone and period, and its result is [generator, period]. at(x, by=snapshot[rep]) joins on rep alone, so the result gains the key column snapshot.
  • No two maps in by= refer to one dimension, because the operand has one coordinate along each of its dimensions.

With at(build, by=line[line]), mathspec prints:

Constraint 'c': at(by=line[line]) names ['line'], a column of 'line' itself. A lookup reads maps at the row. Name one of its maps, ['bus0', 'bus1'].

Partitions#

shift(x, along=d, within=s[m]), sum_back(x, along=d, within=s[m]) and position(d, within=s[m]) step along the key column over d. They join on the other key columns, and split the rows into groups by the maps in within=. The dimensions of the result do not change. The result is absent at a coordinate that the map gives no group. With the map season of snapshot, shift(soc, along=snapshot, offset=1, edge='wrap', within=snapshot[season]) steps back within each season:

A partition by snapshot[season]: shift wraps inside each season The variable soc has four rows, snapshots 1 to 4. The map snapshot[season] puts snapshots 1 and 2 in winter and snapshots 3 and 4 in summer. shift of soc along snapshot, offset 1, edge wrap, within snapshot[season], has one row per snapshot. Snapshot 2 reads soc at 1 and snapshot 4 reads soc at 3. Snapshot 1, the first of winter, wraps to soc at 2, the last of winter, and snapshot 3, the first of summer, wraps to soc at 4. maps: { snapshot: { season: season } } shift(soc, along=snapshot, offset=1, edge='wrap', within=snapshot[season]) soc a variable snapshot 1 2 3 4 snapshot[season] a map snapshot season 1 winter 2 winter 3 summer 4 summer the result an expression snapshot value 1 soc[2] 2 soc[1] 3 soc[4] 4 soc[3] one back group by season 1 and 3 wrap inside their season the group of season winter the group of season summer

The first snapshot of each season takes the value of the last snapshot of the same season, and no value crosses from winter to summer.

within= may list two maps into one dimension, but not a key column. A relation with no map makes no groups, so mathspec refuses it in within=:

Constraint 'd': shift(within=sp[period]): 'sp' has no map, so it makes no groups. Declare the group as a map of the dimension shift steps along, under maps: snapshot:.

Tests#

A condition tests a relation or a map at the coordinates of its own entry. It compares a map, written s[m], against a label, compares two maps, or tests that a row or a label exists. Conditions lists when two maps may be compared. A comparison reads one map, so snapshot[month, weekday] == 'x' is refused, and mathspec suggests the rewrite on the second line of the message:

Variable 'x': Cannot parse the condition "snapshot[month, weekday] == 'x'".
a comparison reads one map of 'snapshot'. Compare each map on its own: snapshot[month] == … AND snapshot[weekday] == ….
Expected end of text, found '['  (at char 8), (line:1, col:9)

A map alone in a condition, such as line[bus1], is true where the map gives the row a label. A relation name alone is true where the relation has a row. On a parameter or a variable, in: states those rows, so mathspec refuses the name in its where:. With in:, a variable is absent at every pair that the relation does not list:

flow.yaml
dimensions:
  generator: { dtype: str }
  bus: { dtype: str }
  snapshot: { dtype: int }
relations:
  connection: { dims: [generator, bus], missing: absent } # the pairs that are wired
parameters:
  load: { dims: [bus, snapshot] }
variables:
  flow:
    dims: [generator, bus, snapshot]
    in: [connection]
    bounds: { lower: 0 }
constraints:
  balance:
    dims: [bus, snapshot]
    expression: sum(flow, over=generator) == load
A relation in in: gives the pairs it lists The relation connection is a table with three rows: wind and north, gas and north, and gas and south. The variable flow, declared with in connection, is a table with the same three pairs of generator and bus. The pair wind and south, which connection does not list, has no row, so flow is absent there. connection a relation generator bus windnorth gasnorth gassouth in: [connection] gives the pairs the relation lists flow a variable generator bus windnorth gasnorth gassouth no row for (wind, south): flow is absent absent: no row

flow exists at the pairs that connection lists and is absent at every other pair, so balance adds up the generators wired to each bus. where: connection in place of the in: is refused:

Variable 'flow': `where: connection` tests the rows of 'connection', which `in:` states. Write `in: [connection]` in place of the `where:`.

python -m mathspec markdown flow.yaml prints the domain of flow and the constraint:

\[ \mathit{flow}_{b,g,t} \ge 0 \qquad \forall\, \left( b,\ g \right) \in \mathrm{connection},\ t \in \mathcal{T} \]
\[ \sum_{g \in \mathcal{G}} \mathit{flow}_{b,g,t} = \mathrm{load}_{b,t} \qquad \forall\, b \in \mathcal{B},\ t \in \mathcal{T} \]

The domain of flow is the pairs (b, g) in connection, at every snapshot. The columns are sorted to (bus, generator). A map test prints as \(\mathrm{bus1}(l) \text{ is defined}\).

See also