Skip to content

Work with data in pieces

Large data often arrives in pieces: a reader's partitions, a simulation's blocks, the share of each GPU or MPI rank. Tack keeps each piece a whole DataSet and adds three things: a container for the pieces, marks saying which cells and points another piece owns, and reductions that combine the pieces into the whole's answer.

  • A PartitionedDataSet is an ordered list of datasets, as VTK's vtkPartitionedDataSet. Each piece has its own topology, numbering, geometry, fields, sets and device. Iterate over it, index it, take its length.
  • Work on one piece takes the piece. Filters, analyses and for_each take a DataSet, as everywhere else, and you loop over the pieces. There is no "apply to every piece" wrapper: the loop says what it does.
  • Only algorithms whose answer needs every piece take the container: statistics and entropy, given a field's name, here; probing, particles, connected regions and the point-cloud filters in Algorithms Across Pieces.

Cut a grid into pieces

split cuts a structured grid by index ranges. ghost_layers=1 adds a layer of each neighbour's cells around every piece:

    whole = td.wavelet(10)                                   # 21^3 points, 8000 cells
    pieces = td.split(whole, 4, ghost_layers=1)              # a PartitionedDataSet
    for piece in pieces:
        marks = piece.ghosts
        print(piece.topology.point_dims,
              owned(marks.cells, piece.num_cells), "cells owned of", piece.num_cells,
              owned(marks.points, piece.num_points, td.Ghosts.SKIPPED_POINTS),
              "points owned of", piece.num_points)

Each piece of this 20 × 20 × 20-cell grid owns 2000 cells and holds 420 more from its neighbours. Every cell and point that several pieces hold has exactly one owner; in every other piece its copy is a ghost. The owned cells of the four pieces are the grid's 8000 cells, and the owned points its 9261 points.

An unstructured mesh is cut by VTK or by the application that made it (see Pieces from VTK).

Ghost marks

data.ghosts is a tack.dataset.Ghosts, or None for data with no pieces. It holds one u8 per cell (ghosts.cells) and one per point (ghosts.points), either of which may be None, with VTK's vtkGhostType bits:

Bit Value Meaning
Ghosts.DUPLICATECELL 1 a copy of a cell another piece owns
Ghosts.REFINEDCELL 8 a cell a finer AMR level covers: valid data, superseded (AMR Hierarchies)
Ghosts.HIDDENCELL 32 a cell whose data is not valid here: a blanked cell
Ghosts.DUPLICATEPOINT 1 a copy of a point another piece owns
Ghosts.HIDDENPOINT 2 a point whose data is not valid here
Ghosts.REFINEDPOINT 8 a point at whose location a finer AMR level has one (Tack's; VTK has no such bit)

The other cell bits (HIGHCONNECTIVITYCELL, LOWCONNECTIVITYCELL, EXTERIORCELL) are carried along unread. Filters that make cells leave Ghosts.MASKED_CELLS (hidden and refined) out. Ghosts.SKIPPED_CELLS and Ghosts.SKIPPED_POINTS are the bits a reduction leaves out. The marks are kept apart from the fields, as the geometry is, so no algorithm mistakes them for data. to_device moves them; vtk_to_dataset and dataset_to_vtk map them to and from vtkGhostType arrays. Set them yourself with data.ghosts = td.Ghosts(cells=..., points=...); marks of the wrong length or type are refused.

Work on each piece

Loop over the pieces:

    surfaces = []
    for piece in pieces:
        piece.fields["grad"] = td.gradients(piece, "RTData")
        surfaces.append(td.contour(piece, "RTData", 160.0))
    surfaces = td.PartitionedDataSet(surfaces)

The ghost layer is why this works near a piece's edge. A point gradient at the edge of a piece averages over every cell around the point, and the ghost layer supplies the cells across the cut, so each piece's gradients at its own points are the whole grid's.

Filters process the ghost cells too, and carry the marks to what they make, as VTK's do. What a filter makes takes the owner of what it came from:

Output Owner
a cell (a contour's triangles, a clip's pieces, a kept cell) its source cell's
a point a filter keeps its own
a point on an input edge (a contour's, a clip's crossings) the edge's: the owner of its smallest point
a point inside a cell (a clip's centroids) the cell's
a surface face the cell it is a side of
a face two cells share, an edge, a vertex the owner of its smallest point

"Smallest" goes by position, so every piece that holds a shared edge or face decides alike, with no communication. With a ghost layer, the piece that owns a point holds every cell around it. So the owned part of a filter's output, over all pieces, is the whole's output, cell for cell and point for point. That holds for every filter that makes cells or points, chained filters included. The ghost output itself is what the next step near the edge needs: point normals, smoothing, a second filter that looks at neighbours.

Reduce over the pieces

    grid_stats = td.statistics(pieces, "RTData")             # each point counted once
    surface_stats = td.statistics(surfaces, "RTData")        # each surface point once
    triangles = sum(owned(s.ghosts.cells, s.num_cells) for s in surfaces)

statistics(pieces, name) takes each piece's moments on its own device and combines them pairwise; entropy(pieces, name, bins) sums each piece's histogram over the whole's range. Both count only what each piece owns (cells and points not marked duplicate or hidden), on a single dataset too. So the statistics of the pieces' isosurfaces are the whole grid's isosurface's, and so is the owned triangle count.

An L2 field goes by its cells' marks. A field on faces or edges, or of order 2, is refused on a dataset with marks, since who owns those values is not decided yet.

Remove the ghosts at the end

Before rendering, writing one file, or handing a piece to code that does not know ghost marks, take the ghost output away:

    finished = td.PartitionedDataSet(td.remove_ghosts(s) for s in surfaces)

remove_ghosts drops the duplicate and hidden cells, and the points no remaining cell uses, as VTK's vtkRemoveGhosts. The points it keeps keep their marks: a point a piece shares is still its owner's, so reductions over the result are unchanged. A point the piece owns stays even when only ghost cells used it. A contour point on an edge belongs to the owner of the edge's smallest point, and that piece's triangles around the edge may all be ghosts. A structured piece whose remaining cells form a block stays a structured grid. dataset_to_vtk keeps the marks, so pieces with ghosts can also go to VTK as they are; VTK leaves ghosts out when it renders.

Pieces need ghost cells

Without ghost cells, a piece cannot tell what its neighbours make or keep from the cells, faces and edges they share. Take a contour point on an edge in the cut: it belongs to the owner of the edge's smallest point, and that piece may have no cell containing the edge. Or a shared point only a neighbour's kept cells use: it disappears from its owner's threshold. Tack does not guess:

    bare = td.split(whole, 4)                                # no ghost cells
    cut = td.PartitionedDataSet(td.contour(piece, "RTData", 160.0) for piece in bare)
    try:
        td.statistics(cut, "RTData")
    except NotImplementedError as error:
        print("refused:", str(error).split(":")[0])

Such outputs record that they lost the marks, and statistics and entropy refuse them with NotImplementedError, naming the cause. What does not depend on neighbours still works on pieces without ghost cells:

  • reductions of the pieces themselves;
  • filters that keep the cells and points (warp, transform, cylindrical, spherical, shrink, as_polyhedra, as_polygons, and probe's target);
  • points chosen by their own values (threshold_points, extract_points, mask_points).

Setting an output's ghosts clears the record.

Hidden cells

Filters never process hidden cells: their data is not valid there. Every filter that makes cells or points leaves them out, as if they were not there:

  • contour, slice, clip, threshold, extract_cells, extract_geometry, mask, extract_surface, extract_edges, triangulate, tetrahedralize;
  • the source dataset of probe, advect and streamlines.

A field passed as an object rather than by name follows along. Two things cannot leave hidden cells out yet and are refused on such data: faces or edges chosen by id, and a reused CellLocator. Algorithms whose output is on the same cells (gradients, projections, calculators) still compute there, and the marks stay on those cells.

Run and check

uv run python docs/examples/data_partitioned.py --check
uv run python packages/tack-vis/examples/44_partitioned_data.py --pieces 8

The checks compare the pieces against the whole grid: owned cells and points, statistics of the grid and of its gradients, and the isosurface's triangles and points, before and after remove_ghosts. Download the complete program.

Continue with Algorithms Across Pieces: pieces from VTK, generated ghost cells, global ids, several devices and MPI ranks, and the algorithms whose answer crosses pieces.