Skip to content

docs: comprehensive documentation uplift#3

Merged
cigrainger merged 7 commits into
mainfrom
cg/docs-uplift
Mar 23, 2026
Merged

docs: comprehensive documentation uplift#3
cigrainger merged 7 commits into
mainfrom
cg/docs-uplift

Conversation

@cigrainger

@cigrainger cigrainger commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

What

Best-in-class documentation for Dux — function groups, rewritten guides with real datasets, cheatsheet overhaul, comprehensive function docs with doctests.

Function documentation

  • 43 doctests (up from 37) — every public function in Dux and Dux.Graph now has at least one meaningful doctest
  • @doc group: metadata on all 38 public functions: Constructors, Transforms, Aggregation, Joins, Reshape, Sorting, IO, Materialization, Distribution
  • Improved docs for: local/1, from_csv/2, from_parquet/2, from_ndjson/2, to_csv/3, to_ndjson/3, ungroup/1, to_tensor/2, head/2, from_edgelist/2, degree/1, vertex_count/1, edge_count/1, triangle_count/1
  • Rewrote Dux moduledoc: build → compile → execute lifecycle, quick start with penguins, distribution overview
  • Added Dux.Remote moduledoc with Mermaid architecture diagram

Guides (all runnable Livebooks, verified cell-by-cell)

Guide Dataset Cells verified
getting-started.livemd penguins 10/10
data-io.livemd (NEW) penguins 4/4
transformations.livemd (NEW) penguins 8/8
joins-and-reshape.livemd (NEW) nycflights13 star schema 7/7
distributed-queries.livemd synthetic + workers verified
graph-analytics.livemd Zachary's karate club 6/6

Dux.Datasets module

Ships CC0 datasets in priv/datasets/ so guides work with Mix.install:

Dux.Datasets.penguins()    # 344 rows
Dux.Datasets.flights()     # 6,099 rows (nycflights13)
Dux.Datasets.airlines()    # 16 rows
Dux.Datasets.airports()    # 1,458 rows
Dux.Datasets.planes()      # 3,322 rows
Dux.Datasets.gapminder()   # 1,704 rows

Cheatsheet

Complete rewrite with ex_doc features:

  • {: .col-2} for macro vs raw SQL side-by-side
  • All join types, aggregate functions, DuckDB-specific features
  • FLAME in distributed section

ex_doc configuration

  • Function grouping via groups_for_docs
  • Module groups: Core API (Dux, Query, Datasets), Graph, Distribution (Remote, Coordinator, Worker, Broadcast, Shuffle), Integrations (Telemetry, Flame)
  • Mermaid diagram support via before_closing_body_tag
  • nest_modules_by_prefix for Dux.Remote.*

Other

  • README updated: removed Rust/NIF references, pure Elixir + ADBC
  • All module groups correct (fixed Shuffle, Datasets placement)

Test plan

  • 43 doctests pass
  • 440 total tests, 0 failures
  • mix docs builds without errors
  • mix credo --strict zero issues
  • mix compile --warnings-as-errors clean
  • Each guide verified cell-by-cell

🤖 Generated with Claude Code

cigrainger and others added 7 commits March 23, 2026 11:18
- Added @doc group: metadata to all 38 public functions in Dux module:
  constructors, transforms, aggregation, joins, reshape, sorting,
  io, materialization, distribution
- Updated mix.exs docs config: groups_for_docs, groups_for_modules
  (removed deleted Dux.Remote, added Telemetry/Flame), groups_for_extras,
  nest_modules_by_prefix, Mermaid via before_closing_body_tag
- Placeholder files for new guides (data-io, transformations, joins)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrote guides as runnable Livebooks using embedded datasets:
- getting-started.livemd — penguins dataset, admonitions, progressive
- data-io.livemd — NEW: CSV/Parquet/NDJSON/globs, SQL escape hatch
- transformations.livemd — NEW: filter/mutate/window/sort deep dive
- joins-and-reshape.livemd — NEW: nycflights star schema, all join
  types, pivot_wider/longer, concat_rows
- distributed-queries.livemd — rewritten with distribute API, Mermaid
  diagram, FLAME, telemetry
- graph-analytics.livemd — rewritten with karate club dataset

Updated README: removed Rust/NIF references, pure Elixir + ADBC.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete cheatsheet overhaul:
- col-2 for macro vs raw SQL (filter, mutate, aggregation, pivot)
- col-2 for read vs write IO
- All join types with examples
- Materialization section with all output functions
- Distributed section with FLAME
- Graph section with all algorithms + distribution
- Nx interop
- DuckDB-specific features (MEDIAN, PERCENTILE_CONT, window functions)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dux.Datasets provides clean access to embedded CC0 datasets:
- penguins (344 rows), gapminder (1,704), flights (6,099),
  airlines (16), airports (1,458), planes (3,322)

Datasets ship in priv/datasets/ (included in hex package) so
guides work with Mix.install — no path hacks needed.

All 5 guides verified cell-by-cell:
- getting-started: 10/10 cells pass
- transformations: 8/8 cells pass
- joins-and-reshape: 7/7 cells pass
- graph-analytics: 6/6 cells pass
- data-io: 4/4 cells pass

435 tests, 0 failures, 0 credo issues.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Moduledoc-only module explaining the distributed system:
architecture diagram (Mermaid), components (Worker, Coordinator,
Broadcast), data flow (S3-native reads, IPC transfer), FLAME link.

Added back to groups_for_modules in docs config.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrote from a terse reference card to a proper introduction:
- Quick start with penguins dataset
- How it works: build → compile → execute lifecycle
- Expression syntax with interpolation explanation
- _with variants for raw DuckDB SQL
- Distribution overview with link to Dux.Remote
- Embedded datasets reference

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Function docs improved across Dux and Dux.Graph:
- local/1: explain inverse of distribute, add doctest
- from_csv/2: add doctest with airlines dataset
- from_parquet/2: add round-trip doctest
- from_ndjson/2: explain format, add doctest
- to_csv/3, to_ndjson/3: add examples, document return value
- ungroup/1: explain context, add doctest
- to_tensor/2: proper @doc with type conversion docs
- head/2: explicit default of 10, add no-arg doctest
- Graph.from_edgelist/2: add options section and doctest
- Graph.degree/1: explain both endpoints, add doctest
- Graph.vertex_count/1, edge_count/1: add doctests
- Graph.triangle_count/1: document distributed support

Fixed module grouping: added Dux.Remote.Shuffle and Dux.Datasets
to groups_for_modules.

43 doctests (up from 37), all passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@cigrainger
cigrainger merged commit eda684a into main Mar 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant