Standard default output should be a valid JSON array

[simonw]

I just spotted list-buckets has the same not-quite-newline-delimited JSON output format, which is a bad default. I should fix that too.

Originally posted by @simonw in https://github.com/simonw/s3-credentials/issues> /28#issuecomment-1014838721

[simonw]

The current documented display options are:

Add --nl to collapse these to single lines as valid newline-delimited JSON.

Add --array to output a valid JSON array of objects instead.

Defaulting to pretty-printed invalid newline-delimited JSON was a weird design choice that I made!

% s3-credentials list-buckets
{
    "Name": "aws-cloudtrail-logs-462092780466-f2c900d3",
    "CreationDate": "2021-03-25 22:19:54+00:00"
}
{
    "Name": "simonw-test-bucket-for-s3-credentials",
    "CreationDate": "2021-11-03 21:46:12+00:00"
}

Fixing this would require a major version bump if I had hit 1.0 already.

[simonw]

In that case I think the safest default would be to turn the above default into a pretty-printed streaming JSON array:

% s3-credentials list-buckets
[
  {
    "Name": "aws-cloudtrail-logs-462092780466-f2c900d3",
    "CreationDate": "2021-03-25 22:19:54+00:00"
  },
  {
    "Name": "simonw-test-bucket-for-s3-credentials",
    "CreationDate": "2021-11-03 21:46:12+00:00"
  }
]

Then drop the --array option but keep --nl, which would output this (already implemented):

% s3-credentials list-buckets
{"Name": "aws-cloudtrail-logs-462092780466-f2c900d3", "CreationDate": "2021-03-25 22:19:54+00:00"}
{"Name": "simonw-test-bucket-for-s3-credentials", "CreationDate": "2021-11-03 21:46:12+00:00"}

And add a new --csv option.

[simonw]

Getting this right will mean I can pipe into sqlite-utils insert easily to create a SQLite database, which would be fun.

Actually this works already:

s3-credentials list-buckets --nl | sqlite-utils insert /tmp/s3.db buckets - --nl

[simonw]

Here's how the current list-buckets implementation works:

s3-credentials/s3_credentials/cli.py

Lines 556 to 564 in aa69024

	if array:
	gathered.append(bucket)
	else:
	if nl:
	click.echo(json.dumps(bucket, default=str))
	else:
	click.echo(json.dumps(bucket, indent=4, default=str))
	if gathered:
	click.echo(json.dumps(gathered, indent=4, default=str))

I need a new abstraction I can call that knows how to turn an iterator of rows into one of the desired formats.

[simonw]

Most interesting new implementation here will be the code that knows how to output something like this in a streaming fashion, without buffering it all in an array first:

[
  {
    "Name": "aws-cloudtrail-logs-462092780466-f2c900d3",
    "CreationDate": "2021-03-25 22:19:54+00:00"
  },
  {
    "Name": "aws-sam-cli-managed-default-samclisourcebucket-1ksajo4h62s07",
    "CreationDate": "2020-06-16 23:13:34+00:00"
  },
  {
    "Name": "blah-bucket-blah",
    "CreationDate": "2021-11-10 23:50:08+00:00"
  }
]

Trick will be to output [ at the start, then two-space indented (with textwrap) json.dumps(..., indent=2) rows with commas after each except the last one - and then a ] at the end.

[simonw]

Good code to imitate from sqlite-utils: https://github.com/simonw/sqlite-utils/blob/74586d3cb26fa3cc3412721985ecdc1864c2a31d/sqlite_utils/cli.py#L1589-L1623 - in particular this CSV/TSV bit:

writer = csv_std.writer(sys.stdout, dialect="excel-tab" if tsv else "excel")
writer.writerow(headers)
for row in cursor:
    writer.writerow(row)

Also https://github.com/simonw/sqlite-utils/blob/74586d3cb26fa3cc3412721985ecdc1864c2a31d/sqlite_utils/cli.py#L2489-L2514

def output_rows(iterator, headers, nl, arrays, json_cols):
    # We have to iterate two-at-a-time so we can know if we
    # should output a trailing comma or if we have reached
    # the last row.
    current_iter, next_iter = itertools.tee(iterator, 2)
    next(next_iter, None)
    first = True
    for row, next_row in itertools.zip_longest(current_iter, next_iter):
        is_last = next_row is None
        data = row
        if json_cols:
            # Any value that is a valid JSON string should be treated as JSON
            data = [maybe_json(value) for value in data]
        if not arrays:
            data = dict(zip(headers, data))
        line = "{firstchar}{serialized}{maybecomma}{lastchar}".format(
            firstchar=("[" if first else " ") if not nl else "",
            serialized=json.dumps(data, default=json_binary),
            maybecomma="," if (not nl and not is_last) else "",
            lastchar="]" if (is_last and not nl) else "",
        )
        yield line
        first = False
    if first:
        # We didn't output any rows, so yield the empty list
        yield "[]"

[simonw]

Actually I probably want to use csv.DictWriter here:

writer = csv.DictWriter(sys.stdout, headers)
dict_writer.writeheader()
dict_writer.writerows(iterator_of_dictionaries)

[simonw]

This outputs 2-indented JSON in a streaming fashion:

def output_rows_json(iterator):
    # We have to iterate two-at-a-time so we can know if we
    # should output a trailing comma or if we have reached
    # the last row.
    current_iter, next_iter = itertools.tee(iterator, 2)
    next(next_iter, None)
    first = True
    for row, next_row in itertools.zip_longest(current_iter, next_iter):
        is_last = next_row is None
        data = row
        line = "{firstchar}{serialized}{maybecomma}{lastchar}".format(
            firstchar="[\n" if first else "",
            serialized=textwrap.indent(json.dumps(data, indent=2, default=repr), '  '),
            maybecomma="," if not is_last else "",
            lastchar="\n]" if is_last else "",
        )
        yield line
        first = False
    if first:
        # We didn't output any rows, so yield the empty list
        yield "[]"

Demo:

print("\n".join(output_rows_json([{"id": 1, "name": "Simon"}, {"id": 2, "name": "Cleo"}, {"id": 3, "name": "Azi"}])))

[
  {
    "id": 1,
    "name": "Simon"
  },
  {
    "id": 2,
    "name": "Cleo"
  },
  {
    "id": 3,
    "name": "Azi"
  }
]

[simonw]

Turned that into a TIL: https://til.simonwillison.net/python/output-json-array-streaming