<!--
Copyright (C) 2023 Luke Shumaker <lukeshu@lukeshu.com>
SPDX-License-Identifier: GPL-2.0-or-later
-->
# lowmemjson
`lowmemjson` is a mostly-compatible alternative to the standard
library's [`encoding/json`][] that has dramatically lower memory
requirements for large data structures.
`lowmemjson` is not targeting extremely resource-constrained
environments, but rather targets being able to efficiently stream
gigabytes of JSON without requiring gigabytes of memory overhead.
## Compatibility
`encoding/json`'s APIs are designed around the idea that it can buffer
the entire JSON document as a `[]byte`, and as intermediate steps it
may have a fragment buffered multiple times while encoding; encoding a
gigabyte of data may consume several gigabytes of memory. In
contrast, `lowmemjson`'s APIs are designed around streaming
(`io.Writer` and `io.RuneScanner`), trying to have the memory overhead
of encode and decode operations be as close to O(1) as possible.
`lowmemjson` offers a high level of compatibility with the
`encoding/json` APIs, but for best memory usage (avoiding storing
large byte arrays inherent in `encoding/json`'s API), it is
recommended to migrate to `lowmemjson`'s own APIs.
### Callee API (objects to be encoded-to/decoded-from JSON)
`lowmemjson` supports `encoding/json`'s `json:` struct field tags, as
well as the `encoding/json.Marshaler` and `encoding/json.Unmarshaler`
interfaces; you do not need to adjust your types to successfully
migrate from `encoding/json` to `lowmemjson`.
That is: Given types that decode as desired with `encoding/json`,
those types should decode identically with `lowmemjson`. Given types
that encode as desired with `encoding/json`, those types should encode
identically with `lowmemjson` (assuming an appropriately configured
`ReEncoder` to match the whitespace-handling and special-character
escaping; a `ReEncoderConfig` with `Compact=true` and all other
settings left as zero will match the behavior of `json.Marshal`).
For better memory usage:
- Instead of implementing [`json.Marshaler`][], consider implementing
[`lowmemjson.Encodable`][] (or implementing both).
- Instead of implementing [`json.Unmarshaler`][], consider
implementing [`lowmemjson.Decodable`][] (or implementing both).
### Caller API
`lowmemjson` offers a [`lowmemjson/compat/json`][] package that is a
(mostly) drop-in replacement for `encoding/json` (see the package's
documentation for the small incompatibilities).
For better memory usage, avoid using `lowmemjson/compat/json` and
instead use `lowmemjson` directly:
- Instead of using <code>[json.Marshal][`json.Marshal`](val)</code>,
consider using
<code>[lowmemjson.NewEncoder][`lowmemjson.NewEncoder`](w).[Encode][`lowmemjson.Encoder.Encode`](val)</code>.
- Instead of using
<code>[json.Unmarshal][`json.Unmarshal`](dat, &val)</code>, consider
using
<code>[lowmemjson.NewDecoder][`lowmemjson.NewDecoder`](r).[DecodeThenEOF][`lowmemjson.Decoder.DecodeThenEOF`](&val)</code>.
- Instead of using [`json.Compact`][], [`json.HTMLEscape`][], or
[`json.Indent`][]; consider using a [`lowmemjson.ReEncoder`][].
- Instead of using [`json.Valid`][], consider using a
[`lowmemjson.ReEncoder`][] with `io.Discard` as the output.
The error types returned from `lowmemjson` are different from the
error types returned by `encoding/json`, but `lowmemjson/compat/json`
translates them back to the types returned by `encoding/json`.
## Overview
### Caller API
There are 3 main types that make up the caller API for producing and
handling streams of JSON, and each of those types has some associated
types that go with it:
1. `type Decoder`
+ `type DecodeArgumentError`
+ `type DecodeError`
* `type DecodeReadError`
* `type DecodeSyntaxError`
* `type DecodeTypeError`
2. `type Encoder`
+ `type EncodeTypeError`
+ `type EncodeValueError`
+ `type EncodeMethodError`
3. `type ReEncoder`
+ `type ReEncoderConfig`
+ `type ReEncodeSyntaxError`
+ `type BackslashEscaper`
* `type BackslashEscapeMode`
A `*Decoder` handles decoding a JSON stream into Go values; the most
common use of it will be
`lowmemjson.NewDecoder(r).DecodeThenEOF(&val)` or
`lowmemjson.NewDecoder(bufio.NewReader(r)).DecodeThenEOF(&val)`.
A `*ReEncoder` handles transforming a JSON stream; this is useful for
prettifying, minifying, sanitizing, and/or validating JSON. A
`*ReEncoder` wraps an `io.Writer`, itself implementing `io.Writer`.
The most common use of it will be something along the lines of
`out = lowmemjson.NewReEncoder(out, lowmemjson.ReEncoderConfig{…})`.
An `*Encoder` handles encoding Go values into a JSON stream.
`*Encoder` doesn't take much care in to making its output nice; so it
is usually desirable to have the output stream of an `*Encoder` be a `*ReEncoder`; the most
common use of it will be
`lowmemjson.NewEncoder(lowmemjson.NewReEncoder(out, lowmemjson.ReEncoderConfig{…})).Encode(val)`.
`*Encoder` and `*ReEncoder` both tend to make many small writes; if
writes are syscalls, you may want to wrap their output in a
`bufio.Writer`.
### Callee API
For defining Go types with custom JSON representations, `lowmemjson`
respects all of the `json:` struct field tags of `encoding/json`, as
well as respecting the same "marshaler" and "unmarshaler" interfaces
as `encoding/json`. In addition to those interfaces, `lowmemjson`
adds two of its own interfaces, and some helper functions to help with
implementing those interfaces:
1. `type Decodable`
+ `func DecodeArray`
+ `func DecodeObject`
2. `type Encodable`
These are streaming variants of the standard `json.Unmarshaler` and
`json.Marshaler` interfaces.
<!-- packages -->
[`lowmemjson`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson
[`lowmemjson/compat/json`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson/compat/json
[`encoding/json`]: https://pkg.go.dev/encoding/json@go1.18
<!-- encoding/json symbols -->
[`json.Marshaler`]: https://pkg.go.dev/encoding/json@go1.18#Marshaler
[`json.Unmarshaler`]: https://pkg.go.dev/encoding/json@go1.18#Unmarshaler
[`json.Marshal`]: https://pkg.go.dev/encoding/json@go1.18#Marshal
[`json.Unmarshal`]: https://pkg.go.dev/encoding/json@go1.18#Unmarshal
[`json.Compact`]: https://pkg.go.dev/encoding/json@go1.18#Compact
[`json.HTMLEscape`]: https://pkg.go.dev/encoding/json@go1.18#HTMLEscape
[`json.Indent`]: https://pkg.go.dev/encoding/json@go1.18#Indent
[`json.Valid`]: https://pkg.go.dev/encoding/json@go1.18#Valid
<!-- lowmemjson symbols -->
[`lowmemjson.Encodable`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#Encodable
[`lowmemjson.Decodable`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#Decodable
[`lowmemjson.NewEncoder`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#NewEncoder
[`lowmemjson.Encoder.Encode`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#Encoder.Encode
[`lowmemjson.NewDecoder`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#NewDecoder
[`lowmemjson.Decoder.DecodeThenEOF`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#Decoder.DecodeThenEOF
[`lowmemjson.ReEncoder`]: https://pkg.go.dev/git.lukeshu.com/go/lowmemjson#ReEncoder