# Backlog

## Current Tasks

- [ ] export: generate a [katydid](https://katydid.github.io/) expression, for a compatible subset
- [ ] pr: add _known limitations_ stuff
- [ ] front: merge many internal options to trigrams
- [ ] bench: case failures handling needs some fixing
- [ ] tests: check more bad urls (eg bad %-escape)
- [ ] tests: check pydantic generated classes
- [ ] static: use combined type test and value extraction when available?
- [ ] static: recursive multi-discriminator hierarchisation? See UI5.
- [ ] static: factor out common checks on `&` with shared constraints?
- [ ] static: check for common object part?
- [ ] front: generate valid minimal, maximal and random values from a model
- [ ] front: generate invalid minimal values from a model
- [ ] optim: simplify some '!=' cases
- [ ] optim: collect and display stats about what works
- [ ] spec: allow to _remove_ definitions, eg `{ "$stuff": {"/": "$foo"}}`
- [ ] spec: restrict regex syntax to a smaller subset
- [ ] perf: add another loop over performance collection?
- [ ] perf: drop and/or merge identical results when comparing
- [ ] backend: check `mv-34/mu` test case which may require some utf normalization
- [ ] runtime: count graphemes instead of characters? for Java, see `java.text.BreakIterator`
- [ ] bench: add stddev performance computation
- [ ] bench: improve resilience to bad overhead estimation
- [ ] bench: collect JSON parse times for comparison with validation times
- [ ] pr: better document optimizations, eg add compiler optimization page
- [ ] static: improve startswith pattern recognition `/^xyz.*/`
- [ ] c-backend: factor out str extraction in or-list in more cases
- [ ] c-backend: improve unique with hash using int set?
- [ ] c-backend: improve str hash?
- [ ] c-backend: use str cmp for ends with optimization as well
- [ ] c-backend: improve partitioned cset prettyprinting
- [ ] c-backend: use str set partitioning
- [ ] c-backend: simplify parameters for accelerated functions
- [ ] c-backend: fast str comparison could take advantage of known strlen??
- [ ] c-backend: property name length for jansson is stored at `(str_ptr - sizeof(size_t))`
- [ ] c-backend: use knowledge of Jansson hash to improve property lookup??
- [ ] c-backend: generate custom character class ascii tables?
- [ ] c-backend: custom char classes: `[:-] [a-zA-Z0-9_\\.\\-\\|@#] [a-zA-Z_$] [a-zA-Z0-9_\\-] [a-zA-Z0-9_.:-]`
- [ ] c-backend: handle small char classes?
- [ ] c-runtime: use own hash instead of qsort/bsearch stuff?
- [ ] static: could sorted property name comparison be used for some scans??
- [ ] backend: check leap-second support on `$DATETIME` and `$TIME`
- [ ] tests: fast mode which does not execute if source has not changed
- [ ] predefs: extension for _uri reference_, _iri reference_, _uri template_
- [ ] predefs: _url_ vs _uri_ vs _iri_
- [ ] predefs: improve `$URI` and `$URL`
- [ ] predefs: add `$SIMPLE_REGEX` for regex syntax subset?
- [ ] docker: enable user remap
- [ ] export: java classes
- [ ] export: go interfaces
- [ ] export: ts interfaces
- [ ] static: make regex groups non capturing when appropriate?
- [ ] static: xor acceleration based on distinct mandatory props
- [ ] static: merge partial mandatory prop discriminants
- [ ] static: merge may/must property handling, or not?
- [ ] static: take into account str _byte_ len when partitioning
- [ ] static: possibly use up to 8 bytes when hashing?
- [ ] static: fix hash size to depends on byte encoding, not str len
- [ ] static: check discriminator first? it help only if it appears first in the values
- [ ] perf: test partitioning for js, java, perl, python…
- [ ] perf: investigate best settings for js, java and python
- [ ] perf: allow to run with any docker image
- [ ] static: maybe skip must checks if all must props already seen
- [ ] backend: cleanup unused assigned variable, see `mv-08/openapi-311.c`
- [ ] backend: fix `unused but set variable` warning and remove `-Wno-…` (dead code)
- [ ] backend: remove useless `ELSE NULL` see `mv-02/geo.sql`
- [ ] command: export before/after model optimization
- [ ] perf: experiment object matching with various mandatory/optional properties
- [ ] backend: optimize pattern `if (res) { res = True; if (res) ... }`
- [ ] backend: optimize pattern "res = E1; return res || E2;" (only one use)
- [ ] backend: optimize pattern "res = E; if (! res) ret F; res not used before set;"
- [ ] backend: optimize pattern "{ decl X = ... ; res = f(X (once) ) }"
- [ ] backend: simplify assignment patterns with ternary operator when available?
- [ ] backend: consider applying language-specific formatting tools? `clang-format`, `black`…
- [ ] backend: actually remove ignore code and sequences
- [ ] c-backend: get pval string into a variable for strcmp unrolling and inlining
- [ ] c-backend: add number of chunks limit to strcmp optimization
- [ ] c-backend: strcmp to const inlining and unrolling for all instances
- [ ] c-backend: more typed cset optimizations?
- [ ] static: partial discriminator with else?
- [ ] static: optimize not string cst test?
- [ ] static: close may/must? merge with open?
- [ ] static: add a dynamic test for different strategies in object checking
- [ ] backend: refactor ir optimizations
- [ ] artifact: make work with dind (docker compose?)
- [ ] static: recognize already compile objects and reuse function esp under no-reporting?
- [ ] tests: add non regression tests to docker build
- [ ] service: json to lang conversion API
- [ ] api: web interface with caching and security
- [ ] web-frontend: dev as a use case for jm
- [ ] service: minimal website with editor
- [ ] tests: fix test with generated json model meta schema
- [ ] java: joy/johnzon bugs `mv-16/or_0b` JSONP, 100E99 <> 10E100; other JSONP issues, maybe int/float
- [ ] pr: write `optimizations.model.json` for illustration
- [ ] pr: write `transformation.model.json` for illustration
- [ ] pr: sync version number for **all** published parts for _2.0_ release
- [ ] design: transformation rename, `~` with a map? move path to path? eg mandatory to optional?
- [ ] design: add predefs for `$UUIDx`…
- [ ] predefs: implement `$I32 $I64 $U32 $U64`
- [ ] export: improve pydantic, do not generate bad fields (dash and other chars)
- [ ] tests: some pydantic exports? imports?
- [ ] frontends: reject json with duplicated props
- [ ] c-backend: header file for generated code?
- [ ] backend: implement str comparison using some fast hash in some cases?
- [ ] command: allow to pass some options from rc file?
- [ ] static: shorten xor exit?
- [ ] export: sqlalchemy?
- [ ] predefs: add country codes, eg `$COUNTRY $COUNTRY2 $COUNTRY3 $COUNTRYN`
- [ ] spec: be precise about predefs standard references, eg `$DATE` and co.

## Internship

- [x] pr: read www, papers, spec… setup dev env, report issues
- [x] pr: try tutorial, report issues and suggest improvements
- [x] tests: add a few tests for jsonrpc model
- [x] model: write a model for MCP
- [x] tests: add a few tests for MCP model
- [x] export: generate ts interfaces
- [x] model: fix IR model and tests
- [x] front: IR re-entrance
- [ ] tests: generate valid (pseudo-random) values
- [ ] tests: generate invalid minimal, maximal, pseudo-random values
- [ ] tests: generate invalid values on each constraint
- [ ] backend: C and runtime with json-c (\*, but beware of runtime integration)
- [ ] backend: C# (\*\*)
- [ ] backend: go (\*\*\*)
- [ ] backend: kotlin (\*\*)
- [ ] backend: C++ (\*\*\*\* see existing fork)
- [ ] backend: scala
- [ ] backend: php
- [ ] backend: ruby
- [ ] backend: lua
- [ ] backend: rust (\*\*\*\*\*)
- [ ] backend: test and fix existing dart backend MR
- [ ] export: generate annotated java interfaces/classes?
- [ ] export: katydid?
- [ ] api: json model as a service backend with flask

## Re Do

- [ ] pr: publish web page
- [ ] pr: publish python jmc package
- [ ] pr: publish python jsu package
- [ ] pr: publish js runtime package
- [ ] pr: publish pgxn package
- [ ] pr: publish docker image
- [ ] pr: publish perl package
- [ ] pr: publish java package
- [ ] pr: check jsonschema-benchmark integration
- [ ] pr: check jsu jsts coverage
- [ ] pr: renew domain

## To Do

- [ ] pr: add docker compose for postgres testing?
- [ ] schema: inline external reference to generate valid schemas
- [ ] xstatic: use threshold limit for cset as well
- [ ] xstatic: any array, skip empty code
- [ ] backend: add check constraint variants per constant type?
- [ ] xstatic: compile exreg to regex when possible
- [ ] xstatic: add option support to property regex?
- [ ] optim: simplify float constraints
- [ ] command: add bash completion
- [ ] python: separate python interface in some file
- [ ] python: check python interface consistency
- [ ] tests: reorganize tests cases in smaller thematic directories
- [ ] tests: review allowed exceptions on schemas
- [ ] model: fix rw issue in `openapi-311-fuzzy`
- [ ] pl-backend: fix reporting
- [ ] sql-backend: fix generated code under `--reporting`
- [ ] js-backend: cleanup map and set
- [ ] js-backend: `let` vs `var` consistency?
- [ ] doc: document `re2` support for C
- [ ] design: rewrite reuse path? rather force defs?
- [ ] tests: with multiple regex engine
- [ ] tests: with multiple java json libs
- [ ] use: js/ts type export?
- [ ] use: py type dataclass/pydantic export?
- [ ] use: test with corrector? ref? kiva?
- [ ] static: sort list of models? option?
- [ ] static: fix double reports, eg constraint type and constraints or object props and object
- [ ] py-backend: fix failure verbosity `ref/json-model.py.check` vs others
- [ ] static: keep deeper names?
- [ ] optim: allow more substitutions _after_ transformations? when/where to trigger optims?
- [ ] py-backend: heterogeneous enum (`enum_05`) improved implementation with `ConstSet`
- [ ] py-backend: heterogeneous disjunction with `ConstMap`?
- [ ] py-backend: unique for scalars (`set` only works for str)
- [ ] static: make loose/strict work locally, triggered by `$name` root comments
- [ ] optim: root xor should be simplified in `geo.model.json`, maybe with inlining?
- [ ] python: how to keep looseness after inlining?
- [ ] static: multi-type object disjunction?
- [ ] tests: fix `./rwt/json-schema-draft-next-fuzzy.x.c unused variable res`
- [ ] tests: improve schema tests by allowing more cases
- [ ] static: improve failure reporting of extended regex
- [ ] design: add `$URI-REF` or similar? `$PATH`?
- [ ] c-backend: add reporting to predef failures?
- [ ] c-backend: generate thread-safe code? option? this currently would require malloc'ed stuff
- [ ] tests: put generated exe in `/dev/shm` instead of locally from `Makefile`?
- [ ] static: improve constraint failing message (length, unique…)
- [ ] schema: inline external references to generate working schemas in more cases? (openapi-311)
- [ ] perf: geo, ansible meta, openapi
- [ ] c-backend: multithreading? one one value??
- [ ] pr: deb and rpm packaging? github package distribution?
- [ ] pl-backend: run perltidy?
- [ ] docker: improve caching/no-caching of some commands (eg force weekly rebuild?).
- [ ] jsu-model: generate `"#.title"` and others to keep all schema docs
- [ ] py-backend: package independent runtime?
- [ ] command: improve cache to keep track of urls?
- [ ] project: establish priority
- [ ] project: improve backlog management with thematic sections
- [ ] design: $-refs may only appear in defs for simpler compiler AND better readability?
- [ ] project: move all code's TODO here
- [ ] py-backend: multi-type enum `enum_05`
- [ ] optim: more constraint simplifications, eg redundant `<` and `<=`.
- [ ] static: if loose, cast all int constants to double?
- [ ] c-backend: add arr and obj to cst management?
- [ ] c-backend: more separate headers to avoid unused warnings.
- [ ] c-backend: add a common prefix to all internal functions.
- [ ] design: should it allow unexpected ints as constants?
- [ ] dynamic: JSON IR to lambda?!
- [ ] backend: add JSON output?
- [ ] dynamic: rely on static and eval
- [ ] frontend: add multi-level naming option
- [ ] optim: ultimate type on loose should yield `float`
- [ ] static: abstract imperative language code generation (wip)
- [ ] backend: do not generate intermediate empty functions for property values when possible
- [ ] backend: reduce/minimize language abstraction? `T_var_val`?
- [ ] backend: rethink function boundaries?
- [ ] backend: rethink `with_report` and `with_path`, keep the former only?
- [ ] c-backend: add support for `re2` wrapper?
- [ ] backend: name-based selector optimized for root.
- [ ] python: refactor in preproc tools…
- [ ] optim: remove unused introduced names `__external_...`
- [ ] optim: remove unused names at depth 1 and more?
- [ ] optim: remove unused local (not exported) definitions?
- [ ] static: detect and remove duplicate functions!
- [ ] static: generate pyright compatible code…
- [ ] tests: remove unstable DO output? remove empty lines?
- [ ] design: handle `.` and `/` references as relative url
- [ ] design: namespace, only some defs exported, and cannot have multi-level refs `$r#N#xxx`?
- [ ] python: use mandatory keyword parameters where appropriate
- [ ] feature: `--extension openapi` to activate an extension ? default `none`, `moschin`…
- [ ] design: add convention for local vs exported definitions? capital?
- [ ] static: separate compilation of included models
- [ ] jmc: add option to do/skip check of `~` model
- [ ] feature: control url caching, change dir location or disable
- [ ] feature: cache expiration?
- [ ] feature: improve error messages, eg missing "$" on a URL reference
- [ ] feature: restrict definitions as proper identifiers
- [ ] tests: re-check some results of json schema analysis paper
- [ ] feature: self trafo example, eg create a variant between local defs
- [ ] tests: measure coverage
- [ ] feature: simplify lib usage wrt script-like features with some functions
- [ ] feature: full inline of externals to generate a working model without includes
- [ ] feature: check/forbid/allow url references with an anchor
- [ ] static: remove spurious newlines
- [ ] tests: move/add more tests to rwt, modval, bads
- [ ] tests: what if a trafo adds/replaces an external reference, is it resolved?
- [ ] tests: fix tests in `bads/wip`?
- [ ] tests: add github CI
- [ ] tests: raise coverage, possibly to 100%?
- [ ] tests: skip url cache while testing
- [ ] tests: move Makefile tasks as `pytest` items with target content as reference
- [ ] output: improve `-P` to have a re-intrant model (see also: inline)
- [ ] output: prettyprint rejection reasons?
- [ ] feature: direct validator
- [ ] feature: static/dynamic js/ts compiler?
- [ ] feature: (limited) pydantic export?
- [ ] feature: integrate or remove `stats`?
- [ ] feature: think about code integration in a realistic project? usability? doc?
- [ ] design: (online) extension register?
- [ ] doc: check and document API entry points
- [ ] extend: direct forms for instance, eg types, constraints, layout? see [FormBuilder](https://formbuilder.online/).
- [ ] extend: direct relational mapping? (eg foreign key/primary key/unique)
- [ ] extend: direct object creation (Python pydantic, JS)?
- [ ] extend: ORM support, eg in the Python ecosystem, see `SQLAlchemy` and `Django`?
- [ ] extend: data structure _documentation_
- [ ] extend: accepting any extension?! eg opened objects?
- [ ] extend: variants, see `models/json-schema-*`
- [ ] extend: pg storage, extension
- [ ] extend: json schema for mongodb storage
- [ ] perf: schema validation on yac data
- [ ] perf: open data, eg RNCP and FRESQ
- [ ] frontend: loop load/valid for perf
- [ ] design: conditionals or not conditionals… eg for interfaces?
  if not in the description language, it should be easy to express outside.
- [ ] doc: create a clean documentation, following Diátaxis (tuto | howto / explain | reference).
- [ ] doc: setup mkdocs
- [ ] doc: tutorial which mimics [json schema](https://tour.json-schema.org/)?
- [ ] research: paper themes? extensions, compilation (CGO tools), ...?
- [ ] feature: use actual JSON path spec for transformations with a filter for applying changes
- [ ] pr: register `json-model.org`? use `github.io`?
- [ ] pr: article submission about data structure trafo? validation performance? high-level IR?
  perf is important because value validation is pure (useless) in-depth overhead in practice.
- [ ] perf: two phase check, without collection and with collection if needed.
- [ ] perf: no reasons/path collection and run with collection only when bad.
- [ ] schema: handle includes? inline?!
- [ ] go-backend: encoding/json vs various faster replacements…
- [ ] ruby-backend: for rails people?
- [ ] rust-backend: see [json crate](https://docs.rs/json/latest/json/).
- [ ] cpp-backend: see [json](https://github.com/nlohmann/json) and `re2`. json lib popularity?
- [ ] cpp-backend: also see [simdjson](https://github.com/simdjson/simdjson)?
- [ ] blaze-backend: use schema export plus simple cpp front or `jsonschema` cli?
- [ ] kotlin-backend: see [serialization](https://kotlinlang.org/docs/serialization.html).
- [ ] mongodb-backend: generate bson stuff, as an option to schema generation?
- [ ] ts-backend: js sharing ?
- [ ] swift-backend: …
- [ ] dart-backend: …
- [ ] php-backend: …
- [ ] c#-backend: …
- [ ] haskell-backend: …
- [ ] ocaml-backend: …
- [ ] scala-backend: …
- [ ] vb-backend: ?
- [ ] julia-backend: ?
- [ ] lua-backend: ?
- [ ] feat: add model predef for URI, URI-REF and the like
- [ ] design: URI vs URL vs IRI?
- [ ] design: documentation use case, with added _markdown description_
- [ ] pr: validation must be optimized for the valid case, as it is the most likely.
- [ ] case: use trafo in geo to have either 2 or 3 vectors, no mix.
- [ ] c-backend: consider [json-c](https://github.com/json-c/json-c)
- [ ] c-backend: consider [yyjson](https://github.com/ibireme/yyjson)
- [ ] c-backend: consider [cJSON](https://github.com/davegamble/cjson) (linked lists? no ints?…)
- [ ] ~c-backend: consider [Parson](https://github.com/kgabis/parson) (unmaintained?)~
- [ ] ~c-backend: consider [json.h](https://github.com/sheredom/json.h) (hmmm, linked lists)~