What's that and what's it for

This is a "long term corporate strategy wishful thinking BS" document gathering - with some explanation - the ideas from countless mailing list/IRC discussions and flame wars about LV2 extensions. It should be treated as a highly imperfect source of ideas of varying degree of importance, not as in "must-have". Very few of these are must-haves, most are more like interesting-ish pseudoresearch areas. Some are really poorly thought out. The polyphony framework idea is something that, as a whole, is less than 3 hours old, and may contain major flaws that make it impossible to use, or minor deficiencies that may limit its usefulness.

Also, most of this document may be hard to understand without concrete examples, code snippets and diagrams. I recall explaining the flexible port dependencies idea to a fellow software engineer a couple of years ago. I had to spend two hours of explaining, with help of multiple diagrams, in order to get him convinced that this idea is possible to implement, and that it is not in fact some crazy man's rambling. This document has no diagrams and it cannot answer the specific questions immediately, so understanding it may require large amounts of patience.

Motivation

Examples of plugins that may not be properly implemented in current LV2 (core+extensions):

  • oscillators with user waveforms - there is no way to pass waveforms to the plugin, or to save current waveforms in a song/project; if a plugin was created in several copies for polyphony, there is no way to indicate that data are supposed to be shared
  • big synths with user waveforms - see above, although possibly without data sharing problem
  • sample player - similar problems as with user waveforms (perhaps with file paths instead of waveform data)
  • sample recorder/looper - as above, but requires two-directional passing of large amounts of data
  • hard disk recorder - same, plus host-managed threads (not really necessary, but potentially useful for CPU management reasons), plus passing directories to store files in
  • sequencer - requires transport support
  • eye candy like in calfjackhost (live updated filter/envelope/osc graphs) - requires working GUI-plugin communication via message context
  • making keytrack curves in Organ compatible with LV2 (configure API is needed) - message context in GUI + save/load of non-port settings in a song
  • ingen ensemble compiler - would require the ability to put some annotated C++/Haskell/whatever code (or a script to generate annotated code)
  • LV2 port of meterbridge, oscilloscope, etc. (there is no autoOpenGUI kind of property to indicate that the plugin's #1 reason to exist is to display something on the GUI)
  • curve mapper for CC and velocity (at least) - again, GUI-plugin message context communication issues
  • polyphonic indicator - one that displays values of a port for each voice separately (because there is no notion of voices in LV2 yet)
  • mixers with aux sends/returns with zero latency - because it's not possible to call the fx chain's process functions before main outputs are supposed to be produced

This section will probably need to be grouped by functionality, or even moved to individual proposals as rationale.

Postulated extensions

Save/load

  • The ability to serialize and restore the plugin's non-port internal state.
  • Data might be in different form - ASCII, RDF tuples, XML, standard/portable binary (e.g. WAV, AIFF, SF2), proprietary/arch-dependent binary (e.g. binary state dumps, dynamic libraries)
  • Data might be dependent on project path (e.g. project index including references to streamed files)
  • For some data, standardized variables names (identified by URI) may be useful for interoperability, but it is not the first priority

Project path

  • For storing streamed files

Standardized messages

  • For writing error messages/warnings/status via host mechanisms
  • For sending simple context menu commands (like Panic!) which might be defined in plugin's RDF

Arbitrary messages

  • For GUI-plugin communication
  • Possibly URI-based variables, but what about type system for values?
  • Caveats: portability wrt. little/big endian, FP representation etc.; efficiency; optimized for common case (in-process plugin and GUI) but must be good enough to handle other cases

Data sharing between plugin and GUI

  • For static or seldom-changed large data (mostly waveforms)
  • Should be network-proof if possible (caching via host mechanisms)

Transport

  • For sequencers, recorders, loopers, time-synced plugins (LFOs, gappers, flangers, phasers)
  • Includes: song position information, song tempo and time signature information, pre-seek information (to allow hard disk recorder plugins to preload and cache sample data between seeking with playback stopped and starting playback); possibly key signature information (for intelligent arpeggiators)
  • BBT-based? sample-based? both
  • Bi-directional?
    • Definitely: sending commands like start/stop/pause/set position/expected position/set tempo/request current tempo? (for new plugins)
    • Possibly: plugins being transport masters
  • Tricky stuff: tempo maps, implementation difficulty (cooperation with JACK transport)
  • Possibly tempo maps and key signature/scale information in separate extensions
  • Suggestion: Use an event port to send an event containing transport information (frame position, BBT position, BPM, time signature, basically the same as JACK transport) to the plugin at the start of every cycle, possibly using type 0 events to save space. Use a non-RT callback to add and remove eventual loop points, to allow plugins to preload sample data and/or precompute transitions. This way it will be easy to cooperate with JACK transport, and we have the added advantage of glitch-free looping with loop points that don't have to sync with period boundaries. Proper tempo map can be done in a separate extension if it is needed, and a host or a plugin can slave transport events to that. Events can also be sent at any time, allowing time signatures and BPM changes in the middle of a run() period.

Port hints

  • Logarithmic scale. This is more for manually adjusted control ports with exactly defined min-max range. Also, min and max must (obviously) be of the same sign.
  • Possibly other scales (quadratic etc.)
  • most of "extended port properties" proposal - except they need to be rethought and rewritten in proper English
  • graphical port decorations, e.g. an arrow in edge-triggered ports (eye candy/usability improvement) like in electronic diagrams
  • The ability to designate a number of ports as "hidden", not available for editing/connecting, because of the fact that their sole purpose is to be used in implicit manner as some sort of IPC mechanism. Message context event ports might be a good example of such ports.

Port flags

If port value is changed, a bit should be set by host (input ports) or plugin (output ports); the same bit may be shared by different ports, bit number (or mask) might be specified as data in RDF. This is an optimization to avoid recomputing rarely-changed values frequently. Doesn't make much sense for audio or event ports, too.

Might make a lot of sense for audio ports as a "no data" flag, i.e. the bit is unset when the audio samples are all 0. This could allow lots of plugins to skip expensive computations.

Plugin hints

  • is a bridge (LADSPA-LV2, DSSI-LV2 etc.)
  • is a large plugin (like an instrument with built in effects or sequencer)
  • is a standalone plugin (like CAPS plugins)
  • is a module for modular environment like Ingen (like functions from larsl's collection)
  • is a monophonic instrument
  • is a polyphonic instrument
  • should auto-embed the GUI on load (plugins that don't make much sense otherwise)
  • do not run process() when outputs are not connected (or the opposite - run process() even when no outputs are connected) - an optimization
  • the criteria above are just an example, someone should put more thought into it

Debug level

A flag that can be passed to a plugin to set its error reporting level. Yes, this is a totally frivolous idea.

Polyphony

This extension would be used by modular hosts for host-managed polyphony. Main ideas are: # Bundling a set of (same) plugin instances in a "poly bundle" (an API-side counterpart of LV2Node) # Allow marking ports as belonging to either individual instances or a bundle (poly ports, mono ports). For example, in Ingen Midi Note plugin, an input MIDI port is a mono port, and the other ports (freq/vel/trig/gate) are poly ports. A Major Chord plugin (stupid as it sounds) may instead have poly freq ports but mono vel/trig/gate ports. # For setting instance-bundle relationships, the plugins implement (via extension_data) two callbacks: #* LV2_PolyBundle_Handle create_poly_bundle(LV2Handle handle); // to create a new bundle (for the first voice) #* void set_poly_bundle(LV2Handle handle, LV2_PolyBundle_Handle poly_bundle); // to make another instance join the existing bundle (or to unconnect the instance from the bundle if poly_bundle is NULL) # Apart from normal process() function of every instance, the bundle has an additional three optional process functions: pre_process_bundle_mono, pre_process_bundle_poly and post_process_bundle. The sequence of calls is: #* optional pre_process_bundle_mono(LV2_PolyBundle_Handle poly_bundle) of a bundle when all input mono ports have their values already calculated #* optional pre_process_bundle_poly(LV2_PolyBundle_Handle poly_bundle) of a bundle when all input poly ports have their values already calculated #* process of each individual instance (unless some special plugin hint says that the plugin doesn't need a process function) - may be called in parallel (host-side multithreading), but must be called between end of pre_process_bundle_poly and start of post_process. If pre_process_bundle_poly is not specified, then individual process functions may not rely on other instances' input poly ports being ready - lack of pre_process_bundle_poly is a way for the plugin to notify host that its individual instances will only rely on their own ports and mono ports, and will not peek into other instances' ports. #* optional post_process_bundle(LV2_PolyBundle_Handle poly_bundle) to calculate output mono ports after all voice-dependent stuff has been run (or to do some per-bundle internal housekeeping) Alternatively, a more flexible data-driven approach may be used (see: Flexible port dependencies extension). The poly bundles may be also used by plugins to keep data that are shared between different poly instances (like waveforms in a sampler).

Flexible port dependencies

Some plugins (like mixers with aux send and aux return) may need to use some external processing to take place in order to be able to produce useful output. The processing in such a plugin can be divided in (at least) three stages:

  • Processing of some inputs (individual inputs) and generating some outputs (aux sends) and some internal state (internal mixing buses)
  • External processing of aux sends to produce aux returns
  • Processing of remaining inputs (aux returns) and internal state (internal mixing buses) to produce remaining outputs (main outputs)

Additionally, some of inputs or outputs may be monophonic while others may be polyphonic (see: polyphony extension ideas). The approach to use is to group dependent inputs and outputs in RDF, and to have a separate process function for each group (plus a "null" process function which is called when no output needs to be produced, but some housekeeping needs to take place anyway). For example, in a mixer:

  • group 1 - mixer inputs (in) and aux sends (out)
  • group 2 - aux returns (in) and master outputs (out), also dependent on group 1 (process_group for that group cannot be called before process_group for group 1)

Another example would be inertia with pluggable rampers (linear, exponential, quadratic, PWM, random). Or envelopes with pluggable stage shapes.

In some sense, it is a way for plugins to use other plugins without actually having to host them. It may not solve all the cases, but might be better than nothing.

There is one highly complicated, but potentially interesting, modification of this idea. It is a combination of flexible port dependencies extension, host-managed polyphony extension and additional one ("joint plugin-host polyphony management extension"). A polyphonic integrated instrument plugin may expose a set of poly aux sends and aux returns to externally process each voice separately. To achieve that, it must request the host to create a poly bundle of each plugin in the aux loop with required number of instances (one per voice).

(I think the explanation above is a bit unclear without a concrete example or a drawing)

Backward compatibility

The correction introduced in LV2 core version 3 solves the backward compatibility problem in the most common situation: new ports being added. However, there are other possible scenarios:

  • Ports being deleted (due to non-use and/or cost of implementation)
  • Ports being replaced by other ports with similar functionality (e.g. instead of sine and cosine outputs in a sine oscillator, a sine plus arbitrary phase shift outputs are provided, with extra input ports for phase shift amount)
  • The new port values may also need to be calculated (which may add unnecessary complexity)
  • New plugins being used as functionally equivalent replacements of other plugins (backward-compatible LV2 versions of LADSPA plugins is an obvious example)
  • Complex plugins being phased out due to inflexibility and replaced by graphs made of simple plugins. In some cases, the graph is always the same and may be expressed in RDF. In other cases, a script may be needed to generate the required tuples based on which ports are connected or initial values of the ports (if those ports may be assumed to be constant - like waveform or filter selection). Example: old saw/sqr osc may be replaced by saw or sqr osc depending on which waveform was active in the original song.

Microtuning

If whole songs were expected to be using the same microtuning or set of microtunings, it might be worthwhile to allow tuning schemes to be available as shared resources, which can be discovered, selected and employed by plugins that require tuning information.

A very simple tuning extension might involve a host-provided callback function to translate a MIDI note to note frequency. A more advanced tuning extension may add the ability for the host to expose more than one tuning.

Additional extension may make it possible for plugins to define the new tunings (either by the means of function/data pointers or as static RDF). Those new tunings would then be provided by the host using the two mechanisms mentioned in the previous paragraph.

Number to string and string to number

  • Traditional %e/%f/%g number formatting methods may produce unnecessary long strings due to rounding errors. Some hosting environments may require optimal use of available screen space (when using a small character-based LCD screen, or when trying to display all control port values in number form in a modular host like Ingen). For that, a printf-style format string hint for a port might be adequate.
  • Some numeric port values may have special meanings (example - mixed enum/float fields). For this set of uses, a plugin might provide a number-to-string function for those ports, or a port-value-to-string function, which could also print non-numeric data like plugin-specific events or special port types. This might be used for diagnostics as well.
  • For text-based entry, an opposite conversion function (string to port value) might exist.

Other

In no specific order:

  • contexts (msg context is implemented in ingen, the rest isn't)
  • port bundles (adding bunches of ports per layer etc) - larsl's group extension is a good start (and maybe end)
  • dynamic ports (adding/removing ports on the fly), especially in context of zynaddsubfx-type layering or multi-strip plugins (mixers etc) - should be somewhat tied to port bundles in some sense (but the individual strips can be heterogenous, ie. mixer strips with EQ and/or aux outs, zynadd/zynsub/zynpad layers etc.)
  • multiconnect/array ports (ie. one port acting as multiple separate ports - each connection goes to different port "instance") - allows for 1-input multi-operand operator plugins - each connection actually creates a different subport, and they're passed as an array of port buffers
    • somewhat kludgy, but better than "2-in mul" "3-in mul" "4-in mul" or doing some insane cascades of mul plugins
  • polymorphic ports (audio/control depending on what's connected) - I'm still unsure if it's necessary, it's nice and generic but might nicely solve the wrong problem while adding unnecessary complexity when trying to use it to solve the right problem (which is the lack of sample-accurate control information)
  • sample-accurate control information - there are several proposed solutions, all somewhat work, none is good enough(?)
  • maybe not a LV2 issue: stopping a layer/voice when it's run to its end
  • C++ code generation
  • Ingen Engine and UI extensions - for automation or interfacing with hardware - beware of API-related paralysis
  • resource and service discovery (samples/waveforms, LFO shapes, envelopes, keytrack/veltrack curves, key/vel mappings etc.), resource-exposing plugins
    • example purpose: global wave table in Buzz (the WAV files that are loaded by Buzz itself and accessible by all plugins as a shared resource)
  • format string or display function for values
  • better control rate data transfer
  • "main" port/group property (main input, main output, main event input etc.)
  • "idle" audio periods like in Buzz (ie. skip calculation of voices/subgraphs that are muted or inactive anyway) - the same functionality may also be available in GStreamer