Extensibility

[bastian]

Hello, Cadence Community!

We would like to add support for extending existing types with additional data and functionality, retro-actively, i.e. without modifying the original declaration.

This was requested in the following GitHub issue: https://github.com/onflow/cadence/issues/357.

The goal of this topic is to work out a pitch and eventually concrete proposal for how to achieve this in Cadence.This could be for example in the form of a Flow Improvement Proposal (template), and/or a Cadence RFC.

We can use this topic to collect e.g. prior art, use-cases, answer open questions, and work together on the design proposal.

Once we have a proposal, we can move on to work on the implementation of it (potentially starting with an MVP/subset).

I’ll update the remainder with the latest state of the proposal.

---

Summary

This proposal describes a new language feature: Support for extending existing types with additional data and functionality, retro-actively, i.e. without modifying the original declaration. This feature is purely additive, i.e. no existing functionality is changed or removed.

Motivation

It is currently not possible to extend existing types unless the original author did explicitly make provisions for future extensions.

For example, to make a resource declaration extensible, its author may add a field that allows any other code to store an extension. However, this requires a lot of boilerplate and is brittle. The original type must be prepared to store additional data with potentially additional functionality.

Use cases

• Adding hats, apparel and accessories to CryptoKitties
• Storing e.g. a signature or edition information for a given NFT, e.g. adding autographs to NBA TopShot moments
• Paying the original creator of an NFT a royalty on resale
• Storing ownership trail information (a NFT owned by somebody might be more valuable? Lets say I own a TopShot moment of Zion that he has owned himself?)
• Add utility functionality, e.g. export a HTML representation of an NFT

Prior art

• Extension methods in C#
• Extension methods in Scala/Dotty
• Extensions in Kotlin
• Extensions in Swift
• EIP-998: ERC-998 Composable Non-Fungible Token Standard

Design

This proposal specifies the ability to extend resources with additional functions. Other extensions are out-of-scope and could be proposed in the future. See the “Future / Out-of-scope” section below.

Declaration

Extension function declarations are able to refer to self, just like original function declarations. This makes them natural to declare and to read.

Extension function declarations have the same access to fields and functions as code outside the original type declaration, i.e. they should be able perform everything that is possible externally, but also not more. This ensures that an extension cannot exploit the extended type by extending original functionality. As a result, an extension function declaration is not able to access private fields or function of the extended type.

Extension function declarations are not able to override existing function declarations. For example, if a resource declares a function named use, extensions may not declare a function named use, even if it has a different signature (different parameters and/or different return type).

Syntax

TBD

Use

The owner of a resource may freely add and remove extensions.

Extensions have an order, the order in which the extensions where added and removed from the extended object. This allows distinguish usage patterns. For example, an art piece might be first framed, and then signed; or it could be first signed and then framed.

TBD

Syntax

TBD

Drawbacks

Adding a new language feature has the downside of complexity: users have to learn yet another concept, and it also complicates the language implementation.

However, this language feature can be disclosed progressively, users can discover and use it when needed, it is not necessary to be understood for core use-cases of the language, i.e. the target audience is mostly “power-users”.

Future / Out-of-scope

Kinds

This proposal only specifies the extension of resources with additional functions.

Other extensions are out-of-scope and could be proposed in the future:

• Extension of other kinds of types, such as structs, contracts, events
• Extension of interfaces
• Providing default implementations of interfaces
• Extension of types to conform to more interfaces

Unremovable extensions

Some extension should not be removable. A future proposal may restrict certain extensions to be removed from an extended object.

Composability

Some code authors may declare up-front that the type they declare should always have an extension, i.e. they want to re-use code that has been written as an extension.
A future proposal may define how this could be achieved.

Open questions

Declaration

• Extension functions:

  • Can extensions be referred to, i.e. do they have a name?

  • Can and if so how are nested types extended, e.g. a resource declared in a contract?

  • What is the syntax for extension function declarations?

  • How are declaration conflicts handled? For example, if two accounts both declare a function with the same name, is it possible to import both, and if so how?

• Should it be possible to declare additional fields? If so

  • How are the extension’s fields initialized, and if they are resource-typed, how are they destroyed?

    • Are extensions able to extend the destructor of resources with arbitrary code?
      • This could prevent resource owners from destroying the extended resource

  • How and where are the values in the fields stored?

Usage

• How can extension be used and what is the syntax for it?

  • Approaches:

    • Import
    • Mixin (like Scala):
      • e.g. R with 0x1.R with 0x2.R
      • order important
      • super calls, in order
      • how can conflicts be handled? see below

  • How are declaration conflicts handled? For example, if two accounts both declare a function with the same name, how can both be used?

    • Approaches:
      • Renaming in import statements? In Kotlin (?) and in C#
      • Mixins: How?

  • How are extensions resolved/dispatched? (See “How do extensions interact with interfaces?”). Statically (like in Kotlin and Swift), or dynamically?

  • Is the extension effective implicitly or explicitly,
    i.e. does the developer need to explicitly specify that an extension is effective for an object,
    or is the extension implicitly effective after an import of code that declares an extension?

• If extensions can declare additional fields, how are they initialized, i.e if the extension would declare an additional initializer for those extension’s fields, how is the initializer called?

• What is the subtyping relation for extension types? When type T is extended with U and then V, is it a subtype of a type T + V + U? (note the difference in order)

Other

• How do extensions interact with interfaces? (See “How are extensions resolved/dispatched?”)

• Can this proposal be added in a backwards-compatible way? Does existing code or stored data need to be migrated?

[bjartek]

https://github.com/bjartek/flow-nft-mixin/tree/trait is a little something I made.

It does not answer all of the questions above but It solves some of them.

[woody]

I would like to be able to create an interface extension to provide default function implementations, the way one can do with Swift protocol extensions.

[bastian]

I’ve updated the proposal and the open questions with the notes from out last call.

I’d be happy to suggest answers to some of the questions, but am also curious to hear what others think, and can also wait until the next call and we can discuss them together.

[bjartek]

The extensions above in your examples are AnyStruct, while in my mixin proposal they are AnyResource. What it the practiacal difference here? Can an struct contain another Resource or Capability?

[bjartek]

Use cases I can see extensions beeing used for right now:

• paying the original creator of an NFT a royalty on resale.
• storing a signature for a given NFT (could be combined with the above)
• storing ownership trail information (a NFT owned by somebody might be more valuable? Lets say I own a TopShot moment of Zion that he has owned himself?)
• storing edition information for an NFT
• formalize how to fetch onChain stored content. Might be possible to chunk up content into base64 batches and store a capability to the chunks and the indexes to fetch?
• export a HTML representation of an NFT

These are just some ideas from the top of my head.

[bastian]

With the example code at the top of the proposal I just meant to show how one could add extensions right now and demonstrate the issues that has. It is not meant to show the proposal’s outcome, and more specifically isn’t supposed to imply that extensions themselves are structures.

Sorry for the confusion, I might remove it.

As for containment of resources in Cadence in general, only resources can contain resources. Nothing would change about that with the proposal.

[bastian]

@bjartek

• paying the original creator of an NFT a royalty on resale.
• storing a signature for a given NFT (could be combined with the above)
• storing ownership trail information (a NFT owned by somebody might be more valuable? Lets say I own a TopShot moment of Zion that he has owned himself?)
• storing edition information for an NFT
• formalize how to fetch onChain stored content. Might be possible to chunk up content into base64 batches and store a capability to the chunks and the indexes to fetch?
• export a HTML representation of an NFT

Great use-cases, I’m adding them!

[bastian]

Some more thoughts / suggestions for answers to the open questions regarding declarations:

Can extensions be referred to, i.e. do they have a name?

I think it would be useful to name extensions to have a way to identify / refer to a specific extension, especially when importing/using the extension. This would also allow the declaration of multiple extension in the same location and allow a user to selectively import them. For example (with straw man syntax):

extension Autograph for CoolNFT {
    fun sign(/* ... */) { /* ... */ }
}

extension Ownership for CoolNFT {
    fun addOwner(/* ... */) { /* ... */ }
}

In some other languages the extension functions are self-standing and the individual extension functions can be imported individually. Imports can then often rename the functions, which enables the use of multiple extension functions that have the same name.

Instead, I think we can handle conflicts by making the usage explicit at the call-site.

Can and if so how are nested types extended, e.g. a resource declared in a contract?

I think this would be required, as currently it is not possible to declare top-level resources, they must be contained in contracts at the moment.

We could allow dotted identifier syntax to refer to a child declaration. For example, given a resource declaration R in contract C:

contract C {
    resource R {}
}

extension E for C.R {

How are declaration conflicts handled? For example, if two accounts both declare a function with the same name, is it possible to import both, and if so how?

This could be solved by naming extensions, see above.

Should it be possible to declare additional fields?

Yes, I think it would be very useful to store additional data.

I can’t find any precedence for this in currently popular general-purpose programming languages, but there should not be any reason to not allow this, except for this proposal also figuring out a way to handle field initialization, and if the field has a resource-type, destruction.

[bastian]

Notes from the last call:

• It would be nice to extend interfaces:

  • e.g. NFT interface with inventory item or autograph

  • 
    resource interface NFT {}
    
    resource Moment: NFT {}
    
    fun use(_ nft: @{NFT}) {
        if let moment <- nft as? Moment {
            // use moment, which is @Moment
        }
    }
    
    resource interface extension Autograph: NFT {
        fun sign(/* ... */) { /* ... */ }
    }
    
    
    fun sign(_ nft: @{NFT}) {
        if let nftWithAuto <- nft as? Autograph {
            // use auto, which is @{NFT} + Autohgraph
            nftWithAuto.sign()
        }
    }
    

  • We had already previously planned to support interface requirements in Cadence:
    An interface may require the conforming type to also conform to other interfaces.
    For example, the SignableNFT can require a concrete type to provide a sign function,
    but also for it to conform to the NFT interface.

    resource interface NFT {}
    
    resource interface SignableNFT: NFT {
        fun sign(/* ... */) { /* ... */ }
    }
    
    resource Moment: SignableNFT {}
    

  • Can we combine the two ideas? If not, this could lead to a lot of code duplication

• We want to promote and drive adoption of standard interfaces (conventions), to avoid fragmentation

  • NPM-style: many small, composable interfaces
  • encourage reuse, “get to the fun stuff faster”

• Idea for declaration syntax:

  • resource extension Autograph: NFT { ... }
    

    • Explicitly state kind again (resource)
    • Reuse existing & familiar : syntax

• It would be nice to provide a default implementation for interfaces:

  • Declaration:

    resource R {}
    
    resource interface Collection {
        fun add()
    }
    
    
    resource extension Collection {
        fun add() { ... }
    }  
    

  • Two options to use the default implementation:

    • Extension, after declaration:

      resource extension R: Collection {}
      

    • At declaration time:

      resource R: Collection {}
      

• Ownership: Who owns / stores the extension’s additional data?

  • Should it be possible to get a public account, get a capability to a resource in that account, and add an extension?

    • What about bad actors?
    • Should this be controlled through an explicit permission, e.g. through an allow list?

  • Should extension control destroy-ability (by aborting in destructor)?

    • What about extensions which should not be able to be deleted? e.g debt, debuffs/effect with drawback shouldn’t be destroyable

• Access in extension on extended type:

  • Example: Moments contract, later we add a few fields through an extension
  • Should it be possible to have “private extensions”, extensions that have access to private fields of extended type?
    • At odds with immutability ideal, impacts integrity, users might loose faith in locked down smart contract
      => don’t adopt, only public access

• Introspectability/reflection:

  • It should be possible to introspect a value and determine what extensions it has.
  • Example: buy character on marketplace, see it has magic sword
  • API:
    • list extensions
      • extension type
      • storage size

  => required for use case:

  • buy item on marketplace
  • wallet software needs to ensure item can be stored in account, needs query to increase storage deposit

• Feature request for Cadence: object size introspection

[bastian]

Some topics related to extensibility came up in our calls, and I think it’s a good idea to quickly explain them. That way we can draw a clear boundary what we want to specify and discuss in this proposal and what is not part of it:

This proposal is regarding extensibility: extending existing types with additional data and functionality, retroactively, i.e. without modifying the original declaration.

What also came up in our discussions is composability: re-using existing code, proactivly (as opposed to retroactively in extensibility).
This is very much something we also want to pursue in Cadence, but I think it’s a good idea to discuss and specify this in a separate proposal.

In addition, I think we should strive to keep the first proposal small, so that we can implement it more quickly and gather feedback on it more quickly.
Though the feature ideas we have discussed so far are great, and we shouldn’t just ignore them, so I suggest extending the “Future / Out-of-scope” section with those items, namely:

• Composability
• Extensions of interfaces
• Default implementation for interfaces

[bjartek]

So should we limit the MVP of this to something that

• cannot use private state of NFT you extend
• state for extensions are stored with the NFT
• you should need access to the resource in order to extend it, not just the reference to it.
  • so in order to sign it with moment they would need to transfer their NFT to moment, they sign it and then they transfer it back again.

The way I see it the following needs to be solved pretty early for NFTs on flow

• identifiers, are they going to be autoincrement and seperate for each kind of NFT?
  • should an identifier change if you store it in another collection?
• extensability (this proposal): the ability to add new functionality to and NFT retroactively and discover that new functionality.
• composability: the ability to reuse code across NFTS, so you do not have to write your own Borrow method or withdraw method
  • handling super() correctly here should also be considered.
  • naming collisions in interfaces
• standard NFT fields, in adition to the identifier mentioned above
  • name, description, content all as String would be enough to solve MANY scenarios.

[bjartek]

• Access of fields, should only be allowed to view visible fields for extensions. public functions call them

• some extensions like signature should not be removable

• other extensions like kittyhat should be removeable. (use attach verb or say removeable extention)?

• only owner of resource should be allowed to remove attachements/removeables

• only owner of resource should be able to attach or extend an NFT

• what happends when you remove an attachment, you will get the resource back

• what happends when you destroy a resource that has an attachment/extension

• you should not be able to destroy an extension without destroying its owner

• should extensions be resource or be allowed to contain resources?

• should attachments be resource or be allowed to contain resources?

• should these capabilities be on NFT, Resource, or use a Virtual Resource and merge them as needed.

• the order of which you extend something matters. A kitty is signed and then framed, vs a kttiy is framed and then signed.

[bastian]

Notes from our call on 2020-11-10:

• Last session

  • “protected” access modifier
  • unremovable modifier
  • permissions
  • attachments vs extensions
  • attach -> detach/remove
    • maybe return attachments as part of destroy statement?
  • ordering example: framed and signed, vs signed and framed
    • method dispatch / overriding?
    • subtyping?
  • extensibility and composability at the same time, reduce boilerplate
  • extended value should be able to used as “unextended” value

• Maks:

  • data is stored externally (e.g. dictionary mapping resource to extension’s data

    • one-to-many possible

    • however, storage needs to be paid for, centralized, extension author would have to pay for users’ data

    • could be “business model”: use of extension requires payment by user

    • when to choose: extension’s data might be large, e.g. TopShot data, art BLOB, etc.

    • centralized storage is a choking point

    • can be already implemented on extension where data is stored in extended object: store pointer (capability) + ID

• start with extensions

• support for collections? e.g. multiple signatures

  • built-in support? or does author need to account for that?

• ownership tracking / royalty extension:

  • how to reverse callback?

[bastian]

Thank you again everyone for joining the last call and discussing the proposal. I have updated the proposal above as far as I could.

So far the calls have been very much in the form of brainstorming potential features and use-cases, so now it would be great to focus more on a concrete proposal, ideally an MVP that we can start to implement.
However, there are still quite a few open questions that we need to answer before we can start implementation.

It would be great if we could all look at the outstanding questions and try to answer them.

For example, so far the proposal only defines that extension can add additional functions to extended objects, but we also discussed how we want to allow extensions to declare additional fields.

As I listed above, how are the extension’s fields initialized, and if they are resource-typed, how are they destroyed? Are extensions able to extend the destructor of resources with arbitrary code? This could prevent resource owners from destroying the extended resource

I’m happy to try an propose concrete answers to the open questions / solutions to the open problems, and then we can discuss and refine them to ensure they allow the use-cases we hope to cover.

[bastian]

Notes from the last call:

• It would be nice to also have support for extending existing types with additional normal structs/resources/etc.,
  i.e. ones that don’t interact with the extended type:

  resource R {}
  
  resource interface HasHat {
  
    let hat: @{NFT}
  
    fun addHat()
  
    fun removeHat()
  }
  
  resource OneHat: HasHat {
  
     let hat: @{NFT}
  
     init(hat: @{NFT}) {
         self.hat <- hat
     }
  
     destroy() {
         destroy self.hat
     }
  
     fun addHat() { ...}
  
     fun removeHat() { ... }
  }
  
  fun test() {
  
      let r: @R <- create R()
  
      // either anonymously:
      let rWithHat: @R with HasHat <- extend r with HasHat {
  
        let hat: @{NFT}
  
          init(hat: @{NFT}) {
              self.hat <- hat
          }
  
          destroy() {
              destroy self.hat
          }
  
          fun addHat() { ...}
  
          fun removeHat() { ... }
      }
  
      // rWithHat should still be an @R
  
  
  
      // or pre-declared:
  
      // @R with OneHat / @R with HasHat
      let rWithHat: @R with HasHat <- extend r with OneHat(...)
  
      // invalid:
      extend rWithHat with TwoHats(...)
    }
  

• Using the extension statically:

  fun staticallyUse(_ r: @R) {
      // cast to statically known type
      let fooWithOneHat <- foo as? @R with OneHat
      fooWithOneHat.addHat(...)
  }
  

• It should be possible to test if an extended type has a run-time type:

  contract ChecksSpecialTypes {
  
      let specialTypes: [Type]
  
      resource R {
  
          fun isSpecial(): Bool {
              let specialTypes: [Type] = ChecksSpecialTypes.specialTypes
              for specialType in specialTypes {
                  if r.isInstance(specialType) {
                      return true
                  }
              }
              return false
          }
      }
  }
  

• Related: It should be possible to check if a run-time type is a subtype of another run-time type:

  let otherType: Type = ...
  someType.IsSubType(otherType)
  

  See https://github.com/onflow/cadence/issues/473

• Related: It should be possible to get the run-time type of a value.
  See https://github.com/onflow/cadence/issues/195

• It should be possible to get the types of all extensions of a value,
  e.g. through a function on all types:

  fun Any.getExtensionTypes(): [Type]
  

• Adding an extension:

  
  resource R2: HasHat {
  
      let hat: @{NFT}
  
      init(hat: @{NFT}) {
          self.hat <- hat
      }
  
      destroy() {
          destroy self.hat
      }
  }
  
  fun test() {
  
      let r <- create R2()
      r.addHat(...)
  }
  

• Handling conflicts:

  fun conflictingUses() {
  
      let foo: @R <- ...
  
      let foo2 <- foo as? @R with SomeExtensionThatHasAdd
      foo2.add(...)
  
      // ...
  
      let foo3 <- foo2 as? @R with AnotherExtensionThatHasAdd
      foo3.add(...)
  
  }
  

• It should be possible to iterate over all extensions of a value:

  resource interface HasName {
  
      fun getName(): String
  }
  
  resource E1: HasName {}
  
  resource E2: HasName {}
  
  resource R {}
  
  fun iterateOverExtensions() {
      let r <- create R()
  
      let rsWithHasName: [@R with HasName] = r.getExtensions<HasName>()
      for r in rs {
          let name = r.getName()
          // ...
      }
  }
  

• Extensions can also have extensions

• Removing an extension:

  let r: @R <- create R()
  let oneHat: @OneHat? <- r.popExtension<OneHat>()
  

• How can removal be prevented?

  • nonremovable resource?
  • Maybe callbacks for pushing/adding as extension / popping/removal as extension?

  interface Extension {
      fun onAddedAsExtension(extendedValue: &T)
  
      fun onRemovedAsExtension()
  }
  
  resource Signature: Extension {
  
      init(signedIdentifier: String) {
          self.signedIdentifier = signedIdentifier
      }
  
      fun onAddedAsExtension(extendedValue: &T) {
          pre {
              extendedValue.identifier == self.signedIdentifier: 
              "cannot attach this signature to the wrong signed value"
          }
      }
  }
  

[bastian]

Notes from our call on 2012-11-27:

• Can extensions be referred to, i.e. do they have a name?

  • So far, yes

• Can and if so how are nested types extended, e.g. a resource declared in a contract?

  contract C {
  
     resource R {}
  }
  
  - `resource extension E: C.R {}`?
  
  

• How are the extension’s fields initialized, and if they are resource-typed, how are they destroyed?

  resource NFT {
      pub let id: String
  }
  
  resource extension Signature: NFT {
  
      init() {
          let id = self.id
      }
  
      fun sign() {
          let id = self.id
      }
  }
  

  • Create and extend:

    let r <- create R()
    let e <- create E(...)
    let rWithE: @R with E <- extend r with e
    
    // r is moved/consumed
    // e is moved/consumed
    // only rWithE is now available
    

    • How does this work for extensions that interact with the extended type?

      let nft <- create NFT()
      let nftWithSignature <- extend r with Signature()
      

      Add sugar:

      let nftWithSignature <- create NFT() with Signature()
      

• Syntax for type with extension?

  • Ideas:

    • T with E1?
    • T with E1 and E2? ++
    • T with E1, E2
    • T with E1 with E2
    • T + E1 in Scala? + is/looks like an operator
    • T & E1 in Swift? & already used by references

  • What about nesting?

    • T with (U1 with V1) and U2
    • T with U1 with V1 and U2
    • T with U1 and V1 and U2 => 3 extensions on T: U1, V1, U2
    • T with U1 with V1 and U2 and W1
      • T with (U1 with V1 and U2 and W1): T with 1 extension: U1 with 3 extensions
      • T with (U1 with V1 and U2) and W1: T with 2 extension: U1 with 2 extensions
      • T with (U1 with V1) and U2 and W1: T with 3 extension: U1 with 1 extension, U2, W1
        => not obvious

  • We could forbid nested types, enforce use of type aliases

  • Do we even need the need for multiple extensions? We can use them one-by-one

• Does combination of multiple extension?

  • Privilege escalation through combination

• Another idea for disambiguation:

  let fooWithSomeAndAnother <- foo as? @R with SomeExtensionThatHasAdd, AnotherExtensionThatHasAdd
  fooWithSomeAndAnother.SomeExtensionThatHasAdd::add(...)
  fooWithSomeAndAnother.AnotherExtensionThatHasAdd::add(...)
  

• Syntax for adding extension? extend keyword?

  • extend r with e
  • r.extend(e)
  • leave out extend keyword: r with e? more consistent with two cases above:
  • with as postfix operator, just like as, as?, etc.
  • mixin?
    => just have with, descriptive, but vague enough

[blurden]

Really excited about the work on this part of flow.
One note, “trail of ownership” is known as “provenance” in the art world.
Question - with the notion of dynamic artwork… a landscape whose time of day matches your location and thus the daylight changes with it. the idea that a work of art needs to be fed, else it looses say the saturation in it’s colors… would these fall into extensibility or composability?

[sainati]

Looking to start implementing this; is there somewhere I can view the alternative solutions that were used in the uses cases like CryptoKitties accessories or additional info on TopShot moments? It would be useful when designing a solution to make sure that the solution would solve the original use cases.

[bastian]

@sainati

is there somewhere I can view the alternative solutions that were used in the uses cases like CryptoKitties accessories or additional info on TopShot moments?

CryptoKitties / KittyHats was one of the first examples of extensibility on Ethereum. You can see the implementation, the Solidity contract, here: The accessory token is attached to a kitty by storing the kitty ID in the token, i.e. it is a bottom-up approach.

This is a really good blog post explaining the bottom-up and top-down approaches (it is even using KittyHats as an example): Top-Down and Bottom-Up Composables, What’s the Difference and Which One Should You Use? | by Nick Mudge | HackerNoon.com | Medium.

ERC-998: Composable Non-Fungible Token specifies both approaches to support backwards-compatibility, i.e. existing contracts (which are immutable on Ethereum).

If we add this proposed extensibility proposal to Cadence, existing contracts would automatically gain support without any changes required by contract authors, which would obsolete the bottom-up approach, as the top-down approach could then always be used.