In the post titled “FLIP 73 - Proposal to Add Event Streaming to Access API” a new functionality for streaming was introduced. The initial implementation included the first two endpoints for polling events through the gRPC API and REST API with basic event filters, as well as block execution data via the gRPC API, all of which were based on the document.
Since the streaming framework has already been built with basic functionality, the logical next step is to expand it by adding more useful endpoints and enhancing existing features. This proposal aims to gather suggestions and ideas for future functionalities and improvements. The outcome of this publication will lead to a new FLIP document proposal, where all ideas will be collected, described, and refined through further in-depth discussions.
To kick off this conversation, we would like to propose the addition of a couple of new subscriptions some of which have already been mentioned in discussions related to FLIP 73:
Streaming of blocks starting from the provided block ID or height. The result for each block should include Header, Payload, and BlockStatus information.
Streaming of block headers starting from the provided block ID or height. The result for each block header should include Header and BlockStatus information.
Streaming of transaction results starting from the provided block ID or height, and /or collection ID and/or transaction ID. The result should consist of an array of TransactionResult.
There were a couple more options proposed in FLIP 73 comment, that could be potential candidates for future implementations. Except for what was proposed above, there are:
Streaming of a transaction and status changes
Streaming of an account state changes
Streaming of resource movement by type
Streaming of resource changes
Additional proposals are welcome, as the primary objective is to develop essential functionality that the community prioritizes.
One more point, as mentioned earlier, the initial version of the subscription framework with a limited API has already been implemented. It would be valuable to gather feedback from community users who have tested it: Is it user-friendly? Is it intuitive and clear? Do you have any suggestions for improvements? Please share your thoughts and ideas with us, so we can address relevant points during the expansion phase.
Streaming of blocks starting from the provided block ID or height. The result for each block should include Header, Payload, and BlockStatus information.
I think this would be little too heavy with network traffic. Will this be duplicated for each BlockStatus as block changes status ? I think putting this in execution state as header is easier option.
Streaming of block headers starting from the provided block ID or height. The result for each block header should include Header and BlockStatus information.
I think this can also be added to execution state streaming. ( block header is currently missing ) But as headers are light, I think would be nice to have.
Streaming of transaction results starting from the provided block ID or height, and /or collection ID and/or transaction ID. The result should consist of an array of TransactionResult.
This is nice to have with optional transactionID filter.
Streaming of a transaction and status changes
subscribing to a tx for status change ( can be covered by proposed APIs with filter ) FCL can utilize this for tx status instead of polling, there were a lot of problems on this because of load balancer and polling.
Streaming of an account state changes
this is very useful for wallets if filtered by account address, so they can immediately query account and show changes. ( also FCL can use this, if we provide accountResponse, for updated key sequence numbers etc ) Also this is very good for exchanges etc monitoring deposits.
Streaming of resource movement by type
Streaming of resource changes
I suggested those but those are not that important
Streaming of blocks with a tx count and a flag if it has service events would be lovely. Atm i poll for this in flowdiver. And having a stream of this data would be very helpfull.
Will this be duplicated for each BlockStatus as block changes status ?
I don’t think it should send multiple responses for a block. Maybe the user could subscribe to either finalized or sealed blocks. If subscribing to a past block for finalized, the first blocks would be sealed until it caught up with the chain.
I think putting this in execution state as header is easier option.
This would require the user to download all of the execution data along with the header. That seems like overkill for the usecase of following blocks. Are you interested in having headers included in the execution data response? That’s possible.
Streaming of blocks with a tx count and a flag if it has service events would be lovely. Atm i poll for this in flowdiver. And having a stream of this data would be very helpfull.
Would you want this for full blocks, or just headers?
FWIW, the number of tx would be possible for finalized or sealed blocks. Whether or not there’s a service event would require the block be executed, so may or may not be available when finalized.
Streaming of blocks starting from the provided block ID or height. The result for each block should include Header, Payload, and BlockStatus information.
The use case here is a client that wants to follow the chain, but can’t use the consensus follower. They may want to verify the block hash and fetch additional data to validate the fields
Streaming of block headers starting from the provided block ID or height. The result for each block header should include Header and BlockStatus information.
The use case for 2 is a client that wants to stay updated on the latest block
For 2&3, in addition to the start block, I think both should accept a flag that specifies whether to stream finalized or sealed blocks/headers.
It may make sense to have a lightweight version that only returns the blockID, height, and maybe timestamp. I suspect a significant number of clients are only interested in these.
Streaming of transaction results starting from the provided block ID or height, and /or collection ID and/or transaction ID. The result should consist of an array of TransactionResult.
This sounds like effectively a streaming version of GetTransactionResultsByBlockID.
I can see some value in searching for transaction ID, where the behavior could be to return the result when the tx is executed/sealed. We’d have to explore this a bit more. I think there may be some open questions for this (e.g. what block is passed in to start? when does it stop searching? what happens if the tx is never finalized?). Much of that would be simplified in an endpoint like the next one.
Streaming of a transaction and status changes
This would be great. Something like a SendAndSubscribe endpoint where you submit a tx, and the status is streamed back until the block the tx was included in is sealed.
Streaming of an account state changes
Streaming of resource movement by type
Streaming of resource changes
these would be awesome. I think there’s some R&D work needed before it’s possible but definitely valuable. Especially the account state changes as bluesign noted.
@petera I was thinking putting on all execution data as a big package, with blocks etc. And user just specify which parts they are interested. For example; if I dont want state changes, I dont get them. Someone doesnt care about events, they dont get events. But essentially data would be single service.
Like blockheader vs block service example; it can be one service, block streaming service with option ‘headers only’
Somehow I think configuring one big data item is easier than carving out many services from same data.
In my opinion, it’s better, cleaner, and in some cases faster, as well as more future-oriented, to opt for a modular approach and have several smaller endpoints. If we rely on a single large endpoint encompassing everything with configurations, it may grow in complexity as per user needs. Eventually, it could become a ‘large beast’ with various filters and configurations. Having multiple smaller endpoints would be easier to implement and maintain. They’d also offer easier expansion - just add a new one instead of modifying existing logic and potentially causing regressions. However, this viewpoint is subjective.
My only concern is there is to manage on client side many subscriptions, if we use duplex streaming ( not sure if it is possible technically ) user can keep a single connection, and subscribe/unsubscribe stuff over it. Otherwise if we put connection limits per IP for example later, this can introduce a problem as we extend API by adding endpoints.
But I don’t have too strong opinions on this also, I think both solutions have cons and pros.
I totally agree with these concerns. I am currently trying to implement event streaming in FCL and am trying to wrap my head around this exact problem regarding multiple socket connections.
When a user subscribes concurrently to several different events, ideally this would be pooled into one connection. Opening up multiple sockets querying the same/potentially overlapping data doesn’t make sense here (and I have concerns surrounding rate limiting). Instead of opening a socket A subscribing to Event1 and socket B subscribing to Event2, it makes sense to only open a single socket that subscribes to both Event1 & Event2, then dispatch the data to listeners in JS accordingly. Then as you alluded to, this problem only gets messier when FCL starts subscribing to even more execution data.
My idea was to open a new connection with different parameters each time the global set of ongoing event subscriptions changes, using this as their single source of truth. However, it gets tricky when managing the disconnection/reconnection phase when subscriptions change and the risk of missing events as we do this.
From the client’s perspective, it would probably be easier if we could maintain a single persistent connection that would be the single source of truth for execution data & route data accordingly. (This is the strategy used by Ethereum JSON-RPC)
I understand your concerns regarding APIs with granular subscriptions. However, it is indeed feasible to manage multiple subscriptions using a single connection. This concept should be similar to the one described in the article WebSocket and React: WSContext Components Subscription Pattern. In general, the approach involves establishing an initial connection for the first subscription and then utilizing the same connection for subsequent subscriptions, taking advantage of WebSocket’s duplex communication capability. The differentiation of messages by type enables handling on the client side. I think, if we decide to go in this direction, the approach should be described in the FLIP document in detail for further approval and then implemented on the backend side.
Since there haven’t been further discussions for some time, I’ve put together a FLIP document to push this topic forward. All the discussed points are incorporated into the FLIP. Please feel free to read through it and share your thoughts.
The resource streaming has been excluded from the scope of this FLIP. However, it will be implemented in the future. More research and discussion are required for that aspect.
Just a reminder about the proposal (FLIP 229) regarding expanding the Access Streaming API. I’ve already received a bunch of comments, but I would greatly appreciate more feedback and thoughts in the comments section. Thank you in advance for your participation!
Final Call for Your Input