Streaming and async APIs

Streaming results

The chat() method does not return any results until the entire response has been received. (It can print the streaming results to the console, but it returns the result only when the response is complete.)

If you want to process the response as it arrives, you can use the stream() method. This may be useful when you want to display the response in realtime, but somewhere other than the R console (like writing to a file, or an HTTP response, or a Shiny chat window); or when you want to manipulate the response before displaying it, without giving up the immediacy of streaming.

The stream() method returns a generator from the coro package, which you can loop over to process the response as it arrives.

stream <- chat$stream("What are some common uses of R?")
coro::loop(for (chunk in stream) {
  cat(toupper(chunk))
})
#>  R IS COMMONLY USED FOR:
#>
#>  1. **STATISTICAL ANALYSIS**: PERFORMING COMPLEX STATISTICAL TESTS AND ANALYSES.
#>  2. **DATA VISUALIZATION**: CREATING GRAPHS, CHARTS, AND PLOTS USING PACKAGES LIKE  GGPLOT2.
#>  3. **DATA MANIPULATION**: CLEANING AND TRANSFORMING DATA WITH PACKAGES LIKE DPLYR AND TIDYR.
#>  4. **MACHINE LEARNING**: BUILDING PREDICTIVE MODELS WITH LIBRARIES LIKE CARET AND #>  RANDOMFOREST.
#>  5. **BIOINFORMATICS**: ANALYZING BIOLOGICAL DATA AND GENOMIC STUDIES.
#>  6. **ECONOMETRICS**: PERFORMING ECONOMIC DATA ANALYSIS AND MODELING.
#>  7. **REPORTING**: GENERATING DYNAMIC REPORTS AND DASHBOARDS WITH R MARKDOWN.
#>  8. **TIME SERIES ANALYSIS**: ANALYZING TEMPORAL DATA AND FORECASTING.
#>
#>  THESE USES MAKE R A POWERFUL TOOL FOR DATA SCIENTISTS, STATISTICIANS, AND RESEARCHERS.

Async usage

ellmer also supports async usage, which is useful when you want to run multiple chat sessions concurrently. This is primarily useful in Shiny applications, where using the methods described above would block the Shiny app for other users for the duration of each response.

To use async chat, instead of chat()/stream(), call chat_async()/stream_async(). The _async variants take the same arguments for construction, but return promises instead of the actual response.

Remember that chat objects are stateful, maintaining the conversation history as you interact with it. Note that this means it doesn’t make sense to issue multiple chat/stream operations on the same chat object concurrently, as the conversation history could become corrupted with interleaved conversation fragments. If you need to run multiple chat sessions concurrently, create multiple chat objects.

Asynchronous chat

For asynchronous, non-streaming chat, you use the chat() method as before, but handle the result as a promise instead of a string.

library(promises)

chat$chat_async("How's your day going?") %...>% print()
#> I'm just a computer program, so I don't have feelings, but I'm here to help you with any questions you have.

TODO: Shiny example

Asynchronous streaming

For asynchronous streaming, you use the stream() method as before, but the result is a async generator from the coro package. This is the same as a regular generator, except instead of giving you strings, it gives you promises that resolve to strings.

stream <- chat$stream_async("What are some common uses of R?")
coro::async(function() {
  for (chunk in await_each(stream)) {
    cat(toupper(chunk))
  }
})()
#>  R IS COMMONLY USED FOR:
#>
#>  1. **STATISTICAL ANALYSIS**: PERFORMING VARIOUS STATISTICAL TESTS AND MODELS.
#>  2. **DATA VISUALIZATION**: CREATING PLOTS AND GRAPHS TO VISUALIZE DATA.
#>  3. **DATA MANIPULATION**: CLEANING AND TRANSFORMING DATA WITH PACKAGES LIKE DPLYR.
#>  4. **MACHINE LEARNING**: BUILDING PREDICTIVE MODELS AND ALGORITHMS.
#>  5. **BIOINFORMATICS**: ANALYZING BIOLOGICAL DATA, ESPECIALLY IN GENOMICS.
#>  6. **TIME SERIES ANALYSIS**: ANALYZING TEMPORAL DATA FOR TRENDS AND FORECASTS.
#>  7. **REPORT GENERATION**: CREATING DYNAMIC REPORTS WITH R MARKDOWN.
#>  8. **GEOSPATIAL ANALYSIS**: MAPPING AND ANALYZING GEOGRAPHIC DATA.

Async generators are very advanced, and require a good understanding of asynchronous programming in R. They are also the only way to present streaming results in Shiny without blocking other users. Fortunately, Shiny will soon have chat components that will make this easier, where you can simply hand the result of stream_async() to a chat output.