Encode images for chat input

These functions are used to prepare image URLs and files for input to the chatbot. The content_image_url() function is used to provide a URL to an image, while content_image_file() is used to provide the image data itself.

Usage

content_image_url(url, detail = c("auto", "low", "high"))

content_image_file(path, content_type = "auto", resize = "low")

content_image_plot(width = 768, height = 768)

Arguments

url

The URL of the image to include in the chat input. Can be a data: URL or a regular URL. Valid image types are PNG, JPEG, WebP, and non-animated GIF.

detail

The detail setting for this image. Can be "auto", "low", or "high".

path

The path to the image file to include in the chat input. Valid file extensions are .png, .jpeg, .jpg, .webp, and (non-animated) .gif.

content_type

The content type of the image (e.g. image/png). If "auto", the content type is inferred from the file extension.

resize

If "low", resize images to fit within 512x512. If "high", resize to fit within 2000x768 or 768x2000. (See the OpenAI docs for more on why these specific sizes are used.) If "none", do not resize.

You can also pass a custom string to resize the image to a specific size, e.g. "200x200" to resize to 200x200 pixels while preserving aspect ratio. Append > to resize only if the image is larger than the specified size, and ! to ignore aspect ratio (e.g. "300x200>!").

All values other than none require the magick package.

width, height

Width and height in pixels.

Value

An input object suitable for including in the ... parameter of the chat(), stream(), chat_async(), or stream_async() methods.

Examples

chat <- chat_openai(echo = TRUE)
#> Using model = "gpt-4.1".
chat$chat(
  "What do you see in these images?",
  content_image_url("https://www.r-project.org/Rlogo.png"),
  content_image_file(system.file("httr2.png", package = "ellmer"))
)
#> Here’s what I see in these images:
#> 
#> 1. **First Image:**  
#>    - This is the logo for the **R programming language**. It features 
#> a stylized capital letter "R" in blue, partially overlaid on a gray, 
#> ellipse-shaped background. R is a popular language used for 
#> statistical computing and graphics.
#> 
#> 2. **Second Image:**  
#>    - This is the hex sticker (logo) for the **httr2** R package. The 
#> design includes the package name "httr2" in a script font, above a red
#> illustration of a baseball player swinging a bat. This imagery is 
#> common in hex stickers for R packages, often representing the package 
#> in a fun or visually memorable way. The **httr2** package is used to 
#> work with HTTP (web) APIs in R.
#> 
#> Both images relate to the R ecosystem, with the first being the 
#> language itself and the second being an R package used for 
#> internet/web communication.

plot(waiting ~ eruptions, data = faithful)

chat <- chat_openai(echo = TRUE)
#> Using model = "gpt-4.1".
chat$chat(
  "Describe this plot in one paragraph, as suitable for inclusion in
   alt-text. You should briefly describe the plot type, the axes, and
   2-5 major visual patterns.",
   content_image_plot()
)
#> This plot is a scatter plot with the x-axis labeled “x” and the y-axis
#> labeled “y.” The data points are arranged in a circular pattern, with 
#> points forming an evenly spaced ring that is centered at the origin 
#> (0,0). The distribution suggests a relationship where all (x, y) pairs
#> lie at a fixed distance from the center, indicative of a mathematical 
#> circle. There are no outlying points, clusters, or trends in a 
#> particular direction besides the uniform circular shape.