Environmental footprint

OpenGateLLM tracks the environmental impact of AI model usage through the EcoLogits library, which provides a comprehensive view of the environmental footprint of generative AI models at inference.

Model carbon footprint configuration

For each model provider, you can define the carbon footprint parameters in the config.yml file. The configuration includes the number of total and active parameters (in billions), as well as the hosting zone.

The following parameters are used for carbon footprint computation:

• model_carbon_footprint_total_params
• model_carbon_footprint_active_params
• model_carbon_footprint_zone

For more information, see Configuration documentation.

Example:

models:
  [...]
  - name: my-language-model
    type: text-generation
    providers:
      - type: openai
        url: https://api.openai.com
        key: ${OPENAI_API_KEY}
        model_name: gpt-4o-mini
        model_carbon_footprint_total_params: 35
        model_carbon_footprint_active_params: 35
        model_carbon_footprint_zone: WOR

info

Carbon footprint computation requires at least model_carbon_footprint_total_params to be defined. If not provided, the environmental impact will not be computed for that model provider.

Carbon footprint is only supported for text-generation and image-text-to-text model types.

Environmental impact metrics

For each call to a generative AI model, the API returns environmental impact metrics in the response, in the usage.carbon field. The metrics include minimum and maximum estimates to account for variability in model efficiency:

• kWh: Energy consumption in kilowatt-hours (kWh), representing the final electricity consumption.
• kgCO2eq: Global Warming Potential (GWP) in kilograms of CO2 equivalent (kgCO2eq), representing greenhouse gas emissions related to climate change.

Example response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "my-language-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30,
    "cost": 0.000015,
    "carbon": {
      "kWh": {
        "min": 0.0001234,
        "max": 0.0001456
      },
      "kgCO2eq": {
        "min": 0.0000567,
        "max": 0.0000672
      }
    }
  }
}

After the request is processed, the carbon footprint of the request is store in usage table by the hooks decorator attached to each endpoint. See usage monitoring documentation for more information.

How it works

Environmental impact is calculated using the EcoLogits library through the compute_llm_impacts function. The computation takes into account:

1. Model parameters:

   • Total number of parameters (model_carbon_footprint_total_params)
   • Active number of parameters (model_carbon_footprint_active_params, defaults to total params if not specified)

2. Token usage: The number of output/completion tokens generated by the model

3. Request latency: The time taken for the inference request (in seconds)

4. Electricity mix: The carbon intensity of electricity based on the hosting zone (model_carbon_footprint_zone). The electricity mix includes:

   • ADPE (Abiotic Depletion Potential - elements)
   • PE (Primary Energy)
   • GWP (Global Warming Potential)

For more information about methodology, see EcoLogits documentation.

note

If total_params is not defined or token_count is zero, the carbon footprint will be returned as zero for both energy and emissions.