Uploaded image for project: 'LLM AI Integration'
  1. LLM AI Integration
  2. LLMAI-76

Measure energy consumption for the benchmark

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Done
    • Major
    • 0.4
    • 0.3.1
    • None
    • Unknown

    Description

      In the LLM benchmark, we need to measure energy consumption of the different tasks. For this, we should measure energy consumption on the inference server and associate this data to the different tasks we execute, depending on the running time. It seems hard to do this exactly, we should therefore probably work with average values and try to come up with some estimates of the consumed power per input and output token.

      We should also compare our measurements to publicly reported performance in particular for parallel requests. When a publication reports a certain number of tokens per second on a certain GPU, we can, based on the maximum power consumption of that GPU, and the tokens per second derive a maximum of the consumed power per token.

      Attachments

        Activity

          People

            ppantiru Paul Pantiru
            MichaelHamann Michael Hamann
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: