vLLM 是一款专为大语言模型推理加速而设计的框架实现了 KV 缓存内存几乎零浪费解决了内存管理瓶颈问题。更多 vLLM 中文文档及教程可访问 →https://go.hyper.ai/Wa62f*在线运行 vLLM 入门教程零基础分步指南源码 examples/offline_inference/tpu.pyfrom vllm import LLM, SamplingParams prompts [ A robot may not injure a human being, It is only with the heart that one can see rightly;, The greatest glory in living lies not in never falling,, ] answers [ or, through inaction, allow a human being to come to harm., what is essential is invisible to the eye., but in rising every time we fall., ] N 1 # Currently, top-p sampling is disabled. top_p should be 1.0. # 当前TOP-P 采样被禁用。 top_p 应为 1.0。 sampling_params SamplingParams(temperature0, top_p1.0, nN, max_tokens16) # Set enforce_eagerTrue to avoid ahead-of-time compilation. # In real workloads, enforace_eager should be False. # 设置 enforce_eager true避免提前汇编。 # 在实际的工作负载中enforace_eager 应该是 False。 llm LLM(modelQwen/Qwen2-1.5B-Instruct, max_num_batched_tokens64, max_num_seqs4) outputs llm.generate(prompts, sampling_params) for output, answer in zip(outputs, answers): prompt output.prompt generated_text output.outputs[0].text print(fPrompt: {prompt!r}, Generated text: {generated_text!r}) assert generated_text.startswith(answer)