MiniCPM4-0.5B-QAT-Int4-GPTQ-format 小显存llm

MiniCPM4-0.5B-QAT-Int4-GPTQ-format · 模型库from modelscope import AutoTokenizer from vllm import LLM, SamplingParams model_name OpenBMB/MiniCPM4-0.5B-QAT-Int4-GPTQ-format prompt [{role: user, content: 推荐5个北京的景点。}] tokenizer AutoTokenizer.from_pretrained(model_name, trust_remote_codeTrue) input_text tokenizer.apply_chat_template(prompt, tokenizeFalse, add_generation_promptTrue) llm LLM( modelmodel_name, quantizationgptq_marlin, trust_remote_codeTrue, max_num_batched_tokens32768, dtypebfloat16, gpu_memory_utilization0.8, ) sampling_params SamplingParams(top_p0.7, temperature0.7, max_tokens1024, repetition_penalty1.02) outputs llm.generate(promptsinput_text, sampling_paramssampling_params) print(outputs[0].outputs[0].text)