Skip to main content

模态

Modal云平台提供了从本地计算机的Python脚本方便的按需无服务器云计算访问。 使用modal运行您自己的自定义LLM模型,而不是依赖LLM API。

本示例介绍了如何使用LangChain与modal HTTPS 网络端点进行交互。

使用LangChain进行问答是另一个如何将LangChain与Modal结合使用的示例。在该示例中,Modal端到端运行LangChain应用程序,并使用OpenAI作为其LLM API。

%pip install --upgrade --quiet  modal
# Register an account with Modal and get a new token.

!modal token new
Launching login page in your browser window...
If this is not showing up, please copy this URL into your web browser manually:
https://modal.com/token-flow/tf-Dzm3Y01234mqmm1234Vcu3

langchain.llms.modal.Modal 集成类要求您部署一个具有符合以下 JSON 接口的 web 端点的 Modal 应用程序:

  1. LLM 提示作为键 "prompt" 下的 str 值被接受
  2. LLM 响应作为键 "prompt" 下的 str 值返回

示例请求 JSON:

{
"prompt": "Identify yourself, bot!",
"extra": "args are allowed",
}

示例响应 JSON:

{
"prompt": "This is the LLM speaking",
}

一个满足此接口的 'dummy' Modal web 端点函数示例是

...
...

class Request(BaseModel):
prompt: str

@stub.function()
@modal.web_endpoint(method="POST")
def web(request: Request):
_ = request # ignore input
return {"prompt": "hello world"}

一旦您部署了 Modal web 端点,您可以将其 URL 传递给 langchain.llms.modal.Modal LLM 类。该类可以作为您链中的构建块。

<!--IMPORTS:[{"imported": "LLMChain", "source": "langchain.chains", "docs": "https://python.langchain.com/api_reference/langchain/chains/langchain.chains.llm.LLMChain.html", "title": "Modal"}, {"imported": "Modal", "source": "langchain_community.llms", "docs": "https://python.langchain.com/api_reference/community/llms/langchain_community.llms.modal.Modal.html", "title": "Modal"}, {"imported": "PromptTemplate", "source": "langchain_core.prompts", "docs": "https://python.langchain.com/api_reference/core/prompts/langchain_core.prompts.prompt.PromptTemplate.html", "title": "Modal"}]-->
from langchain.chains import LLMChain
from langchain_community.llms import Modal
from langchain_core.prompts import PromptTemplate
template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate.from_template(template)
endpoint_url = "https://ecorp--custom-llm-endpoint.modal.run"  # REPLACE ME with your deployed Modal web endpoint's URL
llm = Modal(endpoint_url=endpoint_url)
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"

llm_chain.run(question)

相关


Was this page helpful?


You can also leave detailed feedback on GitHub.

扫我,入群扫我,找书