Skip to content

Support alternative OpenAI hosts (OpenAI.com, GitHub Models, Ollama)#264

Open
Copilot wants to merge 14 commits intomainfrom
copilot/support-alternative-openai-hosts
Open

Support alternative OpenAI hosts (OpenAI.com, GitHub Models, Ollama)#264
Copilot wants to merge 14 commits intomainfrom
copilot/support-alternative-openai-hosts

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 21, 2026

This pull request introduces support for multiple OpenAI-compatible model providers (Azure OpenAI, OpenAI.com, GitHub Models, and Ollama) for both chat and embeddings, enabling flexible configuration via environment variables. It also updates infrastructure and code to generalize model host selection, adds documentation for alternative hosts, and refactors code to use more generic types and improved enum handling.

fixes #74

The most important changes are:

Multi-provider model host support:

  • Added support for selecting chat and embedding model providers via the CHAT_MODEL_HOST and EMBED_MODEL_HOST environment variables, with configuration blocks for Azure, OpenAI.com, GitHub Models, and Ollama in .env.example. Each provider has its own set of environment variables, allowing for flexible provider selection and mixing.
  • Added comprehensive documentation (docs/alternative_openai_hosts.md) explaining how to configure and use each supported provider, including setup instructions and environment variable details.

Infrastructure and deployment updates:

  • Updated Bicep templates and deployment parameters to support the new provider selection mechanism, including new parameters and outputs for all supported providers in infra/main.bicep and infra/main.parameters.json. Deprecated and replaced old variables with the new, more flexible naming scheme. [1] [2] [3] [4] [5] [6]

Codebase refactoring and type improvements:

  • Refactored code to use generic Embeddings and BaseChatOpenAI types instead of Azure-specific classes, allowing for easier integration of alternative providers. [1] [2] [3] [4]
  • Updated setup logic to accept separate configuration for chat and embedding APIs, including keys, endpoints, versions, hosts, and embedding dimensions, to accommodate provider-specific requirements.

Enum and compatibility improvements:

  • Improved enum handling in src/quartapp/approaches/schemas.py by introducing a StrEnum fallback for compatibility with Python versions earlier than 3.11, ensuring consistent behavior across environments.

Copilot AI and others added 2 commits March 21, 2026 23:02
…for chat and embeddings

Add OPENAI_CHAT_HOST and OPENAI_EMBED_HOST environment variables to configure
which OpenAI-compatible provider to use. Supported values: azure (default),
openai, github, ollama.

Key changes:
- utils.py: embeddings_api() and chat_api() now accept host parameter and
  return appropriate LangChain client (Azure or non-Azure)
- base.py/setup.py: Widen type hints to LangChain base types (BaseChatModel,
  Embeddings) for provider-agnostic usage
- config_base.py: Read OPENAI_CHAT_HOST, OPENAI_EMBED_HOST, and
  AZURE_OPENAI_EMBEDDINGS_DIMENSIONS env vars
- Configurable embedding dimensions for compatibility with models that produce
  different dimension sizes (e.g. Ollama's nomic-embed-text at 768 dims)
- .env.example: Add configuration examples for all supported hosts
- README.md: Add documentation section for alternative OpenAI hosts
- Tests: Add tests for all host types in test_utilities.py

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/df9650f5-df31-47d2-ad68-3fd4a83f3e1d
Copilot AI changed the title [WIP] Add support for alternative OpenAI hosts Support alternative OpenAI hosts (OpenAI.com, GitHub Models, Ollama) Mar 21, 2026
Copilot AI requested a review from john0isaac March 21, 2026 23:05
@john0isaac
Copy link
Copy Markdown
Collaborator

@copilot fix failing ci

- Use BaseChatOpenAI instead of BaseChatModel for chat type hints
  (fixes mypy errors for temperature attribute access in rag.py)
- Wrap Ollama API key string with SecretStr (fixes mypy arg-type error)
- Change str+Enum to StrEnum in schemas.py (fixes ruff UP042)
- Widen add_data.py embeddings type from AzureOpenAIEmbeddings to Embeddings
- Apply ruff formatter to utils.py

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/773ee6b2-1010-4dbb-b4e0-d79b407acb5f
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 21, 2026

@copilot fix failing ci

Fixed in 5eef407. Three CI issues:

  1. mypy - BaseChatModelBaseChatOpenAI (fixes temperature attribute access in rag.py), wrapped Ollama API key with SecretStr, widened add_data.py embeddings type to Embeddings
  2. ruff UP042 - (str, Enum)StrEnum in schemas.py (pre-existing on main, fixed here)
  3. ruff format - reformatted vector_store_api signature in utils.py

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • connection_string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/python python -m pytest tests/ -v --tb=short --ignore=tests/e2e.py -k not test_index and not test_favicon and not test_assets (dns block)
  • test-connection-string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/python python -m pytest tests/ -v --tb=short --ignore=tests/e2e.py -k not test_index and not test_favicon and not test_assets (dns block)
  • your-cosmos-db-connection-string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/python python -m pytest tests/ -v --tb=short --ignore=tests/e2e.py -k not test_index and not test_favicon and not test_assets (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@john0isaac
Copy link
Copy Markdown
Collaborator

@copilot there is a docs folder create a new markdown file in it with docs, don't add it to main readme

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 22, 2026

@copilot there is a docs folder create a new markdown file in it with docs, don't add it to main readme

Created docs/alternative_openai_hosts.md and removed the section from the main README in 5f04973.

@john0isaac
Copy link
Copy Markdown
Collaborator

@copilot rename env vars OPENAI_CHAT_HOST to CHAT_MODEL_HOST and OPENAI_EMBED_HOST to EMBED_MODEL_HOST and add more docs about usage with links to external documentations like microsoft docs/ github docs / openai docs for people to learn more

in src/quartapp/approaches/utils.py remove these as they should come from env vars not like this use os.getenv("OLLAMA_ENDPOINT", "DEFAULT_VALUE") where default is what you put

OLLAMA_ENDPOINT = "http://localhost:11434/v1"
GITHUB_MODELS_ENDPOINT = "https://models.github.ai/inference"
OLLAMA_DEFAULT_API_KEY = "nokeyneeded"

…DEL_HOST, use os.getenv for endpoints, enhance docs

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/58584972-4401-4218-ba14-662df5d7913a
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 22, 2026

@copilot rename env vars OPENAI_CHAT_HOST to CHAT_MODEL_HOST and OPENAI_EMBED_HOST to EMBED_MODEL_HOST and add more docs about usage with links to external documentations like microsof...

Done in 6c2b500:

  1. Renamed OPENAI_CHAT_HOSTCHAT_MODEL_HOST and OPENAI_EMBED_HOSTEMBED_MODEL_HOST across all files (Python, tests, infra, .env.example, docs).
  2. Removed hardcoded constants from utils.py — now uses os.getenv("OLLAMA_ENDPOINT", "http://localhost:11434/v1"), os.getenv("OLLAMA_API_KEY", "nokeyneeded"), and os.getenv("GITHUB_MODELS_ENDPOINT", "https://models.github.ai/inference").
  3. Enhanced docs/alternative_openai_hosts.md with an Azure OpenAI section and external documentation links for each provider (Microsoft docs, OpenAI API docs, GitHub Models docs, Ollama docs).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • connection_string
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q -k not test_index (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -q -k not test_index and not test_favicon (dns block)
  • test-connection-string
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q -k not test_index (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -q -k not test_index and not test_favicon (dns block)
  • your-cosmos-db-connection-string
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -x -q -k not test_index (dns block)
    • Triggering command: /usr/bin/python python -m pytest tests/ -q -k not test_index and not test_favicon (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@john0isaac john0isaac marked this pull request as ready for review March 22, 2026 04:54
@john0isaac john0isaac requested a review from Copilot March 24, 2026 08:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds multi-provider OpenAI-compatible client support across the app (chat + embeddings) by routing to Azure/OpenAI/GitHub Models/Ollama implementations based on new environment variables, while keeping Azure as the default for backwards compatibility.

Changes:

  • Add host-based dispatch in chat_api() / embeddings_api() and widen internal type hints to provider-agnostic LangChain base types.
  • Introduce CHAT_MODEL_HOST / EMBED_MODEL_HOST and optional AZURE_OPENAI_EMBEDDINGS_DIMENSIONS config wiring.
  • Update tests, examples, and add documentation for alternative hosts.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/quartapp/approaches/utils.py Dispatch chat/embeddings clients by host; adds Ollama/GitHub endpoint env support.
src/quartapp/config_base.py Reads new host env vars and optional embedding dimensions into Setup.
src/quartapp/approaches/setup.py Plumbs host + dimensions through setup; widens types for multi-host compatibility.
src/quartapp/approaches/base.py Widened constructor type hints to provider-agnostic interfaces.
src/quartapp/approaches/schemas.py Switch enums to StrEnum (with fallback).
src/scripts/add_data.py Widen embeddings parameter type to Embeddings.
tests/test_utilities.py Adds unit tests covering each host branch for chat/embeddings factories.
tests/conftest.py Updates env var fixtures and Setup fixture parameters for new host args.
infra/main.bicep Renames host parameters and exports new env vars (but still has Azure/OpenAI.com-only constraints).
infra/main.parameters.json Renames host parameters to match new bicep parameter names.
docs/alternative_openai_hosts.md New documentation for configuring OpenAI.com, GitHub Models, and Ollama.
.env.example Adds example configuration blocks for selecting hosts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@john0isaac
Copy link
Copy Markdown
Collaborator

@copilot address all of the pr comments, run all checks including tests, linting, formatting before committing

…ust dimensions parsing, updated bicep allowed values and unconditional model env vars

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/95e6b488-a479-4ba1-abe8-0d2e30a41066
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 25, 2026

@copilot address all of the pr comments, run all checks including tests, linting, formatting before committing

Addressed all PR review comments in c722381:

  1. utils.pyopenai host handled explicitly; unknown hosts now raise ValueError with supported values list
  2. infra/main.bicep@allowed updated to ['azure', 'openai', 'github', 'ollama']; model name env vars set unconditionally (fixes empty names for non-Azure hosts)
  3. config_base.pyAZURE_OPENAI_EMBEDDINGS_DIMENSIONS parsing now strips whitespace and raises a clear ValueError for non-integer values
  4. test_utilities.py — Added tests for ValueError on unsupported hosts

All checks pass: ruff lint ✅, ruff format ✅, mypy ✅, 68/68 unit tests ✅

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • connection_string
    • Triggering command: /usr/bin/python python -m pytest tests/ --ignore=tests/e2e.py -v (dns block)
  • test-connection-string
    • Triggering command: /usr/bin/python python -m pytest tests/ --ignore=tests/e2e.py -v (dns block)
  • your-cosmos-db-connection-string
    • Triggering command: /usr/bin/python python -m pytest tests/ --ignore=tests/e2e.py -v (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@john0isaac
Copy link
Copy Markdown
Collaborator

@copilot fix all of these issues support all hosts/ all models in bicep too

Copilot AI and others added 2 commits March 25, 2026 18:51
…oint override; fix bicep OPENAI_API_VERSION and AZURE_OPENAI_ENDPOINT for mixed hosts

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/bb3f28b0-4ce6-4c26-85ef-abeb755905bc
…ENAI_ENDPOINT for Ollama config

Co-authored-by: john0isaac <64026625+john0isaac@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Azure-Samples/Cosmic-Food-RAG-app/sessions/bb3f28b0-4ce6-4c26-85ef-abeb755905bc
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 25, 2026

@copilot fix all of these issues support all hosts/ all models in bicep too

Fixed all issues in a7109b5 and 4e84641:

  1. utils.py — Ollama branches now always use os.getenv("OLLAMA_ENDPOINT", ...) directly, never falling back to azure_endpoint. This prevents misrouting when AZURE_OPENAI_ENDPOINT is set for Azure in mixed-host deployments.
  2. infra/main.bicepOPENAI_API_VERSION now checks both hosts (chatModelHost == 'azure' || embedModelHost == 'azure'). AZURE_OPENAI_ENDPOINT is only set when either host is azure, blank otherwise.
  3. docs & .env.example — Updated Ollama examples to use OLLAMA_ENDPOINT instead of AZURE_OPENAI_ENDPOINT.
  4. Tests — Updated Ollama custom endpoint tests to use OLLAMA_ENDPOINT env var via patch.dict(os.environ).

All checks pass: ruff lint ✅, ruff format ✅, mypy ✅, 68/68 tests ✅

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • connection_string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -v (dns block)
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -q (dns block)
  • test-connection-string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -v (dns block)
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -q (dns block)
  • your-cosmos-db-connection-string
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -v (dns block)
    • Triggering command: /home/REDACTED/work/Cosmic-Food-RAG-app/Cosmic-Food-RAG-app/src/.venv/bin/pytest pytest tests/ --ignore=tests/e2e.py -q (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…e main.parameters.json and add data handling in add_data.py
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

john0isaac and others added 2 commits March 26, 2026 00:10
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@john0isaac john0isaac requested a review from Copilot March 25, 2026 22:10
@john0isaac john0isaac added the enhancement New feature or request label Mar 25, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support alternative OpenAI hosts

3 participants