Skip to content

fix: apply quant bridge host rewrite and file mounts to selected environment - #691

Merged
quantconnect-kronos-reviewer[bot] merged 3 commits into
QuantConnect:masterfrom
pcbeingused333:fix/live-deploy-brokerage-env-settings
Nov 2, 2026
Merged

quantconnect-kronos-reviewer[bot] merged 3 commits into
QuantConnect:masterfrom
pcbeingused333:fix/live-deploy-brokerage-env-settings

Conversation

@pcbeingused333

@pcbeingused333 pcbeingused333 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #690

Cause

LeanRunner only looked at the top level of the Lean config when rewriting quant-bridge-server-host (localhost/127.0.0.1 -> host.docker.internal) and when mounting transaction-log / quant-bridge-symbol-map-file. lean live deploy writes brokerage and live data provider settings under lean_config["environments"][<environment>], and Lean gives those precedence, so the container got the unrewritten localhost and no symbol map mount.

Fix

  • _get_top_level_and_selected_environment returns the top-level scope and the selected environment's entries.
  • _replace_localhost_with_docker_host applies the host.docker.internal rewrite to both scopes (get_basic_docker_config, get_basic_docker_config_without_algo).
  • _mount_common_directories mounts file keys (/Files/<key>) from both scopes. A key set in both places is mounted once, using the environment's value.
  • Debug log lines record each rewrite and mount.

Commands that use no environment (lean backtest, lean data download, research, optimize) only have the top-level scope, so their behavior is unchanged.

Tests

Four new cases in tests/components/docker/test_lean_runner.py: host rewrite with the setting under the selected environment (localhost and 127.0.0.1), symbol map mount from the environment, and a key set in both places (single mount).

lean/ from master with the new tests:

$ python -m pytest tests/components/docker/test_lean_runner.py -q
FAILED tests/components/docker/test_lean_runner.py::test_run_lean_rewrites_localhost_quant_bridge_host_in_selected_environment[localhost]
FAILED tests/components/docker/test_lean_runner.py::test_run_lean_rewrites_localhost_quant_bridge_host_in_selected_environment[127.0.0.1]
FAILED tests/components/docker/test_lean_runner.py::test_run_lean_mounts_quant_bridge_symbol_map_file_from_selected_environment
FAILED tests/components/docker/test_lean_runner.py::test_run_lean_mounts_quant_bridge_symbol_map_file_once_when_set_in_root_and_environment
4 failed, 31 passed, 6 skipped

This branch:

$ python -m pytest tests/components/docker/test_lean_runner.py -q
35 passed, 6 skipped
$ python -m pytest tests --ignore=tests/test_cli.py -q
2783 passed, 81 skipped

tests/test_cli.py (the Regression Testing workflow) drives the CLI through Docker with QuantConnect credentials, so it was not run locally. Its run on this PR needs maintainer approval.

End-to-end run

Run by a maintainer on Windows, Docker Desktop 29.5.2, image quantconnect/lean:latest (Lean 18222), from a CLI root whose lean.json already carries top-level quant-bridge-* keys from an earlier run, including "quant-bridge-server-host": "localhost":

lean live deploy "DatabentoTestAlgoPy" --brokerage "Quant Bridge" --data-provider-live "Quant Bridge" --quant-bridge-connection-type SAPI --quant-bridge-server-auth-id abc --quant-bridge-environment Beta --quant-bridge-server-host localhost --quant-bridge-server-port 8194 --quant-bridge-emsx-broker API --quant-bridge-emsx-account Test --quant-bridge-emsx-team a --quant-bridge-openfigi-api-key a --live-cash-balance "USD:1000" --detach --no-update --verbose
  1. No top-level quant-bridge-symbol-map-file:
    • --verbose output: _replace_localhost_with_docker_host(): replacing 'quant-bridge-server-host' value 'localhost' with 'host.docker.internal' twice (top level and environment), and _mount_common_directories(): mounting 'D:\Source\data\symbol-properties\quant-bridge-symbol-map.json' to '/Files/quant-bridge-symbol-map-file'.
    • docker inspect: bind mount D:\Source\data\symbol-properties\quant-bridge-symbol-map.json -> /Files/quant-bridge-symbol-map-file.
    • config.json inside the container: top level "quant-bridge-server-host": "host.docker.internal"; environments.lean-cli has "quant-bridge-server-host": "host.docker.internal", "quant-bridge-symbol-map-file": "/Files/quant-bridge-symbol-map-file", "live-mode-brokerage": "QuantBridgeBrokerage".
  2. Same command with "quant-bridge-symbol-map-file": "root-symbol-map.json" added at the top level of lean.json:
    • one mount only, the environment's quant-bridge-symbol-map.json; root-symbol-map.json was neither created nor mounted;
    • both the top level and environments.lean-cli read "quant-bridge-symbol-map-file": "/Files/quant-bridge-symbol-map-file", both hosts host.docker.internal.

Not related to this change: Lean then logs Not able to fetch brokerage factory with name: QuantBridgeBrokerage, because the latest published module package QuantConnect.Brokerages.TerminalLink 2.5.18217 (built 2026-10-29) predates the plugin rename and only contains TerminalLinkBrokerageFactory. The config and mounts above are produced before that point. A package built from the renamed plugin is needed for Quant Bridge live deployments to start.


This change was prepared with the assistance of an AI agent (Claude, by Anthropic).

pcbeingused333 and others added 3 commits October 31, 2026 06:32
…ronment

lean live deploy stores brokerage settings under lean_config["environments"][<env>],
so the top-level-only checks in LeanRunner never saw them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- the localhost to host.docker.internal rewrite was copied in both docker config builders, so a later fix could land in one and miss the other
- the method takes the config and returns it, so the change is visible at the call site instead of hidden inside a helper
- debug lines say which host was replaced and which file was mounted where, for live deploy troubleshooting
@Romazes Romazes added the kronos:review Request review label Nov 2, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 2, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is correct and matches #690. Nothing here blocks the code itself. What's missing is evidence, listed below.

What I checked

  • lean live deploy creates lean_config["environments"][environment_name] (lean/commands/live/deploy.py:207-210) and the module settings go into it (lean/components/util/json_modules_handler.py:39). get_complete_lean_config sets lean_config["environment"] (lean_config_manager.py:227), so _get_top_level_and_selected_environment finds the right scope.
  • In the published modules JSON, quant-bridge-symbol-map-file is an internal-input (value quant-bridge-symbol-map.json), not a path-parameter. So mount_paths (step 1 of _mount_common_directories) never writes /Files/... into the environment before step 6 reads it, and nothing gets mounted twice or from the wrong source.
  • A common case is a lean.json that already holds top-level Quant Bridge keys, saved by an earlier lean data download. Here, step 6 now mounts the environment's value and rewrites both scopes to /Files/<key>. The host rewrite also runs on both scopes. Before this change, Lean read the environment's unrewritten values, because environment entries override top-level ones. The new single-mount test covers this case.
  • The top-level-only paths (lean backtest, lean data download, research, optimize) behave as before: the scope list is just [lean_config] when there's no selected environment, and .get() handles a selected environment missing from environments.
  • The environment lookup follows the same pattern mount_paths and _mount_lean_config_and_finalize already use. No other host keys or file keys in LeanRunner have the same problem.

Changes requested

  1. CI hasn't run. The Build and Regression Testing workflows for both pushes are in action_required, waiting for a maintainer to approve them. Because the PR author isn't a member, I only read the code and didn't run it. A maintainer needs to approve the runs, and they need to pass.
  2. Description proof. "They fail before the change and pass after it" needs the commands and their output, e.g. pytest tests/components/docker/test_lean_runner.py on master with the new tests (failing) and on this branch (passing), plus the wider suite you ran. The description also predates the third commit: it still says _get_config_scopes (now _get_top_level_and_selected_environment), and it doesn't mention _replace_localhost_with_docker_host or the new debug log lines.
  3. End-to-end run. This changes what lean live deploy passes to the Lean container. Please run lean live deploy with --brokerage "Quant Bridge" --data-provider-live "Quant Bridge" --quant-bridge-server-host localhost. Then report the outcome: the generated config.json should have host.docker.internal and /Files/quant-bridge-symbol-map-file under the environment, and the container should have the symbol map mount. The command should also run with a top-level quant-bridge-symbol-map-file already in lean.json. Lean doesn't need to connect to a Quant Bridge server for this check: the config and mounts show the result. If the run isn't possible, say why in the description.

Affected repositories: none. Lean already applies environment overrides, and the modules JSON needs no change.

Risk: I set risk:high. The change is small, backwards compatible and easy to revert, but CI hasn't run and there's no end-to-end run yet. Passing CI and the run in item 3 would support risk:low.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:high High risk and removed kronos:review:working review: in progress labels Nov 2, 2026
@Romazes Romazes added the kronos:review Request review label Nov 2, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 2, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to my review of 2210781. There are no new commits, comments or description edits since then, so this only covers CI.

What changed

  • Build passed on 2210781, on every OS and Python version in the matrix. The pull request's own Build run is still action_required, but the same commit was also pushed to a branch in this repository, and that push ran Build. So the unit tests, the new test_lean_runner.py cases included, pass in CI.
  • Regression Testing still hasn't run. Its push trigger is branches: ['*'], and * doesn't match /, so pushing fix/live-deploy-brokerage-env-settings didn't start it. Its pull_request run is still action_required. It's the workflow that runs tests/test_cli.py, which drives the CLI through Docker, and that's the path this change touches. A maintainer needs to approve that run, and it needs to pass.

My earlier read of the code still holds. I found nothing new in it.

Still requested

  1. Regression Testing: approve and pass the run, as above.
  2. Description proof (unchanged from last time): the pytest commands and their output, showing the new tests failing on master and passing on this branch, plus the wider suite you ran. The description also needs to match the code: it names _get_config_scopes, which is now _get_top_level_and_selected_environment, and it doesn't mention _replace_localhost_with_docker_host or the new debug log lines.
  3. End-to-end run (unchanged from last time): lean live deploy with --brokerage "Quant Bridge" --data-provider-live "Quant Bridge" --quant-bridge-server-host localhost. Report two things: that the generated config has host.docker.internal and /Files/quant-bridge-symbol-map-file under the environment, and that the container has the symbol map mount. Also do the same run with a top-level quant-bridge-symbol-map-file already in lean.json. If the run isn't possible, say why in the description.

Affected repositories: none.

Risk: still risk:high. Build is green now, but Regression Testing and the end-to-end run are both missing. With those two, the change is small, backwards compatible and easy to revert, which would support risk:low.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 2, 2026
@Romazes Romazes added the kronos:review Request review label Nov 2, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 2, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to my review of 2210781. The head commit and the description are the same as last time. The one change is that the pull request's CI runs were approved and finished.

What changed

  • CI is green. Build and Regression Testing both passed on 2210781 for the pull request, so item 1 from my last review is done.
  • Keep in mind what Regression Testing actually ran. In Run CLI tests, tests/test_cli.py reports 1 skipped ("API credentials not specified"), because a pull request from a fork gets no repository secrets. That suite doesn't cover lean live deploy anyway, so the skip doesn't hide anything this change touches. But the green run doesn't exercise the deploy path through Docker either. The new test_lean_runner.py cases in the unit run are the only CI coverage of the change.

My earlier read of the code still holds. I found nothing new in it.

Still requested

  1. Description proof (unchanged): the pytest commands and their output, showing the new tests failing on master and passing on this branch, plus the wider suite you ran. The description also needs to match the code: it names _get_config_scopes, which is now _get_top_level_and_selected_environment, and it doesn't mention _replace_localhost_with_docker_host or the new debug log lines.
  2. End-to-end run (unchanged): lean live deploy with --brokerage "Quant Bridge" --data-provider-live "Quant Bridge" --quant-bridge-server-host localhost. Report two things: that the generated config has host.docker.internal and /Files/quant-bridge-symbol-map-file under the environment, and that the container has the symbol map mount. Then do the same run with a top-level quant-bridge-symbol-map-file already in lean.json. Lean doesn't need to connect to a Quant Bridge server for this. Anyone with Docker can run it, including a maintainer. If it can't be run, say why in the description.

Affected repositories: none.

Risk: still risk:high. CI is green now, and the change is small, backwards compatible and easy to revert. What risk:low still lacks is the end-to-end run in item 2, because no test in CI exercises the deploy path through Docker.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 2, 2026
@pcbeingused333

Copy link
Copy Markdown
Contributor Author

Thanks. The description is updated: it now uses the current helper names (_get_top_level_and_selected_environment, _replace_localhost_with_docker_host), mentions the debug log lines, and includes the pytest commands and output (4 failed on master, 35 passed on this branch, 2783 passed in the wider suite without tests/test_cli.py). Regression Testing needs a maintainer to approve the run. The end-to-end lean live deploy run was not possible without a logged-in QuantConnect account and a Quant Bridge server; the description says so.

This comment was drafted with Claude Code.

@Romazes Romazes added the kronos:review Request review label Nov 2, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued and removed kronos:review Request review kronos:review:queued review: queued labels Nov 2, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added the kronos:review:working review: in progress label Nov 2, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to my review of 2210781. The head commit is the same. Since then the description has been updated, which covers both items I had left open.

What changed

  • Description proof: done. The description now has the pytest commands and their output: the 4 new cases fail on master and pass on this branch (35 passed), and the wider suite passes (2783 passed). It also uses the current helper names and mentions the debug log lines.
  • End-to-end run: done. A QuantConnect maintainer ran lean live deploy with Quant Bridge as brokerage and live data provider and --quant-bridge-server-host localhost, from a lean.json that already had top-level quant-bridge-* keys. The run checks what I asked for and would have failed if the bug were still there:
    • localhost is rewritten to host.docker.internal both at the top level and under environments.lean-cli.
    • The container has the bind mount to /Files/quant-bridge-symbol-map-file, and the environment entry points at it.
    • With a top-level quant-bridge-symbol-map-file added, only the environment's file is mounted, and both scopes point at /Files/quant-bridge-symbol-map-file.
      This matches the code: in _mount_common_directories, configs[-1] is the environment's value, and every scope that holds the key gets rewritten.
  • CI: Build (push and pull request) and Regression Testing both pass on 2210781.

My earlier read of the code still holds: the environment lookup matches what mount_paths already does, and commands that don't select an environment (lean backtest, lean data download, research, optimize) behave as before.

A note, not blocking: the run also showed that Lean can't load QuantBridgeBrokerage yet, because the published brokerage module package predates the plugin rename in #685. That's a release step for the brokerage plugin, outside this repository and outside this fix. Quant Bridge live deployments won't start until that package is published.

Affected repositories: none for this change. The brokerage plugin package needs publishing, as noted above.

Risk: I set risk:low. The change is small, backwards compatible and easy to revert. It only touches how LeanRunner applies two existing rewrites to the selected environment. It has unit tests that fail without it, green CI and an end-to-end run that exercises the deploy path.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:low Low risk and removed risk:high High risk labels Nov 2, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot merged commit 3393cbd into QuantConnect:master Nov 2, 2026
72 checks passed
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:low Low risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lean live deploy skips the localhost host rewrite and symbol map mount for brokerage settings

2 participants