|
| 1 | +# Developing the Power BI connector |
| 2 | + |
| 3 | +## Layout |
| 4 | + |
| 5 | +```text |
| 6 | +automation/powerbi/ |
| 7 | +├── DESIGN.md # architecture decisions — read first |
| 8 | +├── README.md # user-facing install/usage doc |
| 9 | +├── contract.json # data contract shared by the connector and backend tests |
| 10 | +├── connector/ # Power Query SDK project (CisoAssistant.pq + assets) |
| 11 | +├── signing/ # certificate generation + customer trust runbooks |
| 12 | +└── samples/ # starter .pbit template |
| 13 | +``` |
| 14 | + |
| 15 | +The backend contract test lives at |
| 16 | +`backend/app_tests/api/test_api_powerbi_contract.py` and runs with the |
| 17 | +regular backend test suite: it asserts that every JSON path consumed by |
| 18 | +`connector/CisoAssistant.pq` (as declared in `contract.json`) is present in |
| 19 | +the live serializer output. **When you change columns, update |
| 20 | +`CisoAssistant.pq` and `contract.json` together.** |
| 21 | + |
| 22 | +## Environment: Windows required for build/test |
| 23 | + |
| 24 | +The M source (`CisoAssistant.pq`) can be edited anywhere (VS Code + |
| 25 | +`PowerQuery.vscode-powerquery` for syntax/IntelliSense on macOS/Linux), but |
| 26 | +building and evaluating requires Windows: |
| 27 | + |
| 28 | +- The **Power Query SDK** VS Code extension (`PowerQuery.vscode-powerquery-sdk`) |
| 29 | + is Windows-only (wraps `PQTest.exe` / `MakePQX.exe`; not available for |
| 30 | + Mac Silicon, Rosetta doesn't work). |
| 31 | +- **Power BI Desktop** is Windows-only. |
| 32 | + |
| 33 | +On Apple Silicon: Parallels Desktop + Windows 11 ARM (~5 GB RAM is enough), |
| 34 | +repo mounted via Parallels shared folders. |
| 35 | + |
| 36 | +## Build & test loop (Windows VM) |
| 37 | + |
| 38 | +1. Open `automation/powerbi/connector/` as the VS Code workspace. |
| 39 | +2. Install the **Power Query SDK** extension; run its *Setup workspace* |
| 40 | + task if prompted (regenerates `.vscode/settings.json`). |
| 41 | +3. **Set credential** task → auth kind *Key* → paste a PAT from your dev |
| 42 | + instance. Credentials bind to the exact base URL in the query file — |
| 43 | + change the URL and the SDK may prompt again (*Clear ALL credentials* |
| 44 | + resets a confused state). |
| 45 | +4. **Point the test query at your dev instance**: the URL is the literal on |
| 46 | + the last line of `CisoAssistant.query.pq`. This is a local, uncommitted |
| 47 | + edit — the repo keeps the neutral `https://localhost:8443` placeholder; |
| 48 | + never commit your instance URL. From a VM, `localhost` is the VM itself: |
| 49 | + target the host instead (Parallels: the Mac is `10.211.55.2`; run |
| 50 | + `ALLOWED_HOSTS=localhost,127.0.0.1,10.211.55.2 python manage.py runserver 0.0.0.0:8000` |
| 51 | + on the host and use `http://10.211.55.2:8000`). Plain `http://` is |
| 52 | + accepted for loopback/private hosts only; the connector rejects it for |
| 53 | + public hosts (PAT travels in a header). |
| 54 | +5. *Evaluate current file* task → results appear in the panel. To inspect a |
| 55 | + specific table or debug a filter, replace the last line with a drill |
| 56 | + expression, e.g. |
| 57 | + `CisoAssistant.Contents("...") {[Name = "Facts"]}[Data] {[Name = "Applied Controls"]}[Data]` |
| 58 | + (optionally wrapped in `Table.SelectRows(..., each [updated_at] >= #datetime(...))` |
| 59 | + to watch fold behaviour in the backend logs) — same rule: revert before |
| 60 | + committing. |
| 61 | +6. *Build* task (pick **MakePQX**, not MSBuild) → `bin/AnyCPU/Debug/connector.mez`. |
| 62 | + MakePQX names the output after the workspace *folder*; the MSBuild path |
| 63 | + (`CisoAssistant.proj`) names it `CisoAssistant.mez` after the project. |
| 64 | + `.vscode/settings.json` points at the MakePQX name. |
| 65 | +7. Copy the `.mez` to the Custom Connectors folder — resolve it via the |
| 66 | + known folder (OneDrive may redirect Documents, especially on fresh VMs): |
| 67 | + ```powershell |
| 68 | + $dir = Join-Path ([Environment]::GetFolderPath("MyDocuments")) "Power BI Desktop\Custom Connectors" |
| 69 | + New-Item -ItemType Directory -Force $dir | Out-Null |
| 70 | + Copy-Item bin\AnyCPU\Debug\connector.mez "$dir\CisoAssistant.mez" -Force |
| 71 | + ``` |
| 72 | + Allow uncertified extensions in Power BI Desktop's security options, |
| 73 | + restart Desktop, and test **Get Data → CISO Assistant** end-to-end. |
| 74 | +8. Run *TestConnection* task before touching gateway-related code. |
| 75 | + |
| 76 | +### Testing the signed .pqx locally |
| 77 | + |
| 78 | +Pack + sign with the extension's bundled MakePQX (path under |
| 79 | +`%USERPROFILE%\.vscode\extensions\...powerquery-sdk...\.nuget\...\tools\`), |
| 80 | +then validate through Power BI Desktop, not MakePQX: |
| 81 | + |
| 82 | +1. `MakePQX pack -mz bin\AnyCPU\Debug\connector.mez -t bin\AnyCPU\Debug\CisoAssistant.pqx` |
| 83 | +2. `MakePQX sign <pqx> --certificate <pfx> --password <pwd> --replace` |
| 84 | +3. Put the `.pqx` in the Custom Connectors folder and **remove the `.mez`** |
| 85 | + there (both present = duplicate connector). |
| 86 | +4. Add the signing thumbprint to |
| 87 | + `HKLM:\Software\Policies\Microsoft\Power BI Desktop\TrustedCertificateThumbprints` |
| 88 | + (see `../signing/trust-thumbprint.md`), set Desktop's Data Extensions |
| 89 | + security to **(Recommended)**, restart. Desktop refusing to load the |
| 90 | + connector at Recommended = signature or trust problem; loading = the |
| 91 | + full customer path works. |
| 92 | + |
| 93 | +Known quirk (SdkTools 2.155.2, any host): `MakePQX verify` crashes with a |
| 94 | +`System.Threading.Tasks.Extensions` assembly-binding error while printing |
| 95 | +its report — the package's `MakePQX.exe.config` lacks binding redirects |
| 96 | +for the System.Text.Json dependency chain. CI patches the redirects into |
| 97 | +the config before running the tools (see the "Patch MakePQX binding |
| 98 | +redirects" step in the workflow); locally, either apply the same patch or |
| 99 | +skip `verify` and use the Desktop trust test above, which is the |
| 100 | +authoritative check anyway. |
| 101 | + |
| 102 | +`.mez` is just a zip of the connector folder contents — CI |
| 103 | +(`.github/workflows/powerbi-connector.yml`) builds it with |
| 104 | +`Compress-Archive`, packs it to `.pqx` with `MakePQX`, and signs it when |
| 105 | +the signing secrets are configured (see `signing/generate-cert.md`). |
| 106 | + |
| 107 | +## Local backend for testing |
| 108 | + |
| 109 | +Any running CISO Assistant works. For a filter/pagination sanity check |
| 110 | +without Power BI: |
| 111 | + |
| 112 | +```bash |
| 113 | +curl -H "Authorization: Token $PAT" \ |
| 114 | + "https://localhost:8443/api/applied-controls/?limit=2&offset=0&updated_at__gte=2026-01-01T00:00:00Z" -k |
| 115 | +``` |
| 116 | + |
| 117 | +## Versioning & release |
| 118 | + |
| 119 | +The connector version is the `[Version = "x.y.z"]` attribute on the first |
| 120 | +line of `CisoAssistant.pq` — semver, aligned with the compatibility policy: |
| 121 | + |
| 122 | +- **patch** — fixes, no contract change |
| 123 | +- **minor** — additive only (new tables/columns/bridges); never breaks an |
| 124 | + existing report |
| 125 | +- **major** — renames/removals in the contract, auth or data source kind |
| 126 | + changes; requires migration notes and should be avoided (`contract.json` |
| 127 | + + the backend contract test exist to keep changes additive) |
| 128 | + |
| 129 | +Release: bump `[Version]` (and `contract.json`/docs if the surface changed), |
| 130 | +merge, then tag `powerbi-v<x.y.z>`. CI refuses tags that don't match the |
| 131 | +`.pq` version, then builds, packs, signs (if the signing secrets are |
| 132 | +present), uploads artifacts, and attaches `CisoAssistant.mez` + |
| 133 | +`CisoAssistant.pqx` to the GitHub Release. Customer upgrade = replace the |
| 134 | +file, restart Power BI (reports and credentials bind to the data source |
| 135 | +kind, not the file or version). |
0 commit comments