🖥️ Orchestrator

Overview

The orchestrator is the central coordination node in the SIMD network. It manages job queues, schedules work across available workers, and aggregates results from completed simulations.

Key responsibilities:

Accepting and queuing simulation jobs
Matching jobs to capable workers based on requirements
Monitoring worker health and availability
Collecting and storing simulation results
Providing status updates and metrics

Subcommands

Command	Alias	Description
`simd orchestrator`	`orch`	Start the orchestrator node
`simd enqueue-job`	`submit`	Submit a new simulation job
`simd list-jobs`	`jobs`	List all jobs in the queue
`simd list-workers`	`nodes`	List connected worker nodes

Start Orchestrator

To start the orchestrator node, you'll need to specify your public IP address so workers can connect to you:

Terminal

sudo simd orchestrator --public-ip 34.172.234.124

Expected output:

Output

==========================================
  SIMD Orchestrator Node
==========================================
  Node ID: 12D3KooWEdHrqTLoXqXfqPSheytnjhR9jh9WLJpF8Q4FuwD3LTwN
  Key Dir: /root/.simd/orchestrator
  Public IP: 34.172.234.124
==========================================

2025-12-16T14:49:03.526279Z  INFO simd::network: Orchestrator WG pubkey: c7N27er/Ztwb7S5GjOoyedJTwXxK8DrECZO1Yw0q7xg=
2025-12-16T14:49:03.746833Z  INFO libp2p_swarm: local_peer_id=12D3KooWEdHrqTLoXqXfqPSheytnjhR9jh9WLJpF8Q4FuwD3LTwN
2025-12-16T14:49:03.747360Z  INFO simd::network: Orchestrator listening on /ip4/127.0.0.1/tcp/4001
2025-12-16T14:49:03.747446Z  INFO simd::network: Orchestrator listening on /ip4/10.128.0.2/tcp/4001
2025-12-16T14:49:03.747467Z  INFO simd::network: Orchestrator listening on /ip4/172.17.0.1/tcp/4001
2025-12-16T14:49:03.747480Z  INFO simd::network: Orchestrator listening on /ip4/10.0.0.1/tcp/4001

Submit Jobs

Submit a new CFD simulation job using the enqueue-job command:

Terminal

sudo simd enqueue-job   --artifact-path ./artifacts.zip

The job will be submitted to the queue and will be processed by the worker nodes. Available options:

--artifact-path — Path to the job artifact ZIP file (required). The artifact currently is static, it contains the laminar.py file, and a start.sh to run the simulation. It will be auto-generated by SIMD Agent in few weeks.
--db-dir — Database directory (default: ./orchestrator_db)
--store-dir — Storage directory (default: ./orchestrator_store)
--artifacts-dir — Artifacts directory (default: ./artifacts)
--min-ram-mb — Minimum RAM required in MB (default: 1024)
--min-vram-mb — Minimum GPU VRAM required in MB (default: 0)
--needs-docker — Require Docker on worker (default: true)
--needs-python — Require Python on worker (default: true)
--needs-cuda — Require CUDA on worker (default: true)

Example output:

Output

==========================================
  Job Enqueued Successfully
==========================================
  Job ID:   01KCKRR8A804XJCW0QP5J0PKNR
  Artifact: ./artifacts/01KCKRR8A804XJCW0QP5J0PKNR.zip
  SHA256:   9f127776bb0fc5de261e05dc5043f50567b7d2119e20cf92724862171cfc17dc
==========================================

Monitor Jobs

View all jobs in the queue:

Terminal

simd jobs

Filter by status:

Terminal

# Show only active jobs
simd jobs --status active

# Show completed jobs
simd jobs --status completed

# Show failed jobs
simd jobs --status failed

Example output:

Output

==========================================
  SIMD Jobs (filter: all, count: 3)
==========================================

  Job: 01KCKT7E0SKGQ306597Y2QXTH0
    Status:  ✅ COMPLETED
    Created: 2025-12-16 14:49:59 UTC
    Worker:  12D3KooWDJUFekzwCRGb...
    Started: 2025-12-16 14:49:59 UTC
    Finished: 2025-12-16 14:50:28 UTC
    Duration: 29s
    Result: 614f085adc2120a2...

  Job: 01KCKT2DJVF7Z0S3MGPZ7F5ZG1
    Status:  ❌ FAILED
    Created: 2025-12-16 14:47:15 UTC
    Worker:  12D3KooWDJUFekzwCRGb...
    Started: 2025-12-16 14:47:16 UTC
    Finished: 2025-12-16 14:47:46 UTC
    Duration: 30s
    Error: artifact download failed: error sending request for url (http://203.0.113.10:4000/artifacts/01KCKT2DJVF7Z0S3MGPZ7F5ZG1)

  Job: 01KCKRR8A804XJCW0QP5J0PKNR
    Status:  ❌ FAILED
    Created: 2025-12-16 14:24:13 UTC
    Worker:  12D3KooWEdJbRgmoq7h4...
    Started: 2025-12-16 14:24:14 UTC
    Finished: 2025-12-16 14:24:44 UTC
    Duration: 30s
    Error: artifact download failed: error sending request for url (http://203.0.113.10:4000/artifacts/01KCKRR8A804XJCW0QP5J0PKNR)

==========================================

Get JSON output for scripting:

Terminal

simd jobs --json

Example output:

Output

[
  {
    "job_id": "01KCKT7E0SKGQ306597Y2QXTH0",
    "state": "Completed",
    "requirements": {
      "min_ram_mb": 1024,
      "min_vram_mb": 0,
      "needs_docker": false,
      "needs_python": false,
      "needs_cuda": false
    },
    "created_at_unix": 1765896599,
    "assigned_to_peer": "12D3KooWDJUFekzwCRGbcj1kTb4Fz6PxeLCFsrdmrYAr2V45k9M9",
    "started_at_unix": 1765896599,
    "finished_at_unix": 1765896628,
    "artifact_sha256": "9f127776bb0fc5de261e05dc5043f50567b7d2119e20cf92724862171cfc17dc",
    "result": {
      "job_id": "01KCKT7E0SKGQ306597Y2QXTH0",
      "ok": true,
      "result_hash": "fe308cc92b9ca47b6fe7230ea577b15ccd890ea2e8ed6d17e44c3011d291a83b",
      "finished_at_unix": 1765896628,
      "error": null,
      "result_artifact_sha256": "614f085adc2120a2c9e3ef4f5b839b95a938f492fa7febedcb76a93acd69eccf"
    }
  },
  {
    "job_id": "01KCKT2DJVF7Z0S3MGPZ7F5ZG1",
    "state": "Failed",
    "requirements": {
      "min_ram_mb": 1024,
      "min_vram_mb": 0,
      "needs_docker": false,
      "needs_python": false,
      "needs_cuda": false
    },
    "created_at_unix": 1765896435,
    "assigned_to_peer": "12D3KooWDJUFekzwCRGbcj1kTb4Fz6PxeLCFsrdmrYAr2V45k9M9",
    "started_at_unix": 1765896436,
    "finished_at_unix": 1765896466,
    "artifact_sha256": "9f127776bb0fc5de261e05dc5043f50567b7d2119e20cf92724862171cfc17dc",
    "result": {
      "job_id": "01KCKT2DJVF7Z0S3MGPZ7F5ZG1",
      "ok": false,
      "result_hash": "82d755d53ae6c95b32e23723731c85d7f7e10ffd817233aaf3359e9285d393ed",
      "finished_at_unix": 1765896466,
      "error": "artifact download failed: error sending request for url (http://203.0.113.10:4000/artifacts/01KCKT2DJVF7Z0S3MGPZ7F5ZG1)",
      "result_artifact_sha256": null
    }
  },
  {
    "job_id": "01KCKRR8A804XJCW0QP5J0PKNR",
    "state": "Failed",
    "requirements": {
      "min_ram_mb": 1024,
      "min_vram_mb": 0,
      "needs_docker": false,
      "needs_python": false,
      "needs_cuda": false
    },
    "created_at_unix": 1765895053,
    "assigned_to_peer": "12D3KooWEdJbRgmoq7h4AiUGFXYxGSSksY7Xr4TfwnFfUSjq9PMq",
    "started_at_unix": 1765895054,
    "finished_at_unix": 1765895084,
    "artifact_sha256": "9f127776bb0fc5de261e05dc5043f50567b7d2119e20cf92724862171cfc17dc",
    "result": {
      "job_id": "01KCKRR8A804XJCW0QP5J0PKNR",
      "ok": false,
      "result_hash": "344fa875b10438833728ee24f5f81d5499e76e69ec76852cb72bf2d0acb935bd",
      "finished_at_unix": 1765895084,
      "error": "artifact download failed: error sending request for url (http://203.0.113.10:4000/artifacts/01KCKRR8A804XJCW0QP5J0PKNR)",
      "result_artifact_sha256": null
    }
  }
]

Monitor Workers

View all connected worker nodes:

Terminal

simd nodes

Example output:

Output

==========================================
  SIMD Workers (count: 2)
==========================================

  Worker: 12D3KooWEdJbRgmoq7h4...
    VPN IP:  10.0.0.2
    Status:  🟢 IDLE
    Last seen: 2025-12-16 14:45:37 UTC

  Worker: 12D3KooWDJUFekzwCRGb...
    VPN IP:  10.0.0.3
    Status:  🟢 IDLE
    Last seen: 2025-12-16 15:12:33 UTC
    RAM: 15788 MB
    GPU: Tesla T4 (15360 MB)
    Capabilities: Python=true, Docker=true, CUDA=true

==========================================

Get detailed JSON output:

Terminal

simd nodes --json

Example output:

Output

[
  {
    "peer_id": "12D3KooWEdJbRgmoq7h4AiUGFXYxGSSksY7Xr4TfwnFfUSjq9PMq",
    "wg_pubkey": "kKzekS10uNFoG0O6Fyh+eslheWlCN1Mo2JB1bXU/jkk=",
    "vpn_ip": "10.0.0.2",
    "artifact_token": "1847895454e7383966175a4177c9d2f123e07d6ff2ddd7f7a9a0237d6508c7f7",
    "last_seen_unix": 1765896337,
    "state": "Idle",
    "report": null
  },
  {
    "peer_id": "12D3KooWDJUFekzwCRGbcj1kTb4Fz6PxeLCFsrdmrYAr2V45k9M9",
    "wg_pubkey": "8GqBh5K5PdGfNC1cOV2XZ3iKe0/+xSxEqHp9iYTD118=",
    "vpn_ip": "10.0.0.3",
    "artifact_token": "ccc4455221450750c35cd45079a453281a61dec1898adcce5587d1813fc3da00",
    "last_seen_unix": 1765898013,
    "state": "Idle",
    "report": {
      "checked_at_unix": 1765896584,
      "has_wireguard_tools": true,
      "has_python": true,
      "has_docker": true,
      "has_cuda": true,
      "gpu": {
        "name": "Tesla T4",
        "vram_mb": 15360
      },
      "cpu_model": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
      "ram_mb": 15788,
      "os_pretty": "Ubuntu 24.04.3 LTS",
      "kernel": "6.14.0-1018-aws",
      "public_ip": null,
      "vpn_ip": "10.0.0.3",
      "network": {
        "ip_route_get_1.1.1.1": "1.1.1.1 via 172.31.0.1 dev ens5 src 172.31.2.102 uid 0 
    cache",
        "hostname_I": "172.31.2.102 172.18.0.1 172.17.0.1 10.0.0.3"
      }
    }
  }
]