# Overview Source: https://io.net/docs/guides/announcements/announcements Stay up to date with the latest updates, migrations, and important news from io.net. This section provides key information on platform upgrades, new features, and essential changes that impact suppliers and users. ### Service Update: Bare Metal (Self Serve) Moving Forward Bare Metal on Demand will no longer be available starting Wednesday, October 1, 2025. * If you need Bare Metal, please contact our sales team at [business@io.net](mailto:business@io.net) for managed services. * For self-service compute, check out our latest option: [Virtual Machines on Demand](/docs/guides/clouds/deploy-vm-on-demand). ### Important: Action Required for Expired Refresh Tokens Some suppliers using older refresh tokens (generated before our switch to non-expiring tokens) may experience token expiration after 365 days — causing worker downtime and authentication errors. For full details and steps to resolve the issue, please refer to the [Action Required for Expired Refresh Tokens](/docs/guides/announcements/important-action-required-for-expired-refresh-tokens) ### Supplier Migration to M4 Series – Upgrade Now! Suppliers on io.net can now seamlessly upgrade their existing inventory of M2 Pro, M2 Max, and M2 Ultra devices to the cutting-edge M4 series. This transition offers enhanced performance, improved efficiency, and ensures continued eligibility for earnings on the io.net platform. For full details on the migration process, please refer to the [Supplier Migration Guide: Upgrading to M4 Series Devices](/docs/guides/announcements/supplier-migration-guide-upgrading-to-m4-series-devices) # Action Required for Expired Refresh Tokens Source: https://io.net/docs/guides/announcements/important-action-required-for-expired-refresh-tokens May 2025 We’ve identified that some suppliers who generated their refresh tokens before we switched to non-expiring tokens may encounter token expiration after 365 days. This can lead to worker downtime and authentication errors such as: ```json json theme={null} {"error":"invalid_grant", "error_description":"Unknown or invalid refresh token."} ``` **Error:** *Authentication failed. Your token might be invalid or expired. Please re-generate one and try again. Use `--no_cache=true` flag to re-authenticate.* To resolve this, please follow these steps to generate a new **non-expiring** refresh token and restart your worker: **Step 1:** Run the `io.net worker` CLI binary with the `--no_cache=true` flag to initiate token regeneration. **Step 2:** Provide any required details to the CLI. Once successful, your worker will launch using the refreshed token. After completing the initial interactive authentication, you’ll see a new --token value. You can reuse this token to update other devices silently. Thank you for your continued support of io.net. The io.net Team # Supplier Migration Guide: Upgrading to M4 Series Devices Source: https://io.net/docs/guides/announcements/supplier-migration-guide-upgrading-to-m4-series-devices April 2025 Suppliers on io.net can now seamlessly upgrade their existing inventory of **M2 Pro, M2 Max, and M2 Ultra** devices to the cutting-edge **M4 series.** This transition ensures enhanced performance, improved efficiency, and continued eligibility for earnings on the io.net platform. **Important Pre-Migration Requirement**: Before initiating the migration, ensure your current M2 worker has been actively earning Block Rewards for at least the last 3 continuous hours (i.e., passing PoW tests and contributing work). If this condition is not met, the new M4 device will be automatically blocked during the migration process. ### Important Migration Dates: * **Migration Start:** UTC 09:00 AM on April 3rd, 2025 * **End of Support for M2 Devices:** UTC 23:59:59 on April 21st, 2025 * **Migration Window Closes:** UTC 23:59:59 on April 21st, 2025 Only currently connected devices with full stake are eligible for migration. ### M4 Series Device Options | Device | Earning Multiplier | Ray Cluster Pricing | Staking Requirement | | ------ | ------------------ | ------------------- | ------------------- | | M4 | 0.5x | \$0.10/hour | \$IO 200 | | M4 Pro | 0.75x | \$0.13/hour | \$IO 200 | | M4 Max | 1x | \$0.15/hour | \$IO 200 | ### Migration Process: ##### Step 1: Disconnect Your M2 Device Before migrating, ensure your **M2 Pro, M2 Max, or M2 Ultra** device is **disconnected** from the io.net network. ##### Step 2: Execute the Migration Command Once your M2 device is disconnected, run the following command on your new **M4, M4 Pro, or M4** Max device: ```curl curl theme={null} ./io_net_launch_binary_mac --device_id={same device id} --user_id={same user_id} --operating_system="macOS" --usegpus=false --device_name="{same device name}" ``` ##### Step 3: Ensure Proper Configuration Your new **M4 series** device should be **correctly configured** with a **stable internet connection** to avoid loss of **Block Rewards** during migration. ### Benefits of Migration: When migrating to the M4 series, your new device will: * **Retain** **the same device\_id** * **Retain the same \$IO staked** (M2 Pro/Max staking requirements match M4/M4 Pro/M4 Max) * **For M2 Ultra migrations**: The excess 50 \$IO (beyond the minimum staking requirement) will be withdrawable without a cooldown period after approximately 24-48 hours post-migration./*/* ### Additional Update: This migration ensures suppliers remain operational on io.net while benefiting from the **next-generation M4 series devices**. We appreciate your continued support as we drive decentralized computing forward. **Thank you for your continued partnership and support as we move towards a more innovative future together.** If you have any questions, please contact io.net support. The io.net Team # Architectural Layers Source: https://io.net/docs/guides/architecture/io-architecture The IO Portal architecture is a multi-layered, cohesive structure that provides a seamless, secure, and efficient user experience. Each layer has a distinct role, working in tandem to ensure the system's optimal performance. The architecture is built upon modern technologies, ensuring scalability, reliability, and robustness. ### User Interface This layer is the visual gateway for users. It comprises the ***Public website***, ***Customers area***, and ***GPU providers area (Workers)***. The design is intuitive and user-centric, ensuring easy navigation and interaction. > Tech Stack: ReactJS, Tailwind, web3.js, zustand. ### Security Layer A pivotal layer ensuring the system's integrity and safety. It encompasses a **Firewall** for network protection, an **Authentication Service** for user validation, and a **Logging Service** for tracking activities. > Tech Stack: Firewall (pfSense, iptables), Authentication (OAuth, JWT), Logging Service (ELK Stack, Graylog). ### API Layer Serving as the communication bridge, this layer has multiple facets: ******Public API for the website, Private APIs for Workers/GPU Providers and Customers, andInternal APIs****** for Cluster Management, Analytics, and Monitoring/Reporting. > Tech Stack: FastAPI, Python, GraphQL, RESTful services, gunicorn, solana. ### Backend Layer The powerhouse of the system. It manages Providers (Workers), Cluster/GPU operations, Customer interactions, Fault Monitoring, Analytics, Billing/Usage Monitoring, and Autoscaling. > Tech Stack: FastAPI, Python, Node.js, Flask, solana, IO-SDK (a fork of Ray 2.3.0), Pandas. ### Database Layer The data repository of the system. It uses **Main storage** for structured data and **Caching** for temporary and frequently accessed data. > Tech Stack: Postgres (Main storage), Redis (Caching). ### Message Broker/Task Layer This layer orchestrates asynchronous communications and task management, ensuring smooth data flow and efficient task execution. > Tech Stack: RabbitMQ (Message Broker), Celery (Task Management). ### Infrastructure Layer The foundational layer. It houses the **GPU Pool** with hardware from our verified partners. Orchestration tools manage deployments, while Execution/ML Tasks handle computations and machine learning operations. Additionally, it provides ***Data Storage solutions***. GPU performance is monitored using Nvidia-smi or NVIDIA DCGM. > Tech Stack: > > * GPU/CPU Pool > * Orchestration: Kubernetes, Prefect, Apache Airflow > * Execution/ML Tasks: Ray, Ludwig, Pytorch, Keras, TensorFlow, Pandas > * Data Storage: Amazon S3, Hadoop HDFS > * Containerization: Docker > * Monitoring: Grafana, Datadog, Prometheus, NVIDIA DCGM ### IO-SDK: The Powerhouse Behind io.net IO-SDK is our specialized fork of Ray, a core technology driving io.net's capabilities. Embracing Ray's native parallelism, IO-SDK effortlessly parallelizes Python functions, enabling dynamic task execution. Its in-memory storage ensures rapid data sharing between tasks, eliminating serialization delays. The dynamic auto-scaling feature means IO-SDK can quickly adapt to computational demands. Moreover, it is not just limited to Python; the language versatility and integration capabilities with leading ML frameworks like PyTorch and TensorFlow make it a robust and flexible choice. Whether on a single machine or a vast cloud platform, IO-SDK ensures io.net's scalability and performance. > Together, these layers, powered by the mentioned tech stacks, form a robust and scalable architecture for the io.net Portal, ensuring it meets the demands of modern users. # IO Network Source: https://io.net/docs/guides/architecture/io-network This article explores the concepts of mesh VPN networks and how we leverage their benefits to improve performance and reliability in the io.net protocol. **IO Network** is a cutting-edge networking backend that utilizes a secured mesh VPN to allow ultra-low latency communication between the antMiners nodes or 'workers'. ### Understanding Mesh VPN Networks A *Mesh VPN network* is a type of virtual private network (VPN) that connects nodes in a non-hierarchical, decentralized manner. Unlike traditional hub-and-spoke VPN architectures, which rely on a central concentrator or gateway, mesh VPN networks allow each node to connect directly to every other node in the network. This structure ensures data packets can travel along multiple paths, increasing redundancy, fault tolerance, and better load distribution. #### Advantages of *Mesh VPN networks*: 1. **Robust**: Mesh networks are resilient to individual node failures, as there are multiple paths for data to travel. This redundancy ensures that the network remains operational when one or more nodes experience issues. 2. **Scalability**: Adding new nodes to a mesh network does not significantly impact the overall network performance, making it easy to expand the network as needed. 3. **Low Latency**: By connecting nodes directly, mesh networks reduce the number of *hops* needed for data to travel between nodes. This reduction in latency enhances the performance of real-time applications and distributed computing. 4. **Optimized Load Distribution**: Mesh networks distribute traffic more evenly across nodes, preventing bottlenecks and ensuring optimal performance. ### Implementation of the *io.net Network* We built an io.net network that creates an efficient, resilient, and scalable networking backend. By adopting mesh networking principles, io.net delivers the following benefits to its users: 1. **Enhanced Performance**: io.net's mesh network architecture minimizes latency by allowing data to travel along the most efficient paths, optimizing application performance and user experience. 2. **Improved Resilience**: The decentralized nature of io.net's mesh network ensures that it remains operational even when individual nodes fail. This resilience translates to increased reliability and uptime for users. 3. **Seamless Scalability**: io.net's mesh network can easily grow to accommodate more nodes as the user base expands, ensuring consistent performance and adaptability. 4. **Distributed Computing**: By allowing direct connections between nodes, io.net's mesh network facilitates efficient distributed computing, which enables resource sharing and collaborative processing across the network. ### Decentralized Architecture and Privacy The decentralized nature of mesh VPN networks contributes to their security and privacy. Some notable aspects include: 1. **No Single Points of Failure**: The absence of a central concentrator or gateway in mesh VPN networks eliminates the risk of a single point of failure, ensuring that the network remains operational even if individual nodes are compromised or experience issues. 2. **Anonymity**: Since data travels along multiple paths within the mesh network, it becomes more difficult for an attacker to trace the origin or destination of the data, enhancing privacy for users. 3. **Traffic Obfuscation**: Mesh VPN networks can employ techniques like packet padding and timing obfuscation to make it harder for eavesdroppers to analyze traffic patterns and identify specific users or data streams. ### Network Access Control and Monitoring Next for io.net Network includes: 1. **Access Control Lists (ACLs)**: Nodes must define and enforce ACLs to restrict communication between specific nodes or groups of nodes, ensuring that sensitive data is only accessible to authorized parties and only available during the time they are hired for a specific cluster. 2. **Regular Auditing and Logging**: To maintain security and compliance, the io.net mesh VPN must be configured to allow us to perform regular audits and maintain logs of network activities, enabling administrators to identify and address potential vulnerabilities or breaches. # IO Tunnels Source: https://io.net/docs/guides/architecture/io-tunnels Reverse tunnels technology provides a way to bypass firewall and NAT restrictions, enabling secure access to remote resources. This article discusses the concept of reverse tunnels, how they work, and how io.net uses them to simplify engineer access to AntMiners Clusters. ### Understanding Reverse Tunnels A *Reverse tunnel* is a method to establish a secure connection from a client to a remote server by opening an inbound connection on the server side. This is the opposite of a conventional forward tunnel, where a client opens a connection to the server. By using reverse tunnels, engineers can access remote resources behind NAT routers and firewalls without the need for complex network configurations. ### How Reverse Tunnels Work 1. The remote server (IO Worker) initiates a connection to the client (engineer's machine) through an intermediate server (io.net server). 2. The io.net server listens for incoming connections from both the client and the remote server. Once the connection is established, data can be exchanged between the client and the remote server through the tunnel as if they were directly connected. The io.net network employs reverse tunnels to simplify access to io.net miners for engineers. The process involves: 1. The IO Worker establishes a connection to the io.net server, creating a reverse tunnel. 2. The engineer's machine connects to the io.net server as an intermediary. 3. The io.net server routes traffic between the engineer's machine and the IO Worker through the reverse tunnel. Engineers can securely access and manage IO Workers without the need for complex network configurations or dealing with firewalls and NAT restrictions. ### Benefits of Reverse Tunnels 1. **Simplified Access**: Engineers can easily access IO Workers without worrying about network restrictions, port forwarding, or VPNs. 2. **Enhanced Security**: Reverse tunnels provide a secure communication channel, ensuring data privacy and integrity. 3. **Scalability**: io.net can manage multiple IO Workers simultaneously, allowing engineers to work efficiently. 4. **Flexibility**: Reverse tunnels work across various platforms, ensuring compatibility with different operating systems and environments. # Overview Source: https://io.net/docs/guides/block-rewards/block-rewards ## Table of Contents * [What Are Block Rewards](/docs/guides/block-rewards/block-rewards#what-are-block-rewards) * [Block Rewards Allocation](/docs/guides/block-rewards/block-rewards#block-rewards-allocation) * [Block Reward Nomination Requirements](/docs/guides/block-rewards/block-rewards#block-reward-nomination-requirements) * [Block Rewards Nomination Checklist](/docs/guides/block-rewards/block-rewards#block-rewards-nomination-checklist) * [Wallets](/docs/guides/block-rewards/block-rewards#wallets) ### What Are Block Rewards? Block rewards are payments made to suppliers who provide their GPUs or CPUs to our network. This incentivizes supply-side network growth. These rewards are accrued hourly in ***\$IO***, following a predefined emission schedule. Rewards are potentially subject to slashing before distribution. ### Block Rewards Allocation Block Rewards are credited to Suppliers on an hourly basis for making their GPU or CPU available on the *IOG Network*, thereby incentivizing supply-side network growth. This payout is in **IO Coin** and follows a predetermined emission schedule. The current allocation of block rewards from each hourly emission is: * 95% to GPUs * 5% to CPUs The ratio of rewards may be adjusted in the future to manage network growth. To learn more, see [IO Coin](https://internet-of-gpus.readme.io/docs/what-is-io-coin). ### Initial Block Reward **The first Block Reward was created on June 25th 2024 12:00 UTC.** We will calculate and publish the initial block rewards, and block rewards use a claim mechanism that is similar to the *Ignition Program*. The first 7 days of block rewards were claimable together with *Ignition Reward Season 3*. Block Rewards for June 25th - June 30th will also be claimable. IO Device Level staking/Minimum staking is now required for Block Rewards. To learn more about staking, see [IO Staking](/docs/io-staking). *IOG Foundation* provides emission rewards to incentivize the correct growth of the IO ecosystem. To better align with *Ignition Reward Season 3*, the *IOG Foundation* has not requested io.net to cap the amount of devices eligible for block rewards. In the future, the *IOG Foundation* will monitor the reward distribution and may potentially cap the amount of devices eligible for the top percentage of nodes. A device cap is used to increase the alignment of the network's structure. ### Block Reward Nomination Requirements The requirements below must be met to be nominated for a Block Reward: * Device *Uptime* must be green for the past **5 hours**. * The minimum stake for the device must be met. To learn more about staking, see [IO Staking](/docs/guides/staking/io-staking). * The account holder for the device must have a valid Solana wallet. * The device’s status is **NOT** terminated or unsupported. * The device’s hardware multiplier is greater than **0**. * The device’s connectivity tier is greater than **0**. When a block is about to close, we test your device for *Uptime* and *PoW* again. ### Block Rewards Nomination Checklist Click the caret on the right side of the status row (**Eligible** in the example) to open the **Block Rewards Nomination Checklist**. Each step of the verification status is displayed, with a green or red check that indicates success or failure. The checklist displays the **current status** of the device's eligibility for a block reward. In the example case below, if a new block was opening, this device would be eligible. The **Block Rewards Nomination Checklist** is designed to offer total transparency into the Block Reward nomination process. **Device Limitations** * M3 devices with 8GB memory can be configured by users for higher memory. The minimum configuration should be 16GB of memory. * As of July, 5th, we removed Block Reward eligibility for Threadrippers. This is due to reports that malicious actors tried to inject Ryzen Threadripper devices into Block Rewards. We will reevaluate this and restore this option in the future. ### Wallets Wallets with *Exchange Deposit Addresses* (Custodial wallets) are not supported. To collect Block Rewards, worker earnings, or seasonal events, connect a *Web3 wallet* (Self-Custodial wallet) in your ***Account Settings***. Exchange Deposit Addresses are not supported because airdrop claims require a smart contract interaction. We do not support Exchange Deposit Addresses (Custodial wallets) for Block Rewards. If you connect a wallet with an Exchange Deposit Address to your account in ***Account Settings***, you will not be able to claim Block Rewards, seasonal events, nor worker earnings. If this is the case, please change it to a Self-Custodial wallet as soon as possible. # Monitor Block Rewards Source: https://io.net/docs/guides/block-rewards/monitor-block-rewards ## Table of Contents * [Block Rewards Tab](/docs/guides/block-rewards/monitor-block-rewards#block-rewards-tab) * [Block Details Table](/docs/guides/block-rewards/monitor-block-rewards#block-details-table) ### Block Rewards Tab The Block Rewards tab provides a transparent view of io.net's Block Rewards and coin emissions. Users can consult this information to monitor worker nominations and their status. Users can track the performance and success rates for worker nominations and block completion. Information on coin emissions and block rewards provide transparency about io.net network’s health. The example below shows the **Block Rewards** tab, with each section explained in detail afterward. The top section of the **Block Rewards** tab provides real-time data about IO Coin emissions and blocks. | Block | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------- | | *Total Coins Distributed* | The cumulative number of IO Coins distributed since inception. | | *Today’s Distributed Coins* | Total number of IO Coins distributed for the current calendar day. | | *Total Epochs Computed* | The cumulative number of epochs successfully added to the blockchain since inception. | | *Next Epoch Start* | The estimated time when the next epoch will be initiated. Blocks are added to the chain at hourly intervals. | | *Total Unique Suppliers Paid* | The number of unique workers that earned a block reward since inception. | | *Today’s Unique Suppliers Paid* | The number of unique workers that earned a block reward for the current calendar day. Each day ends at UTC+0. | ### Epoch Details Table The table provides a detailed overview of the epochs processed in the blockchain. The list provides a transparent and verifiable record of all epochs running in our network. Users can search for specific epochs and view details, verify transactions, and trace the history and integrity of the blockchain. The table is an important tool to monitor the blockchain's performance, transparency, and efficiency, and provides users with insight into the block creation process and the distribution of rewards. | Block | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------- | | *Total Coins Distributed* | The cumulative number of IO Coins distributed since inception. | | *Today’s Distributed Coins* | Total number of IO Coins distributed for the current calendar day. | | *Total Epochs Computed* | The cumulative number of epochs successfully added to the blockchain since inception. | | *Next Epoch Start* | The estimated time when the next epoch will be initiated. Blocks are added to the chain at hourly intervals. | | *Total Unique Suppliers Paid* | The number of unique workers that earned a block reward since inception. | | *Today’s Unique Suppliers Paid* | The number of unique workers that earned a block reward for the current calendar day. Each day ends at UTC+0. | ### Epoch Details Table The table provides a detailed overview of the epochs processed in the blockchain. The list provides a transparent and verifiable record of all epochs running in our network. Users can search for specific epochs and view details, verify transactions, and trace the history and integrity of the blockchain. The table is an important tool to monitor the blockchain's performance, transparency, and efficiency, and provides users with insight into the block creation process and the distribution of rewards. | Block | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------- | | *Total Coins Distributed* | The cumulative number of IO Coins distributed since inception. | | *Today’s Distributed Coins* | Total number of IO Coins distributed for the current calendar day. | | *Total Epochs Computed* | The cumulative number of epochs successfully added to the blockchain since inception. | | *Next Epoch Start* | The estimated time when the next epoch will be initiated. Blocks are added to the chain at hourly intervals. | | *Total Unique Suppliers Paid* | The number of unique workers that earned a block reward since inception. | | *Today’s Unique Suppliers Paid* | The number of unique workers that earned a block reward for the current calendar day. Each day ends at UTC+0. | ### Epoch Details Table The table provides a detailed overview of the epochs processed in the blockchain. The list provides a transparent and verifiable record of all epochs running in our network. Users can search for specific epochs and view details, verify transactions, and trace the history and integrity of the blockchain. The table is an important tool to monitor the blockchain's performance, transparency, and efficiency, and provides users with insight into the block creation process and the distribution of rewards. Screenshot 2026 06 11 At 1 09 48 PM 1 Screenshot 2026 06 11 At 1 09 48 PM 1 Screenshot 2026 06 11 At 1 09 48 PM 1 If a epoch reward is in progress, you are unable to click on it until it is complete. If a epoch reward is in progress, you are unable to click on it until it is complete. If a epoch reward is in progress, you are unable to click on it until it is complete. # Agent Cloud Source: https://io.net/docs/guides/clouds/agent-cloud Empower your AI agents to provision, manage, and scale GPU infrastructure via the io.net network. ## What is Agent Cloud? The **Agent Cloud is a Model Context Protocol (MCP)** server which allows any MCP-compatible AI agent (Claude Code, Cursor, Windsurf, etc.) to interact directly with [**io.net**](http://io.net) APIs. Instead of manual dashboard management, you can now manage your decentralized infrastructure through natural language. ## Why Connect to Agent Cloud? * **Agentic DevOps:** Ask your agent to "Find the cheapest 4x H100 cluster and deploy my PyTorch image," and let it handle the API calls. * **Plug-and-Play:** No custom SDKs to install. Use the universal MCP standard to bridge your AI tools to the large GPU cloud. * **Enterprise Ready:** Supports both simple API key headers for individuals and dynamic key forwarding for multi-tenant applications. ## Quickstart: Connect to the IO Cloud You can link your AI agent to our hosted MCP server in seconds. ### 1. Link Your Agent (Claude Code Example) Replace `` with your personal key from the io.net dashboard with `io-cloud` project scopes: ```shellscript theme={null} claude mcp add io-cloud https://mcp.io.solutions/mcp \ --transport http \ --header "x-api-key: " \ --scope project ``` ### 2. Verify Connection Once added, restart your agent and try these prompts: * *"What io-cloud tools are available to me?"* * *"List all active container deployments in my account."* * *"Find available NVIDIA H100 hardware specs."* ## Desktop Agent Configuration If you prefer using GUI-based agents, copy and paste the configurations below. Replace `` with your actual key with `io-cloud` project scopes. ### Claude Desktop Add this to your `claude_desktop_config.json` (found in `%AppData%\Claude` on Windows or `~/Library/Application Support/Claude` on macOS): ```javascript theme={null} { "mcpServers": { "io-cloud": { "type": "http", "url": "https://mcp.io.solutions/mcp", "headers": { "x-api-key": "" } } } } ``` ### Cursor & Windsurf 1. Open **Settings** > **Features** > **MCP**. 2. Click **+ Add New MCP Server**. 3. **Name:** `io-cloud` 4. **Type:** `command` (or `http` if supported). 5. **URL:** `https://mcp.io.solutions/mcp` 6. **Header:** `x-api-key: ` ## Capabilities & Documentation ### Available MCP Tools | **Tool Group** | **Capability** | **Purpose** | | :--------------- | :------------------------ | :--------------------------------------------- | | **CaaS** | `caas_list_deployments` | Real-time status of your container clusters. | | **VMaaS** | `vmaas_get_hardware_list` | Browsing available GPU/CPU configurations. | | **Provisioning** | `caas_deploy_container` | Instant provisioning of new compute resources. | ### Authentication Methods 1. **Static Header (Recommended for Individuals):** Include your API key with `io-cloud` project scopes in the `x-api-key` HTTP header during setup. 2. **Dynamic Forwarding (For Multi-User Apps):** Pass the key dynamically within the `auth.api_key` argument of any specific tool call. ## Programmatic Usage (Python Example) If you are building a custom integration, you can use the `mcp` Python SDK to interact with the IO Cloud server. ```python theme={null} import asyncio from mcp import ClientSession from mcp.client.http import http_client async def io_cloud_demo(): # 1. Initialize the HTTP Client async with http_client("https://mcp.io.solutions/mcp") as (read_stream, write_stream): async with ClientSession(read_stream, write_stream) as session: await session.initialize() # 2. List available hardware via tool call # Note: We pass the API key dynamically here (Mode A) result = await session.call_tool( "vmaas_get_hardware_list", arguments={ "auth": {"api_key": "YOUR_IO_NET_API_KEY"}, "gpu": "NVIDIA H100" } ) print(f"Available H100s: {result}") if __name__ == "__main__": asyncio.run(io_cloud_demo()) ``` ## Troubleshooting * **Authentication Failure (401/403):** Verify your API key in the io.net dashboard. * **"No API key provided":** Ensure the header is correctly set in your JSON config or passed in the tool arguments. * **Connection Timeout:** Verify your network allows outbound traffic to `https://mcp.io.solutions`. **Ready to scale your infrastructure?** For high-volume requirements or custom integration support, contact us at [support@io.net](mailto:support@io.net) # Confidential Compute Attestation Source: https://io.net/docs/guides/clouds/confidential-compute-attestation-overview Learn what confidential compute attestation is and how it can be utilized to verify secure workloads on io.net Cloud. Explore Trusted Execution Environments (TEE), GPU verification, and hardware-based security for sensitive compute. ## Overview [io.net](http://io.net) provisioned *Confidential Compute VMs* are equipped with NVIDIA H200 GPUs featuring hardware-based confidential computing capabilities. This guide shows you how to verify that your GPUs are genuine NVIDIA hardware with confidential computing features properly enabled. ### Key Takeaways * Understand why GPU verification is essential for confidential computing. * Learn how to confirm GPU authenticity using cryptographic attestation. * Follow clear, step-by-step instructions to run verification on your VM. ### Prerequisites * SSH access to your Confidential Compute VM * Python 3.7 or later (pre-installed on your VM) * Basic command line understanding * 5 minutes to perform the initial set up ## Why does GPU verification matter? ### Trust, but Verify When running sensitive workloads on **Confidential Computing (CC)** infrastructure, it is not enough to rely on a provider’s assurances. You need **cryptographic proof** that the hardware is authentic, correctly configured, and operating in a secure state. GPU attestation provides independent verification that: * **The GPUs are genuine NVIDIA hardware** - not counterfeit, emulated, or misrepresented. * **Firmware is intact and unmodified** - ensuring no tampering with GPU software or drivers. * **Confidential computing features are enabled** - confirming that CC mode is active and functioning properly. * **Hardware measurements match expected “golden” values** - validating that the GPU’s state aligns with NVIDIA’s reference integrity manifests. ### Security and Compliance GPU attestation is a foundational security mechanism for confidential computing environments. It helps ensure: * **Zero-trust architecture** - trust no component by default, verify every claim cryptographically. * **Regulatory and compliance adherence** - meet requirements that mandate hardware verification. * **Data protection** - guarantee that sensitive workloads run only on validated, trustworthy hardware. * **Clear auditability** - produce cryptographic evidence for stakeholders, security teams, and auditors. ### How It Works Attestation relies on **cryptographic proofs** that cannot be falsified, forged or altered. The verification process follows a clear chain of checks: ``` Your VM → Collect GPU Evidence → Verify Certificates → Check Measurements → Validate Signatures → Result: Verified or Check Failed ``` **What gets verified:** During attestation, the following elements are validated: * **A four-level GPU certificate chain**, traced back to the NVIDIA Root Certificate Authority. * **Certificate revocation status**, checked via OCSP to ensure no certificates have been revoked. * **Driver firmware measurements**, consisting of 64 SHA-384 cryptographic hashes. * **VBIOS firmware measurements**, consisting of 64 SHA-384 cryptographic hashes. * **Digital signatures on all attestation evidence**, ensuring integrity and authenticity. ## What is GPU Attestation? ### Cryptographic Proof of Authenticity GPU attestation is a cryptographic verification process rooted in **hardware-based trust**. It provides verifiable proof that a GPU is authentic, securely configured, and operating in a trusted state. Specifically, it validates four core properties: Every NVIDIA GPU that supports Confidential Computing includes: * A **unique hardware identity** permanently embedded in silicon during manufacturing. * A **certificate chain** signed by NVIDIA’s Root Certificate Authority. * **Cryptographic keys** stored in tamper-resistant hardware. **What this proves:**\ The GPU is genuine NVIDIA hardware, produced by NVIDIA, and not counterfeit or emulated. During attestation, the system: * Measures all GPU firmware using **SHA-384 cryptographic hashes.** * Compares measurements against NVIDIA’s **Reference Integrity Manifests (RIMs)**. * Verifies the **digital signatures** on the RIM files. **What this proves:**\ The GPU firmware has not been modified, tampered with, or compromised. The verification process confirms that: * **Confidential Computing (CC)** is enabled. * **Protected PCIe (PPCIE)** is correctly configured. * The GPU is operating in the **expected secure state**. In this context, **CC** refers to the general concept of **Confidential Compute**, where the GPU operates in a secure, protected environment. This is different from **CC mode** (also referred to as `CC State`), which is a specific configuration within the GPU. `CC State` cannot be enabled at the same time as **PPCIe**, the two modes are mutually exclusive. `CC State` specifically refers to scenarios where a single GPU (or multiple GPUs that are **NOT** NVLink-connected) is passed through directly to a virtual machine. **What this proves:**\ Confidential Computing features are actively enabled and functioning, not merely claimed. **How CC and Protected PCIe (PPCIe) Work Together** GPU confidentiality is controlled through **two independent mechanisms**: * **CC (Confidential Compute / CC State)** * **PPCIe (Protected PCIe)** These settings determine *how* and *which* GPUs operate in confidential mode. Each GPU can have CC and PPCIe enabled or disabled independently, but only certain combinations are valid. * **CC = OFF, PPCIe = OFF**\ No confidential computing is enabled. * **CC = ON, PPCIe = OFF**\ Only a subset of GPUs is confidential. This mode is **NOT compatible with NVLink-connected GPUs**. * **CC = OFF, PPCIe = ON**\ The **entire GPU set in the server, including NVLink connected GPUs**, operate in confidential mode. *This is the configuration used in our recommended setup.* * **CC = ON, PPCIe = ON**\ This is an invalid configuration. The virtual machine will either fail to boot or the CUDA drivers will not load. Attestation uses a **nonce** (a cryptographic challenge) to: * Prevent replay attacks. * Ensure the evidence is current, not reused or pre-recorded. * Confirm that measurements reflect the GPU’s **current state**. **What this proves:**\ The attestation is happening in real time on this specific GPU. ### Official NVIDIA Technology **GPU attestation is performed using NVIDIA’s official tooling:** * **Package:** `nv-attestation-sdk` (official NVIDIA Python SDK) * **Source:** PyPI (Python Package Index) * **Version:** 2.6.3 (as of December 2025) * **License:** Apache 2.0 * **Repository:** [https://github.com/NVIDIA/nvtrust](https://github.com/NVIDIA/nvtrust) This is **not a third-party tool**. It is NVIDIA’s production-ready attestation framework, used by enterprise customers worldwide. ### Standards-Based Approach NVIDIA GPU attestation is built on widely adopted industry standards: * **SPDM (Security Protocol and Data Model):** Version 1.1 for device authentication. * **X.509 PKI:** Standard public key infrastructure for certificates. * **OCSP:** Online Certificate Status Protocol for revocation checks. * **TCG standards:** Trusted Computing Group measurement and attestation specifications. ## Quick Reference **One-time setup (approximately 2–3 minutes):** ```bash theme={null} mkdir -p ~/gpu-verification && cd ~/gpu-verification python3 -m venv venv source venv/bin/activate pip install --upgrade pip pip install nv-attestation-sdk # Create the verify_gpu.py script (refer to Step 5 of the GPU Attestation and Verification Guide) ``` **Run verification (approximately 30 seconds):** ```bash theme={null} cd ~/gpu-verification source venv/bin/activate python3 verify_gpu.py ``` **Successful verification output:** ``` ✅ VERIFICATION SUCCESSFUL This system has X genuine NVIDIA GPU(s) with Confidential Computing features enabled and operational. ``` For a more comprehensive guide for verifying your Confidential Compute VM, refer to the [GPU Attestation and Verification Guide](/docs/guides/clouds/confidential-compute-attestation-guide). ### Key Points * **Official NVIDIA tooling** - uses the NVIDIA-provided `nv-attestation-sdk`. * **Cryptographic assurance** - verification cannot be forged or bypassed. * **Comprehensive validation** - certificates, firmware measurements, and signatures are all verified. * **Fast execution** - completes in approximately 30 seconds after initial setup. * **Self-contained operation** - runs entirely within the virtual machine without external accounts. ### Next Steps 1. Complete the one-time setup. 2. Run GPU verification on the virtual machine. 3. Integrate verification into the security or deployment workflow. 4. Perform verification before processing sensitive workloads. 5. Contact support if verification fails. ## FAQs **Recommended frequency:** * **Daily:** Before processing sensitive or regulated workloads. * **After system changes:** Following any reboot, update, or migration. * **Initial provisioning:** When the virtual machine is first received. * **Compliance requirements:** As dictated by organizational security policies. **Why regular verification is important:** * Detects firmware tampering or unauthorized modification. * Confirms that *Confidential Computing* features remain enabled after updates. * Provides a verifiable audit trail for compliance and governance. **Yes.** Verification can be integrated into startup or pre-workload scripts. ```bash theme={null} #!/bin/bash # Example: Run verification before starting an application cd ~/gpu-verification source venv/bin/activate python3 verify_gpu.py if [ $? -eq 0 ]; then echo "✅ Verification passed. Starting workload." # Insert application startup commands here else echo "❌ Verification failed. Aborting startup." exit 1 fi ``` **No.** Verification: * Runs independently of GPU compute workloads. * Typically completes within 30–60 seconds. * Does not interfere with running applications. * Can be executed while GPUs are performing other tasks. **Immediate steps to follow:** 1. Retry the verification once, as failures may be caused by transient network issues. 2. Review system logs using `journalctl -xe`. 3. Confirm that the NVIDIA driver is installed and functioning using `nvidia-smi`. 4. Contact support and provide complete error messages and output. Sensitive or confidential data should not be processed until verification succeeds. **Yes.** This verification process uses: * The official NVIDIA Python SDK (`nv-attestation-sdk`). * The same verification logic used by NVIDIA enterprise customers. * Reference Integrity Manifests and certificate chains served directly by NVIDIA. * Industry-standard cryptographic validation mechanisms. This is not a third-party tool. It is NVIDIA’s official, production-grade attestation solution. The verification process validates the following components: **Hardware** * The GPU is genuine NVIDIA hardware, verified through a certificate chain to the NVIDIA Root Certificate Authority. * The hardware identity matches the issued certificates. * Certificates have not been revoked, as verified through OCSP. **Firmware** * Driver firmware measurements (64 SHA-384 cryptographic hashes). * VBIOS firmware measurements (64 SHA-384 cryptographic hashes). * Measurements match NVIDIA’s reference “golden” RIM values. * All firmware and measurement signatures are cryptographically valid. **Configuration** * Confidential Computing mode is enabled. * Protected PCIe (PPCIE) is correctly configured. * The GPU ready state is operational. **No.** Verification is based on strong cryptographic guarantees, including: * A **hardware root of trust**, with cryptographic keys embedded in tamper-resistant silicon. * **Certificate chains** signed by NVIDIA’s Root Certificate Authority, protected by NVIDIA’s private keys. * **Cryptographic signatures** that cannot be forged without NVIDIA’s private keys. * **Fresh nonces** that prevent replay of previously captured attestation results. Even if the operating system is fully compromised, an attacker cannot: * Forge NVIDIA’s digital signatures. * Create certificates that successfully validate against the NVIDIA Root CA. * Modify firmware measurements without detection. * Bypass the hardware root of trust. GPU attestation verifies GPU authenticity and configuration, but it DOES NOT validate: * Hypervisor security or configuration. * Host operating system security. * Network isolation between virtual machines. * Physical datacenter security. * Guest operating system security within the VM. **For comprehensive protection:**\ GPU attestation should be used as part of a defense-in-depth security strategy. The verification tooling can be independently validated through the following means: 1. **Official NVIDIA package distribution:** Published on PyPI ```bash theme={null} pip show nv-attestation-sdk # Displays: nv-attestation-sdk 2.6.3 ``` 2. **Open-source implementation:** Publicly available for review Repository: [https://github.com/NVIDIA/nvtrust](https://github.com/NVIDIA/nvtrust) 3. **Package signing:** Signed and distributed by NVIDIA ```bash theme={null} pip show --verbose nv-attestation-sdk ``` 4. **Checksum verification:** Validate package integrity ```bash theme={null} pip hash nv-attestation-sdk ``` This is expected for nodes with **NVLink-connected GPUs**. CC State is only used when individual GPUs (or non-NVLink GPUs) are passed into a VM. Because NVLink-connected GPUs must be passed through **as a complete set**, CC State remains `OFF`. For these nodes, confidentiality is provided through **Protected PCIe**, not `CC State`. You can confirm this by checking: * `Multi-GPU Mode: Protected PCIe` → GPUs are running in confidential mode. * `CPU CC Capabilities: INTEL TDX` → CPU is running in confidential mode. # Confidential Compute Source: https://io.net/docs/guides/clouds/confidential-compute-overview Run secure AI workloads with Intel TDX encryption on io.cloud. Protect data in use on H100, H200, and B200 GPUs at up to 90% lower cost. ## Overview **Confidential Compute VMs** on [io.net](http://io.net) provides hardware-based encryption that protects your AI data, model weights, and computations from unauthorized access, including cloud operators, administrators, or malicious actors. Unlike traditional cloud environments that only encrypt data at rest and in transit, **Confidential Compute** maintains end-to-end encryption even in memory while your workloads are running. This capability is powered by **Intel Trust Domain Extensions (TDX)** on **5th** and **6th** **generation** **Intel Xeon processors**, combined with [io.net](http://io.net)’s decentralized GPU network. ## **What Is Confidential Computing?** **Confidential Computing** is a hardware-based security model that ensures your data stays encrypted even while being processed. Using **Trusted Execution Environments (TEEs)**, it creates isolated regions of memory where code and data remain protected from all external access, including the host OS, hypervisor, and cloud administrators. ### **Key Technology** * **Intel Trust Domain Extensions (TDX):** Provides per-VM encryption and isolation for workloads. * **Trusted Execution Environments:** Secure enclaves ensure no unauthorized party can read memory contents. * **Hardware Attestation:** Verifies the integrity of BIOS, firmware, and kernel before execution. [io.net](http://io.net) delivers the same enterprise-grade protection as *Azure* and *Google Cloud* Confidential VMs, at a fraction of the cost, with bare-metal GPU performance and no centralized dependency. ## Why It Matters When training or running AI models on traditional hyperscalers, your data sits unencrypted in memory where it can be exposed through: * Insider threats or compromised infrastructure operators * Cyberattacks targeting shared cloud environments * Government subpoenas or data access requests These risks endanger: * Proprietary model architectures * Customer PII, health records, and financial data * Sensitive inference requests revealing business logic ## **Key Benefits** * **Protect Intellectual Property:** Encrypt training data, model weights, and architectures during AI model development. * **Enable Secure Collaboration:** Support multi-party or federated learning while keeping datasets isolated and private. * **Ensure Regulatory Compliance:** Meet HIPAA, GDPR, and financial data requirements with full auditability. * **Secure Inference at Scale:** Protect live prompts, responses, and intermediate model states. * **Reduce Costs:** Run on a decentralized GPU infrastructure, **up to 90% cheaper** than hyperscalers. ## **Getting Started** Confidential Compute is available upon request through [io.net](http://io.net)’s managed services, supporting machines from H100 up to B200. To begin using **Confidential Compute** on [io.net](http://io.net): 1. **Request Access:** Submit a provisioning request by emailing `business@io.net` . Mention your preferred GPU type (H100, H200, or B200) and provide basic project details. Our specialists will reach out to understand your needs and assist with configuring the machines to your specifications. 2. **Deploy a Confidential VM:** Once set up, deploy your Confidential VM using the web console or API. 3. **Verify Attestation:** Before running sensitive workloads, verify your instance’s attestation report to confirm the integrity of its BIOS, firmware, and TDX configuration. This ensures your VM is operating in a trusted, secure state. 4. **Integrate Seamlessly:** Run workloads on **Ubuntu 24.04 (LTS)** with **built-in Intel TDX support**. Existing MLOps pipelines run without modification. You can migrate existing workflows directly to [io.net](http://io.net) without changing your training or deployment code. 5. **Run Secure Workloads:** Connect to your instance and deploy your AI training or inference workloads as usual. All computations, model weights, and data remain encrypted in memory throughout processing, maintaining full performance and GPU acceleration. Performance impact is minimal and with full GPU capabilities preserved. This validates secure AI pipelines on cost-effective infrastructure. ## **Compliance and Use Cases** **Confidential Compute** supports secure operations across regulated and high-risk industries: * **Healthcare:** Train on PHI and diagnostic data securely. * **Finance:** Protect trading models and risk analytics pipelines. * **Legal:** Safeguard sensitive inference workloads and document data. * **Defense and Government:** Maintain sovereignty over AI models and datasets. Additionally, all workloads comply with: * **HIPAA** (Health Insurance Portability and Accountability Act) * **GDPR** (General Data Protection Regulation) * **PCI DSS** (Payment Card Industry Data Security Standard) # Data Science Specification Source: https://io.net/docs/guides/clouds/data-science-image-full-specification This document provides the full specification of the Data Science Image used in IO.NET cloud deployments. It includes the operating system, core dependencies, and package lists for both the Conda `rapids-25.6.0` environment and the base Python environment. ## Base Information * **OS**: Ubuntu 24.04.1 LTS (Noble Numbat) * **CUDA**: 12.1 * **RAPIDS**: 25.6.0 * **Python**: 3.12 * **Conda**: 25.7.0 ## Package Lists The image comes with a preconfigured Conda environment (`rapids-25.6.0`) as well as a base Python environment.\ To see the full package versions installed, run the following commands inside the image: ### Conda environment (`rapids-25.6.0`) ```bash theme={null} conda run -n rapids-25.6.0 pip3 list ``` ### Base Python environment ``` pip3 list ``` > 📘 \*\*Note: \*\*The package lists may change over time as the image is updated. > > Users should always re-run the commands above to get the most up-to-date package versions. # Bare Metal (Self Serve) Moving Forward Source: https://io.net/docs/guides/clouds/deploy-bare-metal-cluster Bare Metal clusters give you direct access to hardware for maximum performance, low latency, high efficiency, and full control over configuration. ### Service Update: Bare Metal (Self Serve) Moving Forward Bare Metal on Demand will no longer be available starting Wednesday, October 1, 2025. If you need Bare Metal, please contact our sales team at [business@io.net](mailto:business@io.net) for managed services. The Bare Metal Automated Solution provides fast, self-service access to dedicated hardware resources, all of which eliminate delays from custom contracts and manual setup. This solution is ideal for high-performance workloads such as AI model training, making predictions (inferencing), and other compute-heavy tasks. Through an automated deployment flow, you can select and provision devices in minutes, while our real-time dashboard offers visibility into usage, performance metrics, and costs to optimize your deployments.