Migrating from Gemini 3g to Gemini 3h

With the introduction of Gemini 3h, a new chapter begins in our series of Gemini testnets. As we approach the mainnet launch, our software is undergoing significant changes that will impact you – our farmers, operators, nominators, and developers. We deeply appreciate the time and effort you dedicated to the previous Gemini 3g stage. However, to transition to the new Gemini, you must perform a complete wipe of your farmer (deleting all plots you created) and clear all node information. Essentially, everyone is starting the syncing and plotting process from scratch. This process is a familiar step for most of you, as it has been the case for every new network rollout.

With the switch to Gemini 3h, some important changes are coming to the CLI arguments for all the farmers, operators, and developers. Previously, the Subspace node had several arguments inherited from Substrate that were either irrelevant or not supported. With the release of Gemini 3h, we introduce an updated list of CLI arguments that are customized for the Subspace node to enhance the clarity of used flags. The most significant chasubnges will be covered in the next few sections, and the complete list of dropped arguments will also be provided at the end of this post. Note that our documentation is the source of truth for running our software, and it should always be checked carefully when transitioning between versions, especially when switching networks.

Introduction of a run command

Previously, the Subspace Node operated using an implicit command structure, meaning that the ‘run’ command wasn’t explicitly required. With the latest update, this approach has been inverted. Now, to start the Subspace Node, it’s mandatory to explicitly use the run command. The correct syntax is as follows: subspace_node run --base-path PATH_TO_NODE --chain gemini-3h. This change ensures clearer and more intentional command usage for node operation.

Setting base-path becomes mandatory
Setting --base-path is now mandatory, while all other CLI commands with a path were dropped and no longer exist.

Setting chain becomes mandatory
No ambiguity in setting the chain anymore as setting --chain CLI argument becomes mandatory.

Validator replaced with farmer
The Subspace Node farmers would appreciate the added clarity, as the flag --validator is replaced by --farmer.

A more meaningful setting of the RPC port
The awkward combination of the commands --rpc-external, --unsafe-rpc-external, --rpc-port is replaced by a single meaningful command --rpc-listen-on.

Setting the port
The combination of --port and --listen-addr is replaced by a single --listen-on flag. The format it needs to be specified in is multiaddr, for example /ip4/0.0.0.0/tcp/30333.

Newly available wipe command
For additional consistency, we’ve removed purge-chain command that you previously used to remove all the information stored by the Subspace Node and replaced it with wipe command instead. Starting with Gemini 3h, wiping of both node and a farmer will work as follows:

./node_file_name wipe PATH_TO_NODE

./farmer_file_name wipe PATH_TO_FARM

Allowing private IPs
You can use --allow-private-ips instead of dropped --allow-private-ip and --dns-enable-private-ips.

Making life easier for Prometheus users
The awkward combination of --prometheus-port, --prometheus-external, and --no-prometheus is replaced by --prometheus-listen-on followed by the value IP:port.

Directory Structure Change
Directory structure created/used by Subspace-node to the following:

subspace-node/db
subspace-node/domains/0/keystore
subspace-node/domains/0/db
subspace-node/domains/1/keystore
subspace-node/domains/1/db
subspace-node/network

Domain CLI arguments changes

Important Changes for Operators

  • The subspace-node domain key create command is introduced, to make the keypair creation simple, convenient, and safe.
  • The new subspace-node domain key insert command is available, if you need to move the existing keys, or already have a secret phrase you would like to use.
  • Setting --base-path is no longer supported as a domain argument, it’s being derived from consensus’s base path and domain ID.
  • Make sure to use --bootstrap-nodes instead of previously used --bootnodes.
  • --listen-on defaults to /ip4/0.0.0.0/tcp/30334 unless specified explicitly.
  • --rpc-listen-on defaults to 127.0.0.1:9945 unless specified differently.
  • --rpc-methods auto defaults to unsafe for localhost and safe for all other addresses.
  • Chain, network identity, and name are derived from consensus chain and domain ID automatically.
  • The list of the available bootnodes was updated, each getting a readable domain name.

Generating operator keypair
Generation of an operator keypair becomes much easier and safer - just run subspace-node domain key create --base-path /tmp/node --domain-id 0. Make sure to specify the --base-path for key generation and running an operator node and make sure to replace --domain-id value with the domain ID you would like to be an operator on.

Changes in Subspace Node CLI arguments for developers

Using --dev now implies the following:

  • --chain dev
  • --farmer
  • --tmp (unless -base-path specified explicitly)
  • --force-synced
  • --force-authoring
  • --allow-private-ips
  • --rpc-cors all (unless specified explicitly)
  • --dsn-disable-bootstrap-on-start
  • --timekeeper

Other important changes
NUMA support will benefit farmers with large, powerful CPUs and lots of RAM available. The required RAM amount depends on the processor.
Previously plotting/replotting thread pools were created for each farm separately even though only a configured number of them can be used at a time (by default just one).
With the introduction of NUMA support, the farmer application has a thread pool manager that will create the necessary number of thread pools that will be allocated to currently plotting/replotting farms.

When a thread pool is created, it is assigned to a set of CPU cores and will only be able to use those cores. Pinning doesn’t pin threads to cores 1:1, instead, the OS is free to move threads between cores, but only within CPU cores allocated for the thread pool. This will ensure plotting for a particular sector only happens on a particular CPU/NUMA node which is more efficient.

On Linux and Windows, the farmer will detect NUMA systems and create a number of thread pools that correspond to the number of NUMA nodes. This means the default behavior will change for large CPUs and consume more memory as a result, but that can be changed to the previous behavior with CLI options if desired.

NUMA enabling in BIOS is not necessary as the farmer will check L3 cache topology instead, benefiting larger Ryzen/Threadripper/Epyc processors.

To read more about NUMA support and the benefits it provides for large CPUs refer to this forum post.

Staking
We’re excited to introduce staking v1.1 which includes the following changes:

  • Bundles that operators produce now include storage fees for the bundle extrinsics.
  • The challenge period for fraud-proof has changed from 256 domain blocks (~26 minutes) to 14 400 domain blocks (~24 hours).
  • Domain block rewards were renamed to domain block fees. The total fees of the block include all execution and storage fees and tips for all of the transactions included in this block.
  • The minimum nominator stake is set to 1 TSSC. It can be set to higher amounts by operators, but can’t be set lower than 1 TSSC.
  • The locking period is changed from 256 domain blocks (~26 minutes) to 28 800 blocks (~2 days). This locking period is necessary to ensure that the domain block executing the withdrawal is confirmed and not challenged by a fraud-proof and to increase the economic stability of domains.

These are the most significant changes to the CLI arguments we introduced in the Gemini 3h releases. Once again, all changes are reflected in our official documentation which is the best resource to help you start in no time. Please make sure to re-read the forum post and the documentation in case you encounter any errors first, before reaching out to us through the Forum or Discord. Happy farming/operating/developing!

3 Likes

But in gemini-3g-2024-jan-29-2’s changelog we see this:

  • NUMA enabling in BIOS is no longer necessary as farmer will check L3 cache topology instead, benefiting larger Ryzen/Threadripper/Epyc processors

As a result, do I need to activate NUMA in the BIOS or not?

1 Like

Good catch Rabinovitch!
You’re right, it’s no longer necessary to enable NUMA through BIOS manually, the farmer will detect it automatically!

@Emilios

The release you’re using is broken, try using the jan-31-2 version and it should work just fine!

I see the command for Prometheus but how do I access the metrics for the Farmer? I believe metrics for the farmer are available for the farmer in 3h?

Yes, you just hit the HTTP endpoint at /metrics like you would for any other app.

I found the command via the --help. I run docker so I rarely actually run the binaries but figured it might be documented in there (duh). For anyone in the future it was:
“–metrics-endpoints”, “0.0.0.0:9615”,

shouldn’t beIP/port?

1 Like

Corrected to multiaddr, thanks!

Good job
You’re right, it’s no longer necessary to enable NUMA through BIOS manually, the farmer will detect it automatically!

1 Like