Skip to main content

Upgrade Forest Nodes

Detailed Upgrade Steps​

1️⃣ Repository Update​

  • ** Repository**: infrastructure-general
  • ** Forest Version**: x.x.x
  • ** Environments**: Staging and Production

2️⃣ 🔕 Silence Alerts for Forest Nodes​

To prevent alert fatigue and unnecessary distractions, set up an alert silence in Alertmanager for all Forest nodes.

Steps:

  1. Access the 🔔 Alertmanager Console.
  2. Authenticate using credentials stored in 🔐 1Password (team access required).
  3. Navigate to Silences and create a new silence:
    • ⏳ Duration: Define a suitable silencing period.
    • Matchers:
      • job="forest"
      • alertname="InstanceDown"
    • ** Creator**: devops
    • ** Comment**: Upgrade to x.x.x
  4. 💾 Save the silence rule.

🔹 After implementing the alert silence, notify the Forest team that the upgrade has been initiated.

3️⃣ Staging Environment Testing​

Begin with a limited deployment on the staging environment to assess potential issues.

🏗 Forest Staging Upgrade​

🔹 Calibnet Network

cd ansible/forest-team-execution/
ansible-galaxy install -r requirements.yml --force
# Ensure the .vault_pass file contains the correct password from 1Password
ansible-playbook playbooks/start-forest-staging.yml -l fil-staging-ovh-forest-calibnet-0

🔹 Mainnet Network

ansible-playbook playbooks/start-forest-staging.yml -l fil-staging-ovh-forest-mainnet-0

4️⃣ Production Rollout​

After successful testing, proceed with the production upgrade in phases as outlined below.

🔹 4.1 Forest Snapshot Nodes​

  • Upgrade all **Forest snapshot ** by running:
  • Run: 
    cd ansible/filecoin-execution/
    ansible-galaxy install -r requirements.yml --force
    # Ensure the .vault_pass file contains the correct password from 1Password
    ansible-playbook playbooks/start-filecoin-snapshot.yml -l filecoin_snapshot

🔹 4.2 Forest RPC Nodes​

  • Upgrade all Forest RPC nodes by running: 
    ansible-playbook playbooks/start-forest-rpc-node.yml -l forest-rpc-nodes

🔹 4.3 Forest Calibnet Bootnodes​

  • Upgrade Forest Calibnet Bootnodes: 
    ansible-playbook playbooks/start-forest-lotus-calibnet-bootnodes.yml -l forest_calibnet_bootnode

🔹 4.4 Forest Mainnet Bootnodes​

  • Upgrade Forest Mainnet Bootnodes: 
    ansible-playbook playbooks/start-forest-lotus-mainnet-bootnodes.yml -l forest_mainnet_bootnode

5️⃣ 🔍 Health Verification​

Verify the status and health of each node using the following command:

curl -X POST 'http://localhost:2345/rpc/v0' -H 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"Filecoin.NodeStatus","params":[]}' | jq

✅ Ensure all nodes are stable and operational before proceeding to the final step.

6️⃣ 📢 Team Notification​

Once the upgrade and verification steps are completed:
✅ Notify the Forest team with the final upgrade status and any pertinent observations.

📌 🚀 Upgrade successfully completed! 🎉