Skip to content

lablink_allocator_service.reboot

Automated VM reboot service.

Periodically checks for failed VMs and reboots them.

Assigned VMs get a warm reboot (plain sudo reboot) so the container survives via its --restart unless-stopped policy. Unassigned VMs get a cold reboot (cloud-init clean && sudo reboot) that recreates the container from scratch. Both paths fall back to EC2 stop/start (always cold) when SSH is unavailable.

Respects cooldown periods and maximum reboot attempt limits.

Background service that monitors for failed VMs and reboots them.

Parameters:

Name Type Description Default
database VmDatabase

VmDatabase instance for querying/updating VM state.

 required
region str

AWS region where VMs are deployed.

 'us-west-2'
tofu_dir str

Path to the OpenTofu directory (for SSH key retrieval).

 ''
max_attempts int

Maximum reboot attempts per VM before giving up.

 3
cooldown_seconds int

Minimum seconds between reboots of the same VM.

 300
check_interval_seconds int

How often to check for failed VMs.

 60
Source code in packages/allocator/src/lablink_allocator_service/reboot.py 
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def __init__(
    self,
    database: VmDatabase,
    region: str = "us-west-2",
    tofu_dir: str = "",
    max_attempts: int = 3,
    cooldown_seconds: int = 300,
    check_interval_seconds: int = 60,
    provider=None,
):
    self.database = database
    self.region = region
    self.tofu_dir = tofu_dir
    self.max_attempts = max_attempts
    self.cooldown_seconds = cooldown_seconds
    self.check_interval_seconds = check_interval_seconds
    self.provider = provider
    self._stop_event = Event()
    self._thread = None

Start the auto-reboot monitoring thread.

Source code in packages/allocator/src/lablink_allocator_service/reboot.py 
56
57
58
59
60
61
62
63
64
65
def start(self):
    """Start the auto-reboot monitoring thread."""
    self.database.ensure_reboot_columns()
    self._stop_event.clear()
    self._thread = Thread(target=self._run, daemon=True)
    self._thread.start()
    logger.info(
        f"Auto-reboot service started (interval={self.check_interval_seconds}s, "
        f"max_attempts={self.max_attempts}, cooldown={self.cooldown_seconds}s)"
    )

Stop the auto-reboot monitoring thread.

Source code in packages/allocator/src/lablink_allocator_service/reboot.py 
67
68
69
70
71
72
def stop(self):
    """Stop the auto-reboot monitoring thread."""
    self._stop_event.set()
    if self._thread:
        self._thread.join(timeout=10)
    logger.info("Auto-reboot service stopped")