Merge tag 'beta-1.1' into develop

Tagging version beta-1.1 beta-1.1

README.md

@@ -2,14 +2,476 @@
 
 Gateway between multiple HomesteadRelay's and the HomesteadToGo Bot.
 
+---
 
-## dev notes
+## HomesteadGateway Developer Documentation
 
-perhaps drop database, instead
+## Overview
+
+HomesteadGateway is a WebSocket-based message routing gateway that facilitates bidirectional communication between game server mods/plugins and external bots (e.g., Discord bots). It acts as a relay, routing messages based on channel identifiers and managing message queues when endpoints are offline.
+
+**Key Features:**
+- WebSocket-based real-time communication
+- Channel-based message routing
+- Automatic message queuing for offline recipients
+- API key authentication
+- Connection keep-alive via ping/pong
+- Support for multiple concurrent mod connections per channel
+
+---
+
+## Quick Start
+
+1. Start the gateway (build/run your application that embeds the WebSocket gateway).
+2. Connect a client to the WebSocket endpoint:
+    - URL: `ws://<host>:<port>/sync?api_key=<your_api_key>`
+3. Immediately send a handshake JSON with type `mod` or `bot`.
+4. On success, you’ll receive `{"status":"connected","type":"mod|bot"}`.
+5. Exchange messages as JSON.
+
+See the full details below.
+
+---
+
+## Architecture
+
+### Connection Types
+
+The gateway supports two types of connections:
+
+1. **Mod Connection** - Game server mods/plugins that send and receive player messages
+2. **Bot Connection** - External bots (typically Discord) that bridge messages to/from other platforms
+
+### Message Flow
 
 ```
-Mod     -> websocket /register { server_id, channel_id }                   // grabbed from mod config
-Bot     -> websocket /ready { channel_id }                                 // ready if Mod with fitting channel_id has called /register
-Gateway -> memory cache (server_id -> channel_id; channel_id -> server_id) // mem enough
-Mod/Bot -> websocket /ws { ... } -> Bot/Mod                                // sync
+Mod (Server) ←→ Gateway ←→ Bot (Discord)
+     ↓                          ↓
+  Channel A                 Channel A
 ```
+
+Messages are routed based on `channel_id`:
+- **Mod → Bot**: Messages from a mod are forwarded to the bot
+- **Bot → Mod**: Messages from a bot are forwarded to the mod registered for that channel
+
+---
+
+## Connection Setup
+
+### Endpoint
+
+```
+ws://<host>:<port>/sync?api_key=<your_api_key>
+```
+
+**Default Port:** 3333 (configurable)
+
+### Authentication
+
+Authentication is performed via API key, which can be provided in two ways:
+
+1. **Query Parameter** (recommended):
+   ```
+   ws://localhost:3333/sync?api_key=gateway
+   ```
+
+2. **HTTP Header**:
+   ```
+   X-API-Key: gateway
+   ```
+
+### Connection Timeout
+
+After connecting, you **must** send a handshake message within **60 seconds** or the connection will be closed.
+
+---
+
+## Handshake Protocol
+
+### Step 1: Establish WebSocket Connection
+
+Connect to the `/sync` endpoint with your API key.
+
+### Step 2: Send Handshake Message
+
+Immediately after connecting, send a JSON handshake message:
+
+```json
+{
+  "type": "mod",  // or "bot"
+  "data": { ... }
+}
+```
+
+#### Mod Handshake
+
+For game server mods/plugins:
+
+```json
+{
+  "type": "mod",
+  "data": {
+    "server_id": "minecraft-server-001",
+    "channel_id": "123456789"
+  }
+}
+```
+
+**Fields:**
+- `server_id` (string, required): Unique identifier for your server instance
+- `channel_id` (string, required): The Discord channel ID (or equivalent) this mod serves
+
+#### Bot Handshake
+
+For bots (Discord bots, etc.):
+
+```json
+{
+  "type": "bot",
+  "data": {
+    "bot_id": "discord-bot-123"
+  }
+}
+```
+
+**Fields:**
+- `bot_id` (string, required): The bots ID
+
+**Note:** Only **one bot connection** is allowed at a time. Trying to connect a new bot will result in an `409 Conflict` Error.
+
+### Step 3: Receive Acknowledgment
+
+After sending the handshake, wait for an acknowledgment:
+
+**Success Response:**
+```json
+{
+  "status": "connected",
+  "type": "mod|bot"
+}
+```
+
+**Error Responses:**
+```json
+{
+  "message": "Malformed handshake.",
+  "code": 400
+}
+```
+
+```json
+{
+  "message": "Bot already connected.",
+  "code": 409
+}
+```
+
+### Step 4: Begin Message Exchange
+
+Once acknowledged, the connection is established and you can start sending/receiving messages.
+
+---
+
+## Message Protocol
+
+### Sending Messages
+
+After handshake, send messages as JSON:
+
+#### From Mod to Bot
+
+```json
+{
+  "msg_id": "msg-unique-123",
+  "id": "minecraft-server-001",
+  "destination": {
+    "channel_id": "123456789"
+  },
+  "author": {
+    "id": "player-uuid-abc",
+    "name": "PlayerName"
+  },
+  "content": "Hello from the game server!",
+  "meta": {
+    "server_name": "Survival Server",
+    "world": "overworld"
+  },
+  "ts": "2025-12-08T10:30:00Z"
+}
+```
+
+**Required Fields:**
+- `msg_id` (string): Unique message identifier (generate client-side)
+- `id` (string): Server ID (must match your handshake `server_id`)
+- `destination.channel_id` (string): Target channel ID
+- `author.id` (string): User/player identifier
+- `content` (string): Message content (non-empty)
+
+**Optional Fields:**
+- `author.name` (string): Display name for the author
+- `meta` (object): Additional metadata (arbitrary key-value pairs)
+- `ts` (RFC3339 timestamp): Message timestamp (defaults to server time if omitted)
+
+#### From Bot to Mod
+
+```json
+{
+  "msg_id": "discord-msg-456",
+  "id": "123456789",
+  "author": {
+    "id": "discord-user-789",
+    "name": "DiscordUser"
+  },
+  "content": "Hello from Discord!",
+  "meta": {
+    "platform": "discord",
+    "roles": ["admin"]
+  },
+  "ts": "2025-12-08T10:31:00Z"
+}
+```
+
+**Required Fields:**
+- `msg_id` (string): Unique message identifier
+- `id` (string): Channel ID (from which channel the message originates)
+- `author.id` (string): User identifier
+- `content` (string): Message content (non-empty)
+
+**Optional Fields:**
+- `author.name` (string): Display name
+- `meta` (object): Additional metadata
+- `ts` (RFC3339 timestamp): Message timestamp
+
+**Note:** Bot messages do **not** include a `destination` field, as the channel ID in `id` determines routing.
+
+### Receiving Messages
+
+Messages are received as JSON in the same format they were sent:
+
+#### Mod Receives (from Bot)
+
+```json
+{
+  "type": "bot",
+  "channel_id": "123456789",
+  "author": {
+    "id": "discord-user-789",
+    "name": "DiscordUser"
+  },
+  "content": "Hello from Discord!",
+  "meta": {
+    "platform": "discord"
+  },
+  "ts": "2025-12-08T10:31:00Z",
+  "received_at": "2025-12-08T10:31:00.123Z",
+  "forwarded_at": "2025-12-08T10:31:00.125Z"
+}
+```
+
+#### Bot Receives (from Mod)
+
+```json
+{
+  "type": "mod",
+  "channel_id": "123456789",
+  "author": {
+    "id": "player-uuid-abc",
+    "name": "PlayerName"
+  },
+  "content": "Hello from the game server!",
+  "meta": {
+    "server_name": "Survival Server"
+  },
+  "ts": "2025-12-08T10:30:00Z",
+  "received_at": "2025-12-08T10:30:00.100Z",
+  "forwarded_at": "2025-12-08T10:30:00.102Z"
+}
+```
+
+**Additional Fields in Received Messages:**
+- `type` (string): Origin type ("mod" or "bot")
+- `channel_id` (string): The channel this message belongs to
+- `received_at` (RFC3339): When gateway received the message
+- `forwarded_at` (RFC3339): When gateway forwarded the message
+
+### Message Acknowledgments
+
+After sending each message, you'll receive an acknowledgment:
+
+```json
+{
+  "status": "completed",
+  "type": "mod"
+}
+```
+
+**Status Values:**
+
+- `completed`: Message was delivered immediately to recipient
+- `queued`: Recipient is offline; message queued for later delivery
+- `failed`: Message could not be delivered or queued
+
+---
+
+## Message Queuing
+
+When a recipient is offline, messages are automatically queued:
+
+- **Queue Size:** Configurable (default: 8 messages per channel)
+- **Queue Behavior:** Circular buffer (oldest messages are overwritten when full)
+- **Flush Trigger:** When recipient reconnects, all queued messages are delivered
+
+### Example Flow
+
+1. Mod sends message while bot is offline → Message queued
+2. Bot connects and completes handshake → All queued messages flushed to bot
+3. Bot sends message while mod is offline → Message queued
+4. Mod connects → Queued messages flushed to mod
+
+---
+
+## Keep-Alive & Ping/Pong
+
+The gateway sends **WebSocket ping messages every 30 seconds** to maintain connections.
+
+**Client Responsibilities:**
+1. **Respond to pings**: Your WebSocket library should automatically handle pong responses
+2. **Handle pongs**: Set a pong handler to reset read deadlines
+3. **Read Deadline**: The gateway sets a 60-second read deadline, reset on each pong
+
+### Example (Go)
+
+```go
+conn.SetPongHandler(func(appData string) error {
+    conn.SetReadDeadline(time.Now().Add(60 * time.Second))
+    return nil
+})
+```
+
+---
+
+## Error Handling
+
+### Connection Errors
+
+**Handshake Errors:**
+- `400` - Malformed handshake JSON
+- `401` - Invalid or missing API key
+- `409` - Bot already connected (only for bot handshakes)
+- `500` - Internal server error
+
+**Message Errors:**
+- `400` - Malformed message (missing required fields)
+
+Errors are sent as JSON:
+```json
+{
+  "message": "Malformed message.",
+  "code": 400
+}
+```
+
+### Validation Rules
+
+Messages are validated on receipt:
+- `id` must not be empty
+- `msg_id` must not be empty
+- `author.id` must not be empty
+- `content` must not be empty
+- For mod messages: `destination.channel_id` must not be empty
+
+### Websocket Closures
+
+- `1000` - Normal closure
+- `1001` - Going away
+
+Handle other WebSocket closures as unexpected errors.
+
+---
+
+## Rate Limits & Restrictions
+
+- **Message Size Limit:** 2 MB per message (configurable)
+- **Read Limit:** Messages exceeding the limit will close the connection
+- **Concurrent Mods:** Multiple mods *can* connect to the same channel ID (different server IDs)
+- **Concurrent Bots:** Only **one bot connection** allowed globally
+
+---
+
+## Best Practices
+
+### 1. Generate Unique Message IDs
+Always generate unique `msg_id` values for each message. Consider using:
+- UUID v4
+- Timestamp + random suffix
+- Sequential counter with prefix
+
+### 2. Handle Reconnections
+Implement automatic reconnection logic with exponential backoff:
+```
+1st retry: 1 second
+2nd retry: 2 seconds
+3rd retry: 4 seconds
+Max: 30 seconds
+```
+
+### 3. Set Appropriate Timeouts
+- Write timeout: 5 seconds (same as Gateway)
+- Read timeout: 60 seconds (reset on pong)
+
+### 4. Validate Before Sending
+Check required fields locally before sending to avoid validation errors.
+
+### 5. Monitor Acknowledgments
+Track acknowledgment statuses:
+- `completed`: Message delivered
+- `queued`: Message queued
+- `failed`: Log and potentially retry
+
+### 6. Use Metadata
+The `meta` field is used for:
+- Server information (server name, region, version)
+- User context (roles, permissions)
+- Message context (reply-to, thread-id)
+
+### 7. Thread-Safe Writes
+Use mutex/locks when writing to WebSocket from multiple threads.
+
+---
+
+
+## API Reference
+
+### Endpoints
+
+#### `GET /sync`
+WebSocket upgrade endpoint for mod/bot connections.
+
+**Query Parameters:**
+- `api_key` (required): Authentication token
+
+#### `GET /health`
+Health check endpoint.
+
+**Response:**
+```json
+{
+  "status": "healthy"
+}
+```
+
+---
+
+## Configuration
+
+Gateway configuration (`config.toml`):
+
+```toml
+[gateway]
+http_port = 3333          # WebSocket port
+websocket = "gateway"     # API key
+body_size = 1             # Max message size in MB
+queue_max = 8             # Messages per channel queue
+```
+
+---

bot.go

@@ -36,7 +36,7 @@ type Handshake struct {
 }
 
 type BotHandshake struct {
-	ChannelId string `json:"channel_id"`
+	BotID string `json:"bot_id"`
 }
 
 type GatewayAck struct {
@@ -111,7 +111,7 @@ func main() {
 		return nil
 	})
 
-	bhs := BotHandshake{ChannelId: channelID}
+	bhs := BotHandshake{BotID: "discord-bot"}
 	data, err := json.Marshal(bhs)
 	if err != nil {
 		_ = conn.Close()

config.toml

@@ -7,4 +7,4 @@ rotation  = 3 # in days
 http_port = 3333
 websocket = "gateway"
 body_size = 2 # in MB
-queue_max = 32
+queue_max = 8

controller/structs.go

@@ -1,8 +1,6 @@
 package controller
 
-import (
-	"homestead/homestead_gateway/ws"
-)
+import "homestead/homestead_gateway/ws"
 
 type GatewayController struct {
 	Websocket  *ws.WebsocketGateway

main.go

@@ -3,11 +3,20 @@ package main
 import (
 	"flag"
 	"homestead/homestead_gateway/controller"
+	"homestead/homestead_gateway/util"
 	"homestead/homestead_gateway/util/config"
+	"os"
+	"path"
 )
 
 func main() {
-	cfgPath := flag.String("config", "config.toml", "configuration file")
+	dir := util.GetPath()
+	if dir == "" {
+		dir, _ = os.Getwd()
+	}
+
+	file := path.Join(dir, "config.toml")
+	cfgPath := flag.String("config", file, "configuration file")
 	cfg, err := config.LoadConfig(*cfgPath)
 	if err != nil {
 		panic(err)
@@ -19,3 +28,5 @@ func main() {
 		panic(err)
 	}
 }
+
+// todo logs from exe not cwd

sim.go

@@ -23,7 +23,7 @@ const (
 	apiKey     = "gateway"
 	serverID   = "test-server-001"
 	// THE CHANNEL ID the mod says it serves. Must match gateway expectation.
-	channelID = "123456789"
+	channelID = "1444253682777587804"
 )
 
 // send interval range (random between minInterval and maxInterval)

util/logger/log.go

@@ -3,6 +3,7 @@ package logger
 import (
 	"context"
 	"fmt"
+	"homestead/homestead_gateway/util"
 	"homestead/homestead_gateway/util/config"
 	"log/slog"
 	"os"
@@ -23,6 +24,7 @@ func New(id string, cfg config.LogConfig) (*slog.Logger, func() error, error) {
 		cfg.Rotation = 7
 	}
 
+	cfg.Directory = util.NormalizeLogPath(cfg.Directory)
 	console := slog.NewTextHandler(&prefixWriter{inner: os.Stderr, prefix: []byte("[" + id + "] "), startLine: true}, &slog.HandlerOptions{AddSource: true, Level: cfg.Level})
 	router := newFileRouter(cfg.Directory, cfg.Rotation, id)
 	root := slogmulti.Fanout(console, router)

util/util.go

@@ -0,0 +1,51 @@
+package util
+
+import (
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/gorilla/websocket"
+)
+
+func GetPath() string {
+	exe, err := os.Executable()
+	if err != nil {
+		return ""
+	}
+
+	exe, err = filepath.EvalSymlinks(exe)
+	if err != nil {
+	}
+
+	return filepath.Dir(exe)
+}
+
+func NormalizePath(path string) string {
+	return filepath.Clean(filepath.FromSlash(path))
+}
+
+func NormalizeLogPath(path string) string {
+	if filepath.IsAbs(path) {
+		return path
+	}
+
+	return NormalizePath(filepath.Join(GetPath(), path))
+}
+
+//
+
+func CloseConnWithControlMessage(conn *websocket.Conn, typ int, text string) {
+	_ = conn.SetWriteDeadline(time.Now().Add(time.Second))
+	_ = conn.WriteControl(
+		typ, websocket.FormatCloseMessage(typ, text), time.Now().Add(time.Second),
+	)
+	_ = conn.Close()
+}
+
+func CloseConn(conn *websocket.Conn) {
+	CloseConnWithControlMessage(
+		conn, websocket.CloseNormalClosure,
+		"Disconnecting.",
+	)
+}

ws/handlers.go

@@ -68,7 +68,7 @@ func (wsg *WebsocketGateway) handleSync(w http.ResponseWriter, r *http.Request)
 		_ = wsg.sendWebsocketResponse(conn, GatewayAck{Status: "connected", Type: "mod"})
 		wsg.registry.FlushChannelWithSender(mhs.ChannelID, wsg.flush)
 
-		wsg.logger.Info("Mod connected via Websocket.", "remote", conn.RemoteAddr().String())
+		wsg.logger.Info("Mod connected via Websocket.", "remote", conn.RemoteAddr().String(), "id", mhs.ServerID)
 		go wsg.read(conn, "mod", mhs.ChannelID)
 
 	case "bot":
@@ -78,12 +78,12 @@ func (wsg *WebsocketGateway) handleSync(w http.ResponseWriter, r *http.Request)
 			return
 		}
 
-		if bhs.ChannelId == "" {
+		if bhs.BotID == "" {
 			wsg.sendWebsocketError(conn, "Malformed bot handshake.", 400, true)
 			return
 		}
 
-		if !wsg.registerConn(conn, "bot", bhs.ChannelId, "") {
+		if !wsg.registerConn(conn, "bot", "", "") {
 			wsg.sendWebsocketError(conn, "Bot already connected.", 409, true)
 			return
 		}
@@ -91,8 +91,8 @@ func (wsg *WebsocketGateway) handleSync(w http.ResponseWriter, r *http.Request)
 		_ = wsg.sendWebsocketResponse(conn, GatewayAck{Status: "connected", Type: "bot"})
 		wsg.registry.FlushAllToBotWithSender(wsg.flush)
 
-		wsg.logger.Info("Bot connected via Websocket.", "remote", conn.RemoteAddr().String())
-		go wsg.read(conn, "bot", bhs.ChannelId)
+		wsg.logger.Info("Bot connected via Websocket.", "remote", conn.RemoteAddr().String(), "id", bhs.BotID)
+		go wsg.read(conn, "bot", "")
 
 	default:
 		wsg.sendWebsocketError(conn, "Unknown handshake.", 400, true)

ws/registry.go

@@ -2,6 +2,7 @@ package ws
 
 import (
 	"fmt"
+	"homestead/homestead_gateway/util"
 	"time"
 
 	"github.com/gorilla/websocket"
@@ -92,8 +93,7 @@ func (r *Registry) RegisterMod(channelID, serverID string, conn *websocket.Conn)
 	defer e.mu.Unlock()
 
 	if e.Mod != nil && e.Mod.Conn != nil {
-
-		_ = e.Mod.Conn.Close()
+		util.CloseConn(e.Mod.Conn)
 	}
 
 	e.Mod = &ConnWrapper{Conn: conn, ServerID: serverID, LastSeen: time.Now()}
@@ -106,7 +106,7 @@ func (r *Registry) RegisterBot(conn *websocket.Conn) {
 	r.botMu.Lock()
 
 	if r.bot != nil && r.bot.Conn != nil {
-		_ = r.bot.Conn.Close()
+		r.UnregisterBot()
 	}
 
 	r.bot = &ConnWrapper{Conn: conn, LastSeen: time.Now()}
@@ -127,7 +127,7 @@ func (r *Registry) UnregisterMod(channelID string) {
 	e.mu.Unlock()
 
 	if modConn != nil && modConn.Conn != nil {
-		closeConn(modConn.Conn)
+		util.CloseConn(modConn.Conn)
 	}
 }
 
@@ -138,7 +138,7 @@ func (r *Registry) UnregisterBot() {
 	r.botMu.Unlock()
 
 	if botConn != nil && botConn.Conn != nil {
-		closeConn(botConn.Conn)
+		util.CloseConn(botConn.Conn)
 	}
 }
 
@@ -152,7 +152,6 @@ func (r *Registry) Send(channelID string, out GatewayMessageOut, sendOverConn fu
 			if err := sendOverConn(b.Conn, out); err == nil {
 				return true, false, nil
 			}
-			_ = b.Conn.Close()
 			r.UnregisterBot()
 		}
 
@@ -176,7 +175,6 @@ func (r *Registry) Send(channelID string, out GatewayMessageOut, sendOverConn fu
 		if err := sendOverConn(mod.Conn, out); err == nil {
 			return true, false, nil
 		}
-		_ = mod.Conn.Close()
 		r.UnregisterMod(channelID)
 	}
 

ws/structs.go

@@ -106,5 +106,5 @@ type ModHandshake struct {
 }
 
 type BotHandshake struct {
-	ChannelId string `json:"channel_id"`
+	BotID string `json:"bot_id"`
 }

ws/util.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"homestead/homestead_gateway/util"
 	"net/http"
 	"strings"
 	"time"
@@ -55,7 +56,7 @@ func (wsg *WebsocketGateway) sendWebsocketError(conn *websocket.Conn, message st
 	_ = conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
 	_ = conn.WriteJSON(map[string]interface{}{"message": message, "code": code})
 	if close {
-		_ = conn.Close()
+		util.CloseConn(conn)
 	}
 }
 
@@ -64,7 +65,7 @@ func (wsg *WebsocketGateway) sendWebsocketResponse(conn *websocket.Conn, content
 
 	if err := conn.WriteJSON(content); err != nil {
 		wsg.logger.Error("Failed to respond to connection.", "remote", conn.RemoteAddr().String(), "err", err)
-		_ = conn.Close()
+		util.CloseConnWithControlMessage(conn, websocket.CloseAbnormalClosure, "Connection error.")
 		return err
 	}
 
@@ -108,16 +109,6 @@ func (wsg *WebsocketGateway) loggingMiddleware(next http.Handler) http.Handler {
 
 // connections
 
-func closeConn(conn *websocket.Conn) {
-	_ = conn.SetWriteDeadline(time.Now().Add(time.Second))
-	_ = conn.WriteControl(
-		websocket.CloseMessage,
-		websocket.FormatCloseMessage(websocket.CloseNormalClosure, "Disconnecting."),
-		time.Now().Add(time.Second),
-	)
-	_ = conn.Close()
-}
-
 func (wsg *WebsocketGateway) registerConn(conn *websocket.Conn, typ, channelId, serverId string) bool {
 	if typ == "bot" {
 		wsg.registry.botMu.Lock()

ws/websocket.go

@@ -128,13 +128,10 @@ func (wsg *WebsocketGateway) read(conn *websocket.Conn, _type, channelId string)
 			ForwardedAt: time.Now().UTC(),
 		}
 
-		delivered, queued, err := wsg.registry.Send(out.ID, out, func(c *websocket.Conn, m GatewayMessageOut) error {
-			_ = c.SetWriteDeadline(time.Now().Add(5 * time.Second))
-			return c.WriteJSON(m)
-		})
+		delivered, queued, err := wsg.registry.Send(outID, out, wsg.flush)
 
 		if err != nil {
-			wsg.logger.Error("registry send error", "err", err)
+			wsg.logger.Error("Registry queue/delivery error.", "remote", conn.RemoteAddr().String(), "err", err)
 			_ = wsg.sendWebsocketResponse(conn, GatewayAck{Status: "failed", Type: message.Type})
 			continue
 		}