Configuration

An explanation of how to configure VES for your use case

v1.0.0 will introduce breaking changes to this documentation

---

Introduction

The core architecture components, that is; Core Agent, Vector Embedding Engine and Vector Enhanced Search can be configured using their specific configuration TOML files.

Specific configuration options for each component are explained. As much as these explanations are considered a good starting point, it is recommended you play around with your configuration options to find what works best for you.

v0.1.0 only supports configuration for the core agent, see Coming Soon

---

Core Agent Configuration

The agent’s default configuration can be found at the project’s config directory on your local repo

cd VES/services/log-collector/src/config

This is the default configuration file, ensure this is exactly how the configuration file looks like before you set your own configuration options

[general]
enable_local_mode = true
enable_network_mode = true

[watcher]
enabled = true
checkpoint_path = "path/to/checkpoint.json"
log_dir = "path/to/log/file(s)/directory"
poll_interval_ms = 5000
recursive = false

[parser]

[buffer]
capacity_option = "unbounded"
buffer_capacity = 10000
batch_size = 200
batch_timeout_ms = 500
overflow_policy = "drop_oldest"
flush_policy = "batch_size"
drain_policy = "batch_size"

[buffer.durability]
type = "s-q-lite"
path = "/var/ves-sqlite-db/parsed_log_buffer.db"

[shipper]
embedder_target_addr = "https://127.0.0.1:50051"
connection_timeout_ms = 500
max_reconnect_attempts = 10
initial_retry_delay_ms = 500
max_retry_delay_ms = 30000
backoff_factor = 2.0
retry_jitter = 0.2
send_timeout_ms = 3000
response_timeout_ms = 10000

[general]

Configures Core Agent to expect your observability data locally, over-the-network or from both sources

Local mode

[general]
enable_local_mode = true

1. Local mode set to true when Core Agent is running on the same node as your observability data producer
2. Over-the-network mode set to true when Core Agent is running on a separate node from your observability data producer
3. Dual mode set both local and over-the-network mode to true, when Core Agent is running on a node producing observability data but should collect data from another node

[watcher]

Configures Core Agent observability data file watcher, keeping track of the file containing your observability data while the agent works on this data

Enable

[watcher]
enabled = true

• Enable ensures the agent’s observability data watcher is enabled, should always be true
• Checkpoint directory checkpoint file location, allowing the agent’s watcher to resume on restart/crash
• Data directory configures the directory where the agent can find observability data while in local mode
• Watcher poll interval configures the duration before the agent’s watcher checks for updates to your observability data source
• Behavior configures the agent’s behavior (how it keeps track of your observability data source) at runtime

[parser]

Currently for v0.1.0 the Core Agent’s parser is non-configurable :)

Coming Soon.

Check out the latest improvements to v0.1.0. See latest improvements

[buffer]

Configures Core Agent’s buffering and batching implementation, ensuring the normalization pipeline remains performant. See normalization pipeline

Capacity

[buffer]
capacity_option = "bounded"

• Capacity defines capacity type for the agent’s buffers. It is recommended to use “bounded” capacity.
• Capacity size defines the total available capacity the agent allocates in buffers.
• Batch size defines batch size in agent’s buffers, this affects the size of each uniformly structured observability data batch the agent processes.
• Batch timeout defines the duration after which uniformly structured observability data will be taken from the agent’s buffers during processing, the data itself is the data appearing during the duration set.
• Overflow policy is how the agent will handle a scenario where its buffers are overflowing. This prevents buffer overflows which may lead to incorrect capacity calculations and memory safety issues.
• Flush policy is how the agent will handle flushing uniformly structured observability data from its buffers.
• Drain policy is how the agent will handle draining uniformly structured observability data from its buffers.

[buffer.durability]

Configures Core Agent durability setting, whether to persist data to storage or just use in-memory data while running

[buffer.durability]

# Options: (type = "s-q-lite" or type = "inmemory")
[buffer.durability]
type = "s-q-lite"
path = "/var/ves-sqlite-db/parsed_log_buffer.db"

It is recommended to run the agent while persisting data to storage not in-memory, to prevent data loss on restarts/crashes

[shipper]

Configures how the Core Agent ships uniformly structured data to destination sinks and/or the Vector Embedding Engine for storage and vectorization, respectively

Embedding engine addr

[shipper]
embedder_target_addr = "https://127.0.0.1:50051"

• Embedding engine address configures the address of a running vector embedding engine the agent will ship uniformly structured observability data to.
• Connection timeout configures the wait time the agent will have to try connecting to the vector embedding engine before failing the connection attempt.
• Reconnect attempts configures the number of attempts the agent will have to reconnect to the vector embedding engine before failing the reconnection attempt.
• Reconnect retry delay configures initial and max delay times the agent will use between each reconnection attempt made to the vector embedding engine.
• Reconnect safeguards are safety measures for the reconnection to the vector embedding engine, backoff_factor sets a multiplier for reconnection attempts while retry_jitter sets a random jitter percentage to reconnection attempts to prevent a thundering herd problem while attempting a reconnection.
• Request/Response timeouts are timeouts related to the agent---embedding engine gRPC stream, send_timeout_ms sets a max timeout to push a single uniformly structured observability data batch into the vector embedding engine gRPC stream, while response_timeout_ms sets a max timeout to wait for a response from the vector embedding engine before declaring the gRPC stream unhealthy.

Vector Embedding Engine Configuration

Coming Soon!

Vector Enhanced Search Configuration

Coming Soon!

Next Steps

Setup an instance

Setup and configure your own instance locally

Learn more about VES

Understand how the platform is designed and works