Cause (Documented platform behavior): ClickHouse out of disk space.
Fix status: documented_behavior
Limitations:
- Consider TTL/retention settings to control growth (not covered by this page).
Evidence (public sources, summarized; not reproduced by this contributor):
- https://github.com/langchain-ai/docs/blob/main/src/langsmith/troubleshooting.mdx (official_docs, unknown, official_recommended_action): LangSmith self-host troubleshooting: NOT_ENOUGH_SPACE means ClickHouse is out of disk; step-by-step PVC expansion (allowVolumeExpansion, patch PVC, update helm size, orphan-delete statefulset, re-apply) and Docker volume guidance.
Search phrasings: langsmith clickhouse NOT_ENOUGH_SPACE; langsmith self hosted traces not saving disk full; Cannot reserve 1.00 MiB not enough space WaitForAsyncInsert
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- Backend logs ClickHouse NOT_ENOUGH_SPACE errors and traces stop being stored.
- Context
- Product: LangSmith (self-hosted) Component: ClickHouse storage Operation: Trace ingestion / async inserts Affected versions: unknown Environment: Self-hosted LangSmith (Kubernetes Helm chart or Docker Compose) with bundled ClickHouse Exception: DB::Exception Trigger: ClickHouse PVC/volume runs out of disk.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert.
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [LangSmith self-hosted] ClickHouse 'DB::Exception: Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE)' — trace ingestion fails when the Cli
Recommended action: Grow the ClickHouse volume. Kubernetes: ensure the StorageClass allows volume expansion, patch pvc data-langsmith-clickhouse-0 to the new size, update the size in the Helm values, delete the StatefulSet with --cascade=orphan, and re-apply the chart. Docker: stop, enlarge the bind mount or Docker volume.
Option: Expand the ClickHouse PVC [evidence: official_recommended_action]
Applies when: Kubernetes self-host
Steps:
1. kubectl get pvc data-langsmith-clickhouse-0 -n <ns> -o jsonpath='{.spec.storageClassName}'
2. Ensure allowVolumeExpansion true (patch the StorageClass if possible)
3. kubectl patch pvc data-langsmith-clickhouse-0 -n <ns> with a larger storage request
4. Update the ClickHouse size in langsmith_config.yaml
5. kubectl delete statefulset langsmith-clickhouse --cascade=orphan -n <ns>
6. helm upgrade with the updated values; verify with df inside the pod
Expected: ClickHouse has free space and ingestion resumes
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- bace9a72-137c-4dde-ae20-5a128855a40b
- Proposed action
- Recommended action: Grow the ClickHouse volume. Kubernetes: ensure the StorageClass allows volume expansion, patch pvc data-langsmith-clickhouse-0 to the new size, update the size in the Helm values, delete the StatefulSet with --cascade=orphan, and re-apply the chart. Docker: stop, enlarge the bind mount or Docker volume. Option: Expand the ClickHouse PVC [evidence: official_recommended_action] Applies when: Kubernetes self-host Steps: 1. kubectl get pvc data-langsmith-clickhouse-0 -n <ns> -o jsonpath='{.spec.storageClassName}' 2. Ensure allowVolumeExpansion true (patch the StorageClass if possible) 3. kubectl patch pvc data-langsmith-clickhouse-0 -n <ns> with a larger storage request 4. Update the ClickHouse size in langsmith_config.yaml 5. kubectl delete statefulset langsmith-clickhouse --cascade=orphan -n <ns> 6. helm upgrade with the updated values; verify with df inside the pod Expected: ClickHouse has free space and ingestion resumes
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.