dify/api
GareArc bf422dfd13
feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs
Enterprise-only observability layer that emits OpenTelemetry traces, metrics,
and logs for workflow executions, LLM calls, message processing, and node runs.

Key components:
- core/telemetry: CE-safe gateway and event facade (no-ops when EE disabled)
- enterprise/telemetry: full trace/metric/log pipeline with OTLP exporter
- extensions/ext_enterprise_telemetry: Flask extension for lifecycle management
- tasks/enterprise_telemetry_task: Celery task for async metric dispatch

Wiring changes:
- app_factory: register ext_enterprise_telemetry extension
- ext_celery: conditionally import enterprise_telemetry_task
- configs/enterprise: add EnterpriseTelemetryConfig with OTLP settings
- ops_trace_manager: add enterprise telemetry hooks with lazy imports
- trace_entity: add new trace types (WorkflowNode, PromptGeneration, DraftNode)
- ext_otel: improve gRPC TLS auto-detection and optional auth headers

All enterprise imports are guarded behind ENTERPRISE_ENABLED +
ENTERPRISE_TELEMETRY_ENABLED feature flags with graceful fallbacks.
2026-03-24 00:24:20 -07:00
..
.idea
.vscode chore(api): update launch.json.example to include new workflow_based_app_execution. (#32184) 2026-02-10 17:08:43 +08:00
commands refactor: migrate db.session.query to select in infra layer (#33694) 2026-03-19 15:48:29 +09:00
configs feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
constants fix: add missing pipeline_templates (#31528) 2026-03-01 17:33:04 +08:00
context refactor: move workflow package to dify_graph (#32844) 2026-03-02 18:42:30 +08:00
contexts refactor(model): Refactor plugin model schema cache to be process-global to prevent redundant Daemon API calls (#31689) 2026-01-29 14:31:15 +08:00
controllers refactor: select in remaining console app controllers (#33969) 2026-03-24 02:53:05 +09:00
core feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
dify_graph fix: add max_retries=0 for executor (#33688) 2026-03-20 14:07:32 +08:00
docker feat(tasks): isolate summary generation to dedicated dataset_summary queue (#32972) 2026-03-06 14:35:28 +08:00
enterprise/telemetry feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
enums feat: add trial model list in system features (#31313) 2026-01-26 11:52:05 +08:00
events refactor: migrate db.session.query to select in infra layer (#33694) 2026-03-19 15:48:29 +09:00
extensions feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
factories refactor: migrate db.session.query to select in infra layer (#33694) 2026-03-19 15:48:29 +09:00
fields refactor: move workflow package to dify_graph (#32844) 2026-03-02 18:42:30 +08:00
libs fix: handle null email/name from GitHub API for private-email users (#33882) 2026-03-23 14:53:03 +08:00
migrations chore: change draft var to user scoped (#33066) 2026-03-16 14:04:41 +08:00
models refactor: use EnumText for TidbAuthBinding.status and MessageFile.type (#33975) 2026-03-24 05:38:29 +09:00
repositories refactor(api): tighten phase 1 shared type contracts (#33453) 2026-03-17 17:50:51 +08:00
schedule refactor: use EnumText for TidbAuthBinding.status and MessageFile.type (#33975) 2026-03-24 05:38:29 +09:00
services refactor: use EnumText for DocumentSegment.type (#33979) 2026-03-24 05:37:51 +09:00
tasks feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
templates refactor: make url in email template more better (#31166) 2026-01-19 14:28:41 +08:00
tests feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
.dockerignore
.env.example fix: do not block upsert for baidu vdb (#33280) 2026-03-23 20:42:57 +08:00
.importlinter refactor: llm decouple code executor module (#33400) 2026-03-16 10:06:14 +08:00
.ruff.toml refactor(api): move model_runtime into dify_graph (#32858) 2026-03-02 20:15:32 +08:00
AGENTS.md refactor(api): tighten phase 1 shared type contracts (#33453) 2026-03-17 17:50:51 +08:00
Dockerfile ci: Simplify nltk data download in Dockerfile (#33495) 2026-03-16 12:06:20 +09:00
README.md docs(api): simplify setup README and worker guidance (#32704) 2026-02-27 18:12:52 +08:00
app.py chore(typing): reduce ty excludes for A1 (#31721) 2026-01-30 02:38:57 +08:00
app_factory.py feat(telemetry): add enterprise OTEL telemetry with gateway, traces, metrics, and logs 2026-03-24 00:24:20 -07:00
celery_entrypoint.py
cnt_base.sh add cnt script and one more example (#28272) 2025-11-18 16:44:14 +09:00
dify_app.py
gunicorn.conf.py docs(api): update docs about gevent setup in app.py (#27611) 2025-10-30 15:43:08 +08:00
pyproject.toml chore(deps-dev): bump alibabacloud-gpdb20160503 from 3.8.3 to 5.1.0 in /api in the vdb group (#33879) 2026-03-23 17:25:44 +09:00
pyrefly-local-excludes.txt refactor: replace sa.String with EnumText for mapped_columns (#33547) 2026-03-17 23:30:23 +09:00
pyrightconfig.json feat: Add Hologres as a VDB & FullText DB choice (#32830) 2026-03-15 20:55:02 +08:00
pytest.ini chore: add pytest XML and branch coverage reports (#33730) 2026-03-19 17:08:34 +08:00
uv.lock chore(deps-dev): bump the dev group across 1 directory with 12 updates (#33919) 2026-03-23 17:26:47 +09:00

README.md

Dify Backend API

Setup and Run

[!IMPORTANT]

In the v1.3.0 release, poetry has been replaced with uv as the package manager for Dify API backend service.

uv and pnpm are required to run the setup and development commands below.

The scripts resolve paths relative to their location, so you can run them from anywhere.

  1. Run setup (copies env files and installs dependencies).

    ./dev/setup
    
  2. Review api/.env, web/.env.local, and docker/middleware.env values (see the SECRET_KEY note below).

  3. Start middleware (PostgreSQL/Redis/Weaviate).

    ./dev/start-docker-compose
    
  4. Start backend (runs migrations first).

    ./dev/start-api
    
  5. Start Dify web service.

    ./dev/start-web
    
  6. Set up your application by visiting http://localhost:3000.

  7. Start the worker service (async and scheduler tasks, runs from api).

    ./dev/start-worker
    
  8. Optional: start Celery Beat (scheduled tasks).

    ./dev/start-beat
    

Environment notes

[!IMPORTANT]

When the frontend and backend run on different subdomains, set COOKIE_DOMAIN to the sites top-level domain (e.g., example.com). The frontend and backend must be under the same top-level domain in order to share authentication cookies.

  • Generate a SECRET_KEY in the .env file.

    bash for Linux

    sed -i "/^SECRET_KEY=/c\\SECRET_KEY=$(openssl rand -base64 42)" .env
    

    bash for Mac

    secret_key=$(openssl rand -base64 42)
    sed -i '' "/^SECRET_KEY=/c\\
    SECRET_KEY=${secret_key}" .env
    

Testing

  1. Install dependencies for both the backend and the test environment

    cd api
    uv sync --group dev
    
  2. Run the tests locally with mocked system environment variables in tool.pytest_env section in pyproject.toml, more can check Claude.md

    cd api
    uv run pytest                           # Run all tests
    uv run pytest tests/unit_tests/         # Unit tests only
    uv run pytest tests/integration_tests/  # Integration tests
    
    # Code quality
    ./dev/reformat               # Run all formatters and linters
    uv run ruff check --fix ./   # Fix linting issues
    uv run ruff format ./        # Format code
    uv run basedpyright .        # Type checking