Data Flow

Dyngle provides special operators to pass data between steps in an operation, enabling powerful data processing workflows.

Data Flow Operators

Data Assignment Operator (=>)

Captures stdout from a command and assigns it to a named value:

dyngle:
  operations:
    fetch-data:
      - curl -s "https://api.example.com/users" => users
      - echo "Fetched: {{users}}"

The value becomes available in the data context for subsequent steps.

Data Input Operator (->)

Passes a data value as stdin to a command:

dyngle:
  operations:
    process-data:
      - curl -s "https://api.example.com/data" => raw-data
      - raw-data -> jq '.items' => filtered
      - filtered -> python process.py

Combining Operators

You can use both operators in a single step:

<input-variable> -> <command> => <output-variable>

Example:

dyngle:
  operations:
    weather:
      - curl -s "https://api.example.com/weather" => weather-data
      - weather-data -> jq -j '.temperature' => temp
      - echo "Temperature: {{temp}} degrees"

Practical Examples

API Data Processing

dyngle:
  operations:
    get-user-emails:
      return: emails
      steps:
        - curl -s "https://api.example.com/users" => users
        - users -> jq -r '.[].email' => emails

Multi-step Pipeline

dyngle:
  operations:
    analyze-logs:
      return: summary
      steps:
        - curl -s "https://logs.example.com/today" => logs
        - logs -> grep "ERROR" => errors
        - errors -> wc -l => error-count
        - echo "Found {{error-count}} errors" => summary

Data Transformation

dyngle:
  operations:
    transform-json:
      return: result
      steps:
        - cat input.json => raw
        - raw -> jq '.data | map({id, name})' => transformed
        - transformed -> python format.py => result

Important Notes

Operator Spacing

Operators must be isolated with whitespace:

Correct:

- command => output
- input -> command
- input -> command => output

Incorrect:

- command=>output        # Missing spaces
- input->command         # Missing spaces

Operator Order

When using both operators, they must appear in this order:

  1. Input operator (->) first
  2. Command in the middle
  3. Output operator (=>) last

Data Precedence

Values populated with => have the highest precedence in the data context.

See Data and Templates for complete precedence rules.

Using Expressions with Data Flow

You can reference captured data in expressions:

dyngle:
  operations:
    process:
      expressions:
        message: "format('Processed {{count}} items')"
      steps:
        - curl -s "https://api.example.com/items" => items
        - items -> jq 'length' => count
        - echo "{{message}}"

Accessing Nested Properties

When data flow captures structured data (JSON, YAML), use dot notation to access nested properties:

dyngle:
  operations:
    weather:
      steps:
        - curl -s "https://api.example.com/weather" => weather-data
        - echo "Temperature: {{weather-data.temperature}}"
        - echo "City: {{weather-data.location.city}}"

Remember: Dot notation works only for dictionary properties, not for array indices. For arrays, extract values using expressions or tools like jq.

Next Steps