Langfuse API: Investigate fetch failures

[sentry]

Is your feature request related to a problem?
The Langfuse API is currently unavailable, resulting in multiple consecutive trace fetch failures. This leads to interruptions in processes that rely on trace scores, causing delays and inefficiencies.

Stacktrace
Sentry Issue: KAAPI-PRODUCTION-11

JSONDecodeError: Extra data: line 1 column 5 (char 4)
  File "langfuse/api/resources/trace/client.py", line 90, in get
    _response_json = _response.json()
  File "httpx/_models.py", line 832, in json
    return jsonlib.loads(self.content, **kwargs)
  File "__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 341, in decode
    raise JSONDecodeError("Extra data", s, end)

ApiError: status_code: 429, body: 429 - rate limit exceeded
(1 additional frame(s) were not displayed)
...
  File "app/crud/evaluations/langfuse.py", line 493, in fetch_trace_scores_from_langfuse
    trace_data = future.result()
  File "app/crud/evaluations/langfuse.py", line 425, in _fetch_single_trace
    trace = langfuse.api.trace.get(trace_id)

RuntimeError: Langfuse API unavailable: 5 consecutive trace fetches failed. Aborting to prevent prolonged outage.
  File "app/crud/evaluations/langfuse.py", line 532, in fetch_trace_scores_from_langfuse
    raise RuntimeError(

[fetch_trace_scores_from_langfuse] Failed to fetch trace scores | dataset=goldenqna2upli2 | run=InitialTestingV6 gpt4o | Langfuse API unavailable: 5 consecutive trace fetches failed. Aborting to prevent prolonged outage.

Describe the solution you'd like

• Investigate the source of the Langfuse API failures.
• Implement retry logic for failed trace fetches.
• Provide clear error messages for users during outages.
• Evaluate the rate limiting on the Langfuse API and adjust if possible.

[AkhileshNegi]

This is initially raised by Glific team when experimenting with an NGO named Sagex with lagnfuse free account
Discord Glific support thread
Internal Support thread
We're parking this for now as have higher-priority items to finish before the AI Cohor. In the meantime, suggested to use Tech4Dev Langfuse account for these runs since it's on a paid plan with higher limits, so rate limiting shouldn't get in the way or use evals with smaller limits

[AkhileshNegi]

Glific team has incorporated similar error at their side as well so we can close it for now